Agreed here. I haven't actually used staging env as described here (single instance where you can test. We have used what is called preview instances for quite a while. We tend to keep 2 hot, and they get recycled ever X hours and new binaries are installed. These allow a single person to check it out and use it (checking one out causes a new instance to be spun up).
We still have a sandbox instance, and that is hooked up to the rest of the stack.
It works pretty well if you need to iterate on something quickly that you are unable to do with a test.
Coherence is obviously trying to sell you their product here but I'm not sure preview environments and staging environments are opposed.
They way I've set things up at my current company is this:
When a developer creates a pull request an ephemeral (preview) environment is generated.
When that code is merged it automatically gets deployed first to our staging environment and then immediately to production.
Staging gets a sanitized copy of production's data every night so both the code and the data are close mirrors of production and that's what developers (and those preview environments) integrate with when they're calling services other than their own.
If you were to get rid of staging would you stand up your entire backend every time you made a preview environment? Does that include replicating databases or would you just use seed data that may or may not resemble what's in production?
It seems like the solution they're pitching works great if you have a small team and only a few services/sites.
> If you were to get rid of staging would you stand up your entire backend every time you made a preview environment? Does that include replicating databases or would you just use seed data that may or may not resemble what's in production?
What I've seen done is that the staging environment is skipped, but instead concentric releases are performed on the customers. So released code goes first to a small group of users, and then widens from there if everything looks good.
> What I've seen done is that the staging environment is skipped, but instead concentric releases are performed on the customers. So released code goes first to a small group of users, and then widens from there if everything looks good.
If you really want to invest in extensive testing, you can do both! Hell, _and_ you can have a load testing environment as well.
Yeah, we had tooling to load a snapshot of customer data (with PII removed) into an ephemeral environment for when developers needed data that was more authentic. This worked for performance related work or just for features that needed a complex data model to be useful (ML apps, etc)
Yeah. And it's also not that impressive. We've had what they call "preview environments" for a very long time and at my previous companies too. With GitHub and AWS/Cloud it's trivially easy to do.
And like you, we have both "preview environments" and staging and for the exact reason you've highlight. I'm rather skeptical of the company behind this product now since they don't seem to understand the difference and really overselling something that's quite common.
Yes, this is one of those things that was easier before kubernetes.
Before: Your developers have a shared Linux box that serves the preview environment. Your ruby app has a tiny loader in front of it that loads the app code from a specific directory based on subdomain.
After: You need to route subdomains to different kubernetes containers, and handle deploying new containers when the developer opens a PR (because it's much more work than just copying files onto a server), and you need to handle destroying old containers (because it's much more work than just deleting files on a server).
Not to mention, usually kubernetes is managed by "the ops" who don't know anything about the app. Much more difficult to interface with them when it's not "oh one server, and the developers figure it out, and if they royally fuck it up we'll rescue them".
It’s really not that difficult to automate it over helm or something. All you need is a registry, and most cloud services provide a registry, even GitHub has one.
Routing domains to a container shouldn’t be significantly different to doing it in production.
Helm, registry, firewall, ingress, reverse proxy, these are all concepts that whoever sets up this hypothetical staging environment will have to know about, and will also have to be a developer (in order to care). So basically you need a senior dev to either love kubernetes or learn enough of it to get the job done. This is what I mean by "harder".
I have spent a lot of time thinking about this problem. I would gladly pay for a product that could make a real, honest to god preview environment that you could spin up on-demand and populate it with all of the data you need.
This isn't it, and I think it's pretty much an impossible problem to solve without an entirely new engineering culture around testing.
How do you duplicate the behavior of the 10 direct data dependencies of your app and the constellation of transitive microservice dependencies? Wingman, Galactus, etc. At a company with 1000+ microservices?
And how do you populate it with actual data so you can log in and perform actions? Every single microservice probably needs test data, and it would take a lot of engineering forethought to populate the entire graph.
It gets gnarly. I've been on teams at the center of it all - user accounts and login, representation of the core business entities that the rest of the company is built upon. We couldn't even solve the problem for ourselves. There were so many permutations of account states, creating a dedicated API to make test data deviates from a first class API. So how do you coordinate with downstream services to skip IDV (matching SSN), compromised password checks, etc.
Perhaps you only bring online a subset of services and fake out the rest. At what level can you fake stuff out? And how do you ensure that the test data works with whatever subset you don't fake? If a single service responds in an unexpected way, it could break or corrupt the state you're relying upon to test.
Maybe "preview" works for frontend. Backend, not so much. This is such a massive problem, and I'm only scratching the surface with my description of the kinds of issues you have to solve.
Microservices became this mess because people only think about production. Nobody builds microservices as standalone units that should be testable and runnable in isolation, with ZERO external data. Instead, everything relies on everything, so it's just a monolith split up into microrepos. Worse, people have started putting app configuration in S3 and other dumb things (I've seen it...) do you cannot run things locally, or even match production unless you have AWS creds. Madness.
> Nobody builds microservices as standalone units that should be testable and runnable in isolation, with ZERO external data. Instead, everything relies on everything, so it's just a monolith split up into microrepos.
Yeah this is the surprising thing to me - I've worked with otherwise bright people but whenever we've introduced services/microservices, it seems the default reaction is to just make a "distributed monolith" with circular dependencies and tight coupling.
I'm only half joking that 90% of my "architect" role is just point out we shouldn't have circular dependencies.
> Instead, everything relies on everything, so it's just a monolith split up into microrepos
…and with added network calls!
> Microservices. grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too. seem very confusing to grug
The way I originally did it at my previous company was to have each service have a "dev" Flask blueprint that got registered on non-production environments, that had an endpoint that could populate the database with sample data, and a coordinating service to populate them in the right order (so IDs matched up). Worked pretty well.
Sounds like a large dependency graph and multiple variants of spinning up the environments.
I'll try to simplify what I mean by that but something akin to having Services A, B, and C.
Each of A, B, and C need their own definitions to define how they need to be spun up as well as a definition on how to populate their data. That data population, say through RDS, needs to be spin up RDS instances and make them ready and available whenever someone creates a corresponding ephemeral version of a service. Toss the database afterwards.
In the above case, assuming you needed a full graph, you would spin up the three services as well as attach each service to its corresponding database. This approach works well when the number of services remains small.
Once that number starts growing, it may not be cost effective to spin everything up. Say service N is needed, but you really only need to read from it. In this case, you might have a dedicated staging environment for N and create a dependency graph that says "in my ephemeral environment, create new instances of A, B, and C with their databases, but also allow network requests to make their way to N."
That might be over simplifying, but I've been working in this space for the past few years and have been thinking about this exact issue recently
While the problem is still hard, it is definitely easier to solve with fewer, larger services. If your backend consists of 3-5 services, the problem is far more tractable than if it consists of 1000+ services.
Yeah that's a really good point. Our microservices are like 30? 1000+ is intractable but up to 50 or even a 100 it's doable. We rely pretty heavily on AWS Cloudformation to deploy. All our services have Cloudformation templates. I'm sure Terraform would work too or be even better. Since every service knows how to deploy itself, building a preview environment wasn't too difficult. It's just a matter of stitching them all together, which was made a lot easier with Cloudformation nesting of stacks.
I think what made it quite easy for us is that we maintain a staging environment. Staging is constantly tested by QA simulating our workflows so non-production data is being constantly generated. When we spin up our preview environments, we use snapshots from the staging systems to seed the preview environments.
Your point on data though really gets to the crux of the problem. Deploying the services is trivial these days with k8s, Cloud Formation, Terraform, Docker, etc. but the data part is hard.
I'd love to learn more about the challenges you have with data. My company essentially provides data-anonymization-as-a-service and we integrate with ephemeral environments, but we could probably improve the integrations.
Got a link to your company's service? Would love to learn more. We work in healthcare so data-anonymization is essential. If we can take prod data and erase all the PII, it would be AMAZING.
> How do you duplicate the behavior of the 10 direct data dependencies of your app and the constellation of transitive microservice dependencies? Wingman, Galactus, etc.
How long do your PRs stay open? Could you explain a bit more about what happens between opening the PR and merging it, like automated/manual testing? At my company we do something similar, except we trigger production deployments off tags, not merged PRs. Merging goes to staging only.
It really depends on the PR. Most do get merged fairly quickly (same day) but some where there's more discussion can stay open for several days or even a week.
All of our codebases have some level of automated testing that runs when the PR is created. That could be unit tests of a single function or Playwright tests which exercise an UI flow. What those tests are dependent on the type of software and the team building it.
The preview environment links that are generated when the PR is open are given to QA, Design, and Product to take a look at. Not every one of those roles review every PR, it really depends on the task. But if their feedback is required, they'll give it at this stage.
Probably worth mentioning at this point that we have separate repos for each web application and service. It's service oriented but not microservices, services at our company encapsulate a relatively large domain and it's similar with web applications, right now each web application has its own subdomain.
So PR gets merged, the pipeline will deploy that code to staging. At this point the service owners have the option to define a set of integration tests, not every repo has them but it is an option to run them. If those tests fail we do a rollback. If those tests pass (or if they didn't have any defined) then a second deployment to production is triggered.
All of our deploys are blue-green, so no downtime except for the occasional blocking database migration in which we have to schedule downtime. User facing features on the web applications are all required to be released first behind a feature flag. For those there will be several releases behind a feature flag, then we'll flip the flag first in staging. We'll have Product and QA do a complete run through and make sure they're okay with everything, then the "real" release happen when we flip the feature flag in production (no deployment needed).
Every night we clone the production database, sanitize it (swapping out things like names, phone numbers, etc.) then point staging to that clone. Then we run an end to end regression test that goes through all of the happy path flows for all of our apps.
My company can anonymize production data while preserving the utility of the data. You get privacy-safe data that looks just like production data with the same distributions. It's real data.
The typical workflow is to anonymize production data, snapshot it, and make it available to preview and testing environments as replicas. It's pretty fast.
The reason every complex application I've worked on has had a staging environment is because you do need to test production deploys in an environment that mirrors production dataset and infrastructure. Especially with data migrations, distributed databases. That is prohibitively expensive and not feasible to run in n+1 envs.
Big if, but if you can use database containers it's relatively low cost to spin them up in a namespace, load data from a database snapshot, run the migration, and then tear down the container or even the entire namespace.
Setting up preview environments automatically (in kubernetes or azure) with sanitized production data is something that always sounds amazing to me, but I have always had difficulty to come up with ideas to implement it. Every company/team uses different tools and ways to implement this, so there isn't a one-size-fits-all solution.
I agree completely. At my last company, we built internal tools to provide these ephemeral environments on demand to engineers, QA, and even sometimes product managers. Eventually, we grew tired of managing internal tools and tried to use off-the-shelf deployment tools, but the lack of good support for ephemeral environments really held back productivity.
Its great to see next-gen infrastructure tools support this, both from the article and also from infrastructure products like planetscale and railway.
This is not a great technical article. I don’t find it makes me curious. It’s entirely content marketing, zero technical content, clearly aimed at people who manage engineers but aren’t engineers. Doesn’t discuss drawback (doesn’t really define anything with enough clarity for there to be drawbacks anyway) and ends with a sales pitch.
The obvious problem here is that this approach is far too expensive for any org that isn’t a tiny startup whose production system fits on a half dozen hosts. Consider a column store DB that’s configured in a multi region manner. Is every engineer bringing that up in their preview environment? If they aren’t, it isn’t faithful to a critical production performance constraint. If they are, the company is probably paying twice the cloud costs of their competitors at least, and they don’t get the capacity planning benefits of a fixed staging deployment.
Why would you configure your column database as multiregion in every environment? You can run 99% of your tests, iteration, validation, etc, against a single shard. And even the same instance if you don't need to make any schema changes. You don't need a faithful production performance match until you want to validate performance characteristics, which often comes long after you've validated other correctness criteria that doesn't need prod-scale infra.
So once you've validated everything, you maybe stand up the prod-scale stack for less than an hour to run your at-scale tests, then tear it down again. Definitely not doubling your cost because you're not keeping anything around, and you're only using it when needed. Most places in my experience actually don't do this, and instead have a copy stood up somewhere just wasting money as you've described.
One reason is that transaction performance could be very different, even regardless to data volume.
Your points are all correct of course, but then that wouldn’t be a preview environment, would it? It would be shared database state in a database staging environment.
Which, for what it’s worth, is also how I have seen this problem managed when it came up.
As for costs, maybe true, but it could also take a lot longer than an hour to bring up a large deployment. And if every engineer is doing this for one hour a week, and you have a few dozen engineers, there’s a 2x cost increase with a tougher capacity planning problem since your load is now tied to your hiring plans.
I like the idea of ephemeral / preview environments, but I'm a bit fuzzy on the details. For a microservice application, would you have to spin up dedicated, ephemeral instances of all the components of your app? Isn't that inherently cost-prohibitive? Would some components persist, like in traditional staging environments? I'd love to see some case studies of this concept in action.
We certainly do at my company. Most microservices are rather small and easy to spin up on smaller AWS instances, especially if they're Docker based. Our AWS bill isn't that high and our preview environments don't contribute much to our costs at all. We do have a process that nags developers if their environment has been up for a while and make them confirm they still need it.
Preview environments are a lot like microservices in that they’re much more of an organisational tool than they are a technology tool. They’re great for organisations with many independent parallel streams of work owned by different people but they can be a headache for smaller teams where non-engineers are thinking about the product as a whole, not individual pieces.
I made the mistake of bringing preview environments into a small company and quickly discovered I was pushing responsibility for reconciling different streams of work onto the non-technical people.
Traditional staging environments that sit as the stage before production are problematic because they become a living thing that has to be maintained, but there’s a middle ground between preview environments and traditional staging environments: one staging environment that is ephemeral specifically for the next release, with the ability to spin up more if a situation necessitates it (e.g: a long term project that needs to be tested while other streams of work continue).
The rise of preview environments? They've been popular for years. We were using them casually 10 years ago and didn't feel like it was on the bleeding edge. Kind of a weird marketing post.
I think their target audience isn't the typical HN reader, but the rest of the industry, where multiple staging environments are still a common thing (including my current employer).
The biggest issue we had with preview/review environments was resources not being cleaned up properly after destroying the environment (along with the associated costs), but that was more an indictment of our IaC code than anything else. It also didn't eliminate staged environments for things like the non-standard DB that we were using. As long as the changes on the DB side were net-new, it wasn't really an issue, though.
You're 100% correct that the target audience of this isn't HN readers. Having worked in this space for a number of years and having written this same fluff article in 2020, this type of content is marketed directly to engineering managers and VP types who are looking for solutions to issues around their SDLC or developer productivity.
The technical implementation part can be interesting to readers and potential engineers who will use the platform and is worthwhile to have available, but this content is purely meant to cast a wide net.
I worked at a company that had built a tool for basically this. You can deploy your service changes into prod, get a special cookie and url that you could share with other users, and see prod with your changes. It was super helpful, but it did bias toward manual QA in a conference room rather than FE automated testing.
What they're describing as a "preview" environment is literally my definition of a good staging environment. The death of staging is greatly exaggerated.
Maybe I'm missing something here, but I haven't worked anywhere for years that's had a "staging" environment, as least not as this article describes it.
Everywhere I've worked either has spin up environments, a developer issued test environment or various lower testing environments where we can deploy individual features for testing and QA.
But more to the point everywhere I've worked has also combined these lower level testing environments with some kind of pre-production environment which features more data (typically a clone of prod). On this environment often a further layer of tests are ran and new features are manually reviewed one last time before being deployed to production.
Where does a "staging" environment fit in this? Unless I'm miss understanding the article seems to suggest code goes, dev -> staging -> prod, but aside from one developer role I had in the early 00s it's always been, dev -> test env -> pre-prod -> prod.
Is the article suggesting you can just do away with pre-prod environments for "preview" environments? And why would you even want to do that? As I understand it one of the features of a pre-prod environment is that is persists and isn't a clean slate each time. Any mess that can accumulate in prod can accumulate in a pre-prod environment.
I guess I'm not following what this article is advocating for.
Not always. Not every company likes continuous deployment. Sometimes staging is the server that demonstrates the feature of the day or of the week. Preprod is where all the software goes after getting a green light in staging, waiting to go in production maybe a few times per month or even per year.
> But more to the point everywhere I've worked has also combined these lower level testing environments with some kind of pre-production environment which features more data (typically a clone of prod). On this environment often a further layer of tests are ran and new features are manually reviewed one last time before being deployed to production.
fwiw it's generally not compliant to copy prod data into an alternative env without doing major scrubbing. Else your preprod env now has to have SOC compliance things like auditing of access to DBs, GDPR deletion etc.
> fwiw it's generally not compliant to copy prod data into an alternative env without doing major scrubbing. Else your preprod env now has to have SOC compliance things like auditing of access to DBs, GDPR deletion etc.
Which it absolutely should. If the idea is to go from your local environment with test data, to some staging or pre-prod environment in the cloud with (potentially) prod data. Then that cloud environment (regardless if it's called staging or pre-prod, or if it houses prod data or not), should be an exact replication of your production environment. Down to every detail.
If the data contains sensitive information about individuals (PII), you can't replicate that across all environments if you're in a regulated industry. There are ways to anonymize the data, eliminating the PII risks while preserving data utility so testing is accurate.
Anonymisation has always seemed to me to be a reduction in the probability of exposure rather than a complete guarantee of safety.
Arguably you should need compliance on both staging and prod, but you do the anonymisation to reduce the risk of exposure from (less tested) code in staging.
I’ve always viewed staging as the environment that gets prod data (maybe anonymised), but has ideally no exposure to actually affecting prod, rather than a completely fake environment.
And in 15 yrs of my experience I've never seen a company keep a compliant pre-production environment. It's pretty hard work to do with one env, let alone many.
"Staging" means "an environment where multiple unreleased pieces of work are tested together before being released to production". Pre-prod and staging are synonymous in most organisations.
A preview environment is an ephemeral environment for testing a single piece of unreleased work. Framing it according to the terms you've used, the article is advocating for replacing "dev -> test env -> pre-prod -> production" with "dev -> test env -> production".
In that case what they're advocating for seems like a really bad idea. One nice thing about having staging / pre-prod environment is that it can replicate prod architecture, content and config (for the most part anyway) which allows for things like performance checks to be ran after code has been tested for functionality in lower environments. Something I've also seen is that prod websites typically load various analytic and marketing scripts which might not be included on dev environments. Even if those scripts are not maintained by the dev team, you probably still want to be checking any code being deployed isn't going to break an analytic script because it's expecting links to have a certain class, etc.
The article is about spinning up multiple staging servers, maybe one per branch/feature and possibly leaving them there for hours or days, to be inspected at each own URL until somebody confirms that the implementation is OK. All of this is automated (a git push?)
The usual staging and/or preproduction environments would be the preview of their own branches.
It sounds nice. The only problems I can see are:
1) The cost, because each one of those instances could have its own queues, databases, connections with third party APIs, etc.
2) Seeding the database with a meaningful amount of data. I often see preproduction database built by testers month by month after a minimal seeding. Each instance here has to be seeded with all data plus what's required to demo the new feature. That must be saved before destroying the database, reused for further branches, merged with the data for other features.
There are ways to anonymize production data and make it available in preview and testing environments. My company has been doing that for customers and it has been effective. You get privacy-safe data that is still very close to production data.
Depending on the application architecture, you can spin up database containers and seed them with replicas of the anonymized data.
There are lots of organizations that have embraced trunk based or other fast-path development cycles. In general, it leads to better sanitization practices as it's too easy to break things and devs get more careful and setup their own guard rails.
As opposed to the last place I worked, where there were 2-3 month+ long branches alive in various stages of features/testing/bugfixes. THAT was painful. Give me trunk to production with a small team over that mess any day.
In my experience reliance on these kinds of "test" environments stems from a massively over estimated ROI on manual testing. (Remember, to compete you can't just have positive ROI, you must have optimal ROI or you'll be out capitalized)
There are plenty of automated and technology driven ways to assert the quality of a system, which are far more reliable, faster, widespread (across more of the system) than preview envs. A lot of testing involves creating chains of logic to make resultant assertions. If A works, and B works when A works, then B works.
And things like compile time checkers (eg types), linters, and TLA+ so I've heard, are examples of things that are better than humans, rapid once setup, and in my experience give massive ROI.
I had a manager the other day say every PR needed a manual "smoke" test before release. I challenged him why would I spend 15-60 minutes manual testing one PR when I could spend that time writing tests that check _every PR_ ? He essentially told me to just fall in line and do as I was told. (No rational rebuttal)
In general, it's super useful to be able to have non-technical members of the team play with a specific feature in development, with a simulacrum of live data, independent of other ongoing projects. There may be significant problems that need to be addressed, including data loss problems that make it so you can't just uncheck the feature flag in prod, and blocking staging (or worse, prod) until things can be rolled back creates a significant bottleneck.
We've been experimenting with a pretty reliable way to do this in-house, though. The key is that a single Kubernetes namespace per preview app iteration contains everything your preview app needs. We have a Github action, triggered by a "preview" label on a PR and any subsequent commits, that:
- builds and pushes a docker image with a dedicated tag
- within the specific preview namespace named after the PR ID, spins up and seeds with test data (in our case, a sanitized subset of production) a dedicated database statefulset
- runs the Helm chart that we use for production, but against this specific preview namespace, with this specific image tag, and overriding our normal ingress domain with one specific to this preview app
Then when the label is removed, we drop the namespace and reclaim all resources. "Rebasing" off later data, or if reviving a stale PR, is as simple as removing and re-adding the label.
We've seen customers do something very similar and we've helped them get privacy-safe data into each preview. Handwaving a lot here, but the steps are 1) anonymize production data to a destination, 2) snapshot the anonymized data, and 3) replicate the anonymized data to all of the previews.
In the environment I am currently working in we have 250 developers and 13 permanent environments. We also have 200 PRs open at any time. I am not quite sure if we could really save money on environments like the article claims.
Staging means different things to different people. To me it is a “swap” environment so you can deploy to production with no downtime and do testing prior to swapping in a new version to production.
I'm surprised there's so much resistance to this concept on HN. I'm completely unrelated to anybody remotely related to Coherence AFAIK and I've been of this same opinion ever since I got the first taste of cloud, and definitely after k8s made some of this more standardized and portable.
The points of traditional dev and qa environments are to lower the risk of unproven code, but the shared nature often causes more problems than it solves. Ad-hoc environments provides the opportunities to better utilize infrastructure, better test changes in isolation, and have one granular purpose per environment.
Another point is that this doesn't mean shared-nothing. You don't have to stand up expensive infrastructure every time if that infrastructure should be indistinguishable between environments. You can build smarts into how it's done such that multiple databases go onto the same DB server, or you reuse instances until something meaningfully changes in an isolated env you need (e.g. by looking for migration version differences). All of this is much more tenable now that infrastructure is so much more easily programmable and declarative.
And all of this works at multiple scales. Large horizontally-scalable databases don't have to be deployed at production scale on your machine. Do that in a load test env that is a last stop before prod, if you need to (e.g. if you don't trust how well you scale with # shards). Add in options from tools/concepts like teleport, and you can potentially extend this sharing to local dev environments for when things no longer fit on a dev laptop.
My experience despite having 7ish years with k8s is that people on k8s still have their dev, qa, prod clusters and dogmatically stick with that despite the fact that there are definitely better (and now simple) ways, and you really get better economics if you can binpack in the same set of machines and prioritize work appropriate to its classification.
Netlify and Vercel made preview environments a standard part of the workflow for all frontend developers. my usual soundbite on this stuff is we need to treat our envs like cattle, not pets, to make up and discard as casually as a git branch.
i think there are a number of startups (eg Railway, Brev) are all feeling their way towards this future. my one wish is that someone make a credible industry standard for environment branching that everyone can build toward, including 3rd party vendors (eg Planetscale/Neon) and open source multi-service frameworks (eg Temporal, Airbyte where i work).
Ephemerals are great, local environments are great, staging (or beta, or pre-prod) are great.
Not every product is the same, and there's no reason for a single deployment approach to the problem. These different types of deployments are just tools to be picked and chosen from to deliver the best solution for a given problem. This article made Coherence look not particularly savvy, and I'm not sure I'd trust them as an expert partner after reading this. If anyone from Coherence is reading this, you can talk to experts in deployments by talking about how complementary preview environments can help solve problems in staging while not handwaving over the necessity of pre-production environments like you did in the Traditional staging bottlenecks section.
In the environments that I worked in (2007-2015) staging was more of an environment for security and performance testing. My past QA environments were enough like production to make functionality testing there practical. The only other environments that I dealt with were more like demonstration environments, where the CEO could show off new functionality to key customers or investors. I'll highlight that containers hadn't really taken off then, and the companies I was part of were small (VC round A through D).
Coherence seems to be targeting those not using Kubernetes?
A K8s based tool is Signadot: https://www.signadot.com/
I haven't used either yet but I am looking into ways to automate ephemeral environments.
100 comments
[ 3.5 ms ] story [ 175 ms ] threadWe still have a sandbox instance, and that is hooked up to the rest of the stack.
It works pretty well if you need to iterate on something quickly that you are unable to do with a test.
They way I've set things up at my current company is this:
When a developer creates a pull request an ephemeral (preview) environment is generated.
When that code is merged it automatically gets deployed first to our staging environment and then immediately to production.
Staging gets a sanitized copy of production's data every night so both the code and the data are close mirrors of production and that's what developers (and those preview environments) integrate with when they're calling services other than their own.
If you were to get rid of staging would you stand up your entire backend every time you made a preview environment? Does that include replicating databases or would you just use seed data that may or may not resemble what's in production?
It seems like the solution they're pitching works great if you have a small team and only a few services/sites.
What I've seen done is that the staging environment is skipped, but instead concentric releases are performed on the customers. So released code goes first to a small group of users, and then widens from there if everything looks good.
If you really want to invest in extensive testing, you can do both! Hell, _and_ you can have a load testing environment as well.
Agree that it would be harder to implement with customers sharing a schema, but you can still have different release pods
And like you, we have both "preview environments" and staging and for the exact reason you've highlight. I'm rather skeptical of the company behind this product now since they don't seem to understand the difference and really overselling something that's quite common.
Before: Your developers have a shared Linux box that serves the preview environment. Your ruby app has a tiny loader in front of it that loads the app code from a specific directory based on subdomain.
After: You need to route subdomains to different kubernetes containers, and handle deploying new containers when the developer opens a PR (because it's much more work than just copying files onto a server), and you need to handle destroying old containers (because it's much more work than just deleting files on a server).
Not to mention, usually kubernetes is managed by "the ops" who don't know anything about the app. Much more difficult to interface with them when it's not "oh one server, and the developers figure it out, and if they royally fuck it up we'll rescue them".
Routing domains to a container shouldn’t be significantly different to doing it in production.
This isn't it, and I think it's pretty much an impossible problem to solve without an entirely new engineering culture around testing.
How do you duplicate the behavior of the 10 direct data dependencies of your app and the constellation of transitive microservice dependencies? Wingman, Galactus, etc. At a company with 1000+ microservices?
And how do you populate it with actual data so you can log in and perform actions? Every single microservice probably needs test data, and it would take a lot of engineering forethought to populate the entire graph.
It gets gnarly. I've been on teams at the center of it all - user accounts and login, representation of the core business entities that the rest of the company is built upon. We couldn't even solve the problem for ourselves. There were so many permutations of account states, creating a dedicated API to make test data deviates from a first class API. So how do you coordinate with downstream services to skip IDV (matching SSN), compromised password checks, etc.
Perhaps you only bring online a subset of services and fake out the rest. At what level can you fake stuff out? And how do you ensure that the test data works with whatever subset you don't fake? If a single service responds in an unexpected way, it could break or corrupt the state you're relying upon to test.
Maybe "preview" works for frontend. Backend, not so much. This is such a massive problem, and I'm only scratching the surface with my description of the kinds of issues you have to solve.
At a minimum, the product owner should be able to try out the new feature in the UI without having to deal with a staging environment.
Yeah this is the surprising thing to me - I've worked with otherwise bright people but whenever we've introduced services/microservices, it seems the default reaction is to just make a "distributed monolith" with circular dependencies and tight coupling.
I'm only half joking that 90% of my "architect" role is just point out we shouldn't have circular dependencies.
…and with added network calls!
> Microservices. grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too. seem very confusing to grug
(Source: https://grugbrain.dev/)
I'll try to simplify what I mean by that but something akin to having Services A, B, and C.
Each of A, B, and C need their own definitions to define how they need to be spun up as well as a definition on how to populate their data. That data population, say through RDS, needs to be spin up RDS instances and make them ready and available whenever someone creates a corresponding ephemeral version of a service. Toss the database afterwards.
In the above case, assuming you needed a full graph, you would spin up the three services as well as attach each service to its corresponding database. This approach works well when the number of services remains small.
Once that number starts growing, it may not be cost effective to spin everything up. Say service N is needed, but you really only need to read from it. In this case, you might have a dedicated staging environment for N and create a dependency graph that says "in my ephemeral environment, create new instances of A, B, and C with their databases, but also allow network requests to make their way to N."
That might be over simplifying, but I've been working in this space for the past few years and have been thinking about this exact issue recently
I think what made it quite easy for us is that we maintain a staging environment. Staging is constantly tested by QA simulating our workflows so non-production data is being constantly generated. When we spin up our preview environments, we use snapshots from the staging systems to seed the preview environments.
Your point on data though really gets to the crux of the problem. Deploying the services is trivial these days with k8s, Cloud Formation, Terraform, Docker, etc. but the data part is hard.
I'd love to learn more about the challenges you have with data. My company essentially provides data-anonymization-as-a-service and we integrate with ephemeral environments, but we could probably improve the integrations.
For those, like me, who had to think for a moment where they had heard that reference: https://www.youtube.com/watch?v=y8OnoxKotPQ
All of our codebases have some level of automated testing that runs when the PR is created. That could be unit tests of a single function or Playwright tests which exercise an UI flow. What those tests are dependent on the type of software and the team building it.
The preview environment links that are generated when the PR is open are given to QA, Design, and Product to take a look at. Not every one of those roles review every PR, it really depends on the task. But if their feedback is required, they'll give it at this stage.
Probably worth mentioning at this point that we have separate repos for each web application and service. It's service oriented but not microservices, services at our company encapsulate a relatively large domain and it's similar with web applications, right now each web application has its own subdomain.
So PR gets merged, the pipeline will deploy that code to staging. At this point the service owners have the option to define a set of integration tests, not every repo has them but it is an option to run them. If those tests fail we do a rollback. If those tests pass (or if they didn't have any defined) then a second deployment to production is triggered.
All of our deploys are blue-green, so no downtime except for the occasional blocking database migration in which we have to schedule downtime. User facing features on the web applications are all required to be released first behind a feature flag. For those there will be several releases behind a feature flag, then we'll flip the flag first in staging. We'll have Product and QA do a complete run through and make sure they're okay with everything, then the "real" release happen when we flip the feature flag in production (no deployment needed).
Every night we clone the production database, sanitize it (swapping out things like names, phone numbers, etc.) then point staging to that clone. Then we run an end to end regression test that goes through all of the happy path flows for all of our apps.
Quite the importance over a constant wave of changing code on development in certain projects.
Staging seems like a more code locked stable area for all previews that can be polished via quality practices and keen eyes.
Now it seems a little fleeting to me as well to have the shock of previews enter into this instead.
Let the paint dry adequately before taking it outside.
The typical workflow is to anonymize production data, snapshot it, and make it available to preview and testing environments as replicas. It's pretty fast.
Do you have any pointers?
Its great to see next-gen infrastructure tools support this, both from the article and also from infrastructure products like planetscale and railway.
The obvious problem here is that this approach is far too expensive for any org that isn’t a tiny startup whose production system fits on a half dozen hosts. Consider a column store DB that’s configured in a multi region manner. Is every engineer bringing that up in their preview environment? If they aren’t, it isn’t faithful to a critical production performance constraint. If they are, the company is probably paying twice the cloud costs of their competitors at least, and they don’t get the capacity planning benefits of a fixed staging deployment.
So once you've validated everything, you maybe stand up the prod-scale stack for less than an hour to run your at-scale tests, then tear it down again. Definitely not doubling your cost because you're not keeping anything around, and you're only using it when needed. Most places in my experience actually don't do this, and instead have a copy stood up somewhere just wasting money as you've described.
Your points are all correct of course, but then that wouldn’t be a preview environment, would it? It would be shared database state in a database staging environment.
Which, for what it’s worth, is also how I have seen this problem managed when it came up.
As for costs, maybe true, but it could also take a lot longer than an hour to bring up a large deployment. And if every engineer is doing this for one hour a week, and you have a few dozen engineers, there’s a 2x cost increase with a tougher capacity planning problem since your load is now tied to your hiring plans.
I made the mistake of bringing preview environments into a small company and quickly discovered I was pushing responsibility for reconciling different streams of work onto the non-technical people.
Traditional staging environments that sit as the stage before production are problematic because they become a living thing that has to be maintained, but there’s a middle ground between preview environments and traditional staging environments: one staging environment that is ephemeral specifically for the next release, with the ability to spin up more if a situation necessitates it (e.g: a long term project that needs to be tested while other streams of work continue).
The biggest issue we had with preview/review environments was resources not being cleaned up properly after destroying the environment (along with the associated costs), but that was more an indictment of our IaC code than anything else. It also didn't eliminate staged environments for things like the non-standard DB that we were using. As long as the changes on the DB side were net-new, it wasn't really an issue, though.
The technical implementation part can be interesting to readers and potential engineers who will use the platform and is worthwhile to have available, but this content is purely meant to cast a wide net.
Everywhere I've worked either has spin up environments, a developer issued test environment or various lower testing environments where we can deploy individual features for testing and QA.
But more to the point everywhere I've worked has also combined these lower level testing environments with some kind of pre-production environment which features more data (typically a clone of prod). On this environment often a further layer of tests are ran and new features are manually reviewed one last time before being deployed to production.
Where does a "staging" environment fit in this? Unless I'm miss understanding the article seems to suggest code goes, dev -> staging -> prod, but aside from one developer role I had in the early 00s it's always been, dev -> test env -> pre-prod -> prod.
Is the article suggesting you can just do away with pre-prod environments for "preview" environments? And why would you even want to do that? As I understand it one of the features of a pre-prod environment is that is persists and isn't a clean slate each time. Any mess that can accumulate in prod can accumulate in a pre-prod environment.
I guess I'm not following what this article is advocating for.
Sounds like staging to me...
Which it absolutely should. If the idea is to go from your local environment with test data, to some staging or pre-prod environment in the cloud with (potentially) prod data. Then that cloud environment (regardless if it's called staging or pre-prod, or if it houses prod data or not), should be an exact replication of your production environment. Down to every detail.
Arguably you should need compliance on both staging and prod, but you do the anonymisation to reduce the risk of exposure from (less tested) code in staging.
I’ve always viewed staging as the environment that gets prod data (maybe anonymised), but has ideally no exposure to actually affecting prod, rather than a completely fake environment.
> dev -> test env -> pre-prod -> prod
You're just using a different word and adding one additional environment. You swapped out the word staging and replaced it with pre-prod.
A preview environment is an ephemeral environment for testing a single piece of unreleased work. Framing it according to the terms you've used, the article is advocating for replacing "dev -> test env -> pre-prod -> production" with "dev -> test env -> production".
In that case what they're advocating for seems like a really bad idea. One nice thing about having staging / pre-prod environment is that it can replicate prod architecture, content and config (for the most part anyway) which allows for things like performance checks to be ran after code has been tested for functionality in lower environments. Something I've also seen is that prod websites typically load various analytic and marketing scripts which might not be included on dev environments. Even if those scripts are not maintained by the dev team, you probably still want to be checking any code being deployed isn't going to break an analytic script because it's expecting links to have a certain class, etc.
The usual staging and/or preproduction environments would be the preview of their own branches.
It sounds nice. The only problems I can see are:
1) The cost, because each one of those instances could have its own queues, databases, connections with third party APIs, etc.
2) Seeding the database with a meaningful amount of data. I often see preproduction database built by testers month by month after a minimal seeding. Each instance here has to be seeded with all data plus what's required to demo the new feature. That must be saved before destroying the database, reused for further branches, merged with the data for other features.
Depending on the application architecture, you can spin up database containers and seed them with replicas of the anonymized data.
it's the place where more biz-facing people can tell you the color's off by like half a shade just before a big release :)
Do not walk; run away from that trashcan bonfire.
As opposed to the last place I worked, where there were 2-3 month+ long branches alive in various stages of features/testing/bugfixes. THAT was painful. Give me trunk to production with a small team over that mess any day.
> Give me trunk to production
Okay, but three times a day?
Three test builds off the trunk in a day, I can swallow.
There are plenty of automated and technology driven ways to assert the quality of a system, which are far more reliable, faster, widespread (across more of the system) than preview envs. A lot of testing involves creating chains of logic to make resultant assertions. If A works, and B works when A works, then B works.
And things like compile time checkers (eg types), linters, and TLA+ so I've heard, are examples of things that are better than humans, rapid once setup, and in my experience give massive ROI.
I had a manager the other day say every PR needed a manual "smoke" test before release. I challenged him why would I spend 15-60 minutes manual testing one PR when I could spend that time writing tests that check _every PR_ ? He essentially told me to just fall in line and do as I was told. (No rational rebuttal)
We've been experimenting with a pretty reliable way to do this in-house, though. The key is that a single Kubernetes namespace per preview app iteration contains everything your preview app needs. We have a Github action, triggered by a "preview" label on a PR and any subsequent commits, that:
- builds and pushes a docker image with a dedicated tag
- within the specific preview namespace named after the PR ID, spins up and seeds with test data (in our case, a sanitized subset of production) a dedicated database statefulset
- runs the Helm chart that we use for production, but against this specific preview namespace, with this specific image tag, and overriding our normal ingress domain with one specific to this preview app
Then when the label is removed, we drop the namespace and reclaim all resources. "Rebasing" off later data, or if reviving a stale PR, is as simple as removing and re-adding the label.
How about mobile apps?
How about 3rd party integrations? And integration with legacy systems?
How about systems where there's non-trivial data that needs to be set up?
The points of traditional dev and qa environments are to lower the risk of unproven code, but the shared nature often causes more problems than it solves. Ad-hoc environments provides the opportunities to better utilize infrastructure, better test changes in isolation, and have one granular purpose per environment.
Another point is that this doesn't mean shared-nothing. You don't have to stand up expensive infrastructure every time if that infrastructure should be indistinguishable between environments. You can build smarts into how it's done such that multiple databases go onto the same DB server, or you reuse instances until something meaningfully changes in an isolated env you need (e.g. by looking for migration version differences). All of this is much more tenable now that infrastructure is so much more easily programmable and declarative.
And all of this works at multiple scales. Large horizontally-scalable databases don't have to be deployed at production scale on your machine. Do that in a load test env that is a last stop before prod, if you need to (e.g. if you don't trust how well you scale with # shards). Add in options from tools/concepts like teleport, and you can potentially extend this sharing to local dev environments for when things no longer fit on a dev laptop.
My experience despite having 7ish years with k8s is that people on k8s still have their dev, qa, prod clusters and dogmatically stick with that despite the fact that there are definitely better (and now simple) ways, and you really get better economics if you can binpack in the same set of machines and prioritize work appropriate to its classification.
Netlify and Vercel made preview environments a standard part of the workflow for all frontend developers. my usual soundbite on this stuff is we need to treat our envs like cattle, not pets, to make up and discard as casually as a git branch.
i think there are a number of startups (eg Railway, Brev) are all feeling their way towards this future. my one wish is that someone make a credible industry standard for environment branching that everyone can build toward, including 3rd party vendors (eg Planetscale/Neon) and open source multi-service frameworks (eg Temporal, Airbyte where i work).
You can take my localhost from my cold dead hands :p
I could get behind the death of staging tho.
Not every product is the same, and there's no reason for a single deployment approach to the problem. These different types of deployments are just tools to be picked and chosen from to deliver the best solution for a given problem. This article made Coherence look not particularly savvy, and I'm not sure I'd trust them as an expert partner after reading this. If anyone from Coherence is reading this, you can talk to experts in deployments by talking about how complementary preview environments can help solve problems in staging while not handwaving over the necessity of pre-production environments like you did in the Traditional staging bottlenecks section.
...there you go.