> When I test my code, I want to know that it works in production. Staging, you can’t give me that.
This sounds like more of a problem with the author's staging environment than a general issue with the concept of staging environments.
The idea is staging and production should mirror eachother as closely as possible. If your staging environment is wildly different, of course it will be less useful. But it's also your responsibility to maintain a useful-enough staging environment. Why not just say, "it's too much effort for me to maintain a separate, identical environment" rather than dismiss the concept entirely?
It sounds like a problem with the author, one probably drilled into their head by management.
The longer you are here, the more weird shit you see, and the definition of “know” gets a lot murkier.
You aren’t going to “know” if it works in production, even if you put it into production. Which is something that only makes sense about the fourth time you encounter a piece of shipping code that is so fundamentally rotten that you are not sure how it ever came up with the right answer.
You don’t know if the whole car is working. You only know it’s moving the direction you wanted it to move and it hasn’t exploded, made horrible sounds, or smells. So it’s mostly working.
Every tool and process should be working to make you less afraid of trying things (like deploying to prod). Otherwise they’re just helping you build a bureaucracy instead of software.
There will always be known differences between staging and production, e.g. service names, networking.
The point about staging is that it lets you run risk-free tests on a set of things, in a way that give you confidence about the things that are the same as production. The fact that a set of things works in staging is never an absolute guarantee that it will work in production.
If you are in the position of needing to ship releases rapidly, and you can't run meaningful tests in the available time, then no, it's not useful.
If you spend $100K on a staging environment and it doesn't give you confidence that it won't fail on production, the problem is Your Staging, not The Staging.
If we have something fail in production, the first question we ask ourselves is: Why didn't we catch this in staging? 90% of the time, the answer is: "We didn't test if properly but thought we did."
Our rate of patching for defect is rare.
As far as the cost, our staging is run in hardware retired from production, with some upgrades (mostly RAM and SSDs). I'm not sure we've spent the authors $100K over 2 decades, let alone in a single year. But YMMV.
When AWS releases a new instance generation, everyone flocks to them. For previous generations theres usually ample capacity to run as spot instances uninterrupted over long periods of time, giving your staging environment significant savings.
I'd call that a hand me down.
Except the top level comment was doubting they ever spent 100k total in the history of their staging system. That’s not going to be true for AWS even with the nosebleed seats, is it?
And are you really going to run your entire staging environment in spot instances? What happens on peak days?
I didn't mean to imply that the author couldn't have spent $100K. There are clearly situations where an expensive staging environment would be needed. I was just saying that we've been able to do our staging for a fraction of that price, and had good results.
A year and a half ago, IBM did a study and found that most companies are only 20 percent to cloud transformation. The reality is that there is a ridiculous amount of private datacenters and iron around.
This makes it sound like 100% is the goal. Some companies don't get added value by going to the cloud. The cloud isn't cheap for example (among many others) for storage heavy workloads. And sometimes it's also a regulatory thing.
Thank you for saying this, I am more than frustrated by that mentality whenever I go to conferences or message boards. There’s a big world out there and most of does not resemble the practices of internet companies. It’s this fallacy of inevitability for cloud.
Once I visited a customer site who were having a performance problem. They ran everything on AWS. The day before I arrived on-site, they spun up a brand new production-sized test environment just for me to debug their issue in. I spent a day or two on-site working out what was causing their performance problem. When I left, they deleted that environment. They only paid for its existence for 2-3 days. So with cloud you can get away from fixed always-on staging environments, and move to "staging environments on demand".
(My recollection was they loved everything about AWS except the price tag, although creating a personal staging environment for me to debug their issue in would have only made a marginal difference to that total cost.)
This article is too simplistic to take seriously. Basically it says: "staging is expensive (but $100k !?) and is not the same as prod because data (how many tests fail because of this?), so I'll use feature flags and test in prod".
Data consistency between prod and stage is also simple to solve for use cases where it’s not an issue. For example, we have a job which restores each day’s prod backup to staging.
This article has reduced the argument for staging, calling it unnecessary. Its upto the devops to use staging effectively. It you have a waterflow like deployment and useless e2e tests running in staging, then it wont be helpful. Instead, if you use staging as another prod (Blue Green deployment cough cough.. ) then, it would be a lot more useful.
Than blaming staging env as a whole, I would blame archaic devops practices for SaaS.
The Knight Capital bankruptcy was caused in part by an error related to a feature flag, Which cost them ~US$440 million in trading errors in ~45 minutes.
I would actually venture to say that the way they are usually perused, they increase the risk!
Common approach is to just plaster the codebase with "if feature_flag(x)", so you complicate dozens of files/modules/functions, make them harder to reason about and debug. And cleaning it up is commonly a mess.
Another approach is to duplicate code which frequently leaves one branch untested.
FWIW, I do like feature flags, but as a solution for a different problem: long-lived feature branches (forks) of the master. And even then the complexity they introduce is sometimes a burden, but overall it's worth it over alternative approaches.
And sure, you can start implementing stuff iteratively, in small chunks and perusing feature flags to hide bits of unfinished user flow, but the benefits you are getting there are from the former (iterative, small changes), rather than from feature flags. It is still no substitution for testing in a prod-like environment, at the very least that you are not breaking any of the integration places which are not covered by tests (and integration and system tests are slow, which always puts a boundary on how many you can have).
>FWIW, I do like feature flags, but as a solution for a different problem: long-lived feature branches (forks) of the master
But isn't this case what staging originally tried to solve? "this big feature isn't ready for production but we need to regression test in a setup as similar to production as possible" ?
Not that I know: if you needed a "staging" for each of the feature branches, the complexity and cost grew, but problems remained (i.e. merging trunk and a branch back and forth).
For the last 15 years I've been at teams using a "staging", it was simply a single production-like environment (well, my first project was exactly like production on slightly less beefy hardware with sanitized production DB import nightly) that gets the "trunk" (or "main") deployed to automatically (on my current team, it can be used to run non-production code as part of the deployment process which you bail out of).
I am pretty sure there is no one true definition of staging, so it's a spectrum rather than an exact way to do things.
I'm talking about the prehistoric time before git and feature branches. Where all the developers hacked away for months on That New Feature and all code just went into trunk, cause that's where the development is. Back then staging was invaluable to get the confidence to do that (bi?)annual major release.
Sure, that's the time we learned why feature branches and not releasing often sucks :) And yes, you sometimes had "staging" track your trunk, when you would be missing a staging environment for production where you'd want to test critical fixes, so ultimately, you needed both then.
Staging has always been about QA before deployment in production. For tools you buy, one might just end up forgoing staging where you know it must work. Though, what to do if an update bricks the system? For any non-trivial and complex software, you still end up with having at least one test-environment though.
Feature flags is about multiple branching. Staging doesn't really give you that in full, just by hacking the environment. It's then that you end up with environments that don't properly reflect future production and start missing stuff.
The author sounds like someone familiar with testing, that needs to learn about CI/CD. Sure, for trivial front end stuff, you can often get away with doing it in production. But without synchronization between environments, you end up in trouble and take on more risk, just for the luxury of treating your paying customers as beta-testers.
It's a valid opinion depending on what one is working on, but I bet it doesn't apply for most outside unimportant side-projects.
I suspect it’s the same quality of thought that can’t fathom that a test could have a bug.
Implementing a flag takes fewer lines of code than the feature, and so while the odds are very high that the bug will be within a feature, there is a non zero chance that the bug is in the flag.
A bug might break a new feature. It might break the old behavior. It may turn the system into a latch (you can launch it darkly, turn it on, and then not be able to turn it off).
And then you can break the system while retiring the flag and making the new path permanent.
> I suspect it’s the same quality of thought that can’t fathom that a test could have a bug.
I have to admit that I often find it funny when some talk about unit tests being the end of all. No manual QA needed, no staging needed, tests passed and everything is good to roll to production.
Unit tests are important, no question but it’s still a piece of code which could have bugs or not enough scenarios covered.
Yep, your feature flag is probably not going to prevent data from getting into an inconsistent state. Maybe it's possible in a system with very loosely related entities but for most non-trivial systems this isn't true.
> The code that that was updated repurposed an old flag that was used to activate the Power Peg functionality
A practice you can adopt which can prevent this: suppose you use JIRA (the practice works the same for any such system, I'm just using JIRA as an example), and for every new feature you must have a JIRA issue, e.g. FEAT-1234. Now you have the rule that the feature flag name must embed both (1) the year and quarter the feature was merged and (2) the JIRA issue. So as part of FEAT-1234 I create feature flag "2020q3-FEAT-1234-split-big-trades". And then you have a custom field in JIRA called "feature flag name", and you record "2020q3-FEAT-1234-split-big-trades" in that field for FEAT-1234. And then you need to have this as a standard check in processes such as code review and release management. It might even be useful to have a report which compares JIRA to the feature flag system to check this rule is being followed. If they'd adopted this practice, that particular bug would likely have been avoided. Adding a new feature, you would not be allowed to reuse an old feature flag from years ago, for the year and quarter and the JIRA ticket would all be different.
If userdata is whats failing you in staging there's simple ways to go about pulling in a sanitised version of prod data to making it more prod like.
We had some issues with Akamai WAF blocking things in PROD which only got caught when pushing to PROD so we added staging to Akamai now its closer to 1:1!
For example oracle e-business suite runs completely different with 900GB of prod data vs 100gb of staging data.
ASP.Net applications are slower talking to databases with 1600gb of MSSQL data in prod vs 40gb in staging vs 3gb in test etc.
DR performance different again because its scaled down version of PROD but has the same amount of data and traffic loads when it takes full production load.
Due to privacy and GDPR laws, this is bad practice and outlawed in many parts of the world. We used to mirror from production, but you can't really do that with user data without exposing some personal information. Even a timestamp from a transaction can be enough to identify a real person.
I know we're mentioning the $100k number, but I'm genuinely curious: how do you spend $100k on staging? I currently spend $75k/year on production, $3k/year on staging, and around $2k/year for a full cloud environment per developer that wants one. For me, $100k for staging would mean >200k monthly active users on staging.
You want staging to be as close to production as possible, but obviously not exactly he same size.
If you spend 1M on production, you might have a staging environment 1/10th of that size. Say 10 machines in a cluster instead of 100. But some technologies also have minimum implementation cost requirements per tier of usage. Like if you have a multiple data centers in production, you will need to test that in staging. You can’t just have 1 machine in a cluster, so your staging environment needs at least 3 to make a quorum. Therefore, to have 5 data centers in staging, you now need 15 machines... which is more than the base of 1/10th of production.
Issues like that can balloon staging costs if you try to properly implement it.
it depends however whether you want to test the code or the infrastructure
my preproduction uses a three machines setup so it can catch clustering problems, but the code itself can run just as well on a single machine interacting with a local single instance db, and devs run it like that.
Honestly with the right architecture you shouldn't even need 10 machines in a cluster, 2 should do. After all, the whole point of clusters/load-balancing etc. is that 2 machines behind your LB should function the same as 1000 machines behind your LB, just the latter gives you a lot more throughput.
Depends on the business you’re in. Sometimes an “environment” is a rack (or 2) of servers and that might be 100k-400k/rack before space & power & bandwidth.
having a database schema at all is the root of that problem. i wish people who are writing systems from scratch would reflect on whether they actually need an RDBMS or RDBMS-like schemas.
Everything I've worked on in the last few years would have been in far, far better shape if they had just started with a no-schema data management system like Bigtable and just adapted their backend code around its limitations.
Imagining you are safe because you are smoke-checking DML statements in staging is just begging for a week-long outage.
I can easily imagine majority of the developers trying to solve trivial serialization and locking issues on their own (which RDBMS provide) and failing spectacularly!
Heck, I don't need to imagine, I have seen it time and time again.
Curious about this line of thought because for most apps I reach for a RDBMS because I want strong data integrity and a data model that’s easy to reason about. Maybe you tend to work in different domains. Can you give an example of one of the projects better served by Bigtable?
Everything? When the topic is operability it is easily shown that a database schema that you have to change atomically and the change becomes visible to all clients instantly is very incompatible with the ideas of rolling releases, rollback safety, zero maintenance windows, etc. If you can see that centralizing the schema doesn’t work well then you can also see that the schema doesn’t belong in the database at all, it belongs in the clients where it can be rolled forward and backward safely if you never try to change a field, only add them, which is a best practice anyway.
After you’re onboard with this philosophy then I’m ready to sell you the fact that you don’t need joins, indexes, or transactions. If a giant complicated service like gmail doesn’t need any of those things then very likely that your thing doesn’t need them either.
the intermediate layer between clients and the database is responsible to negotiate a protocol version, you shouldn't expose the storage model to the api layer, then schema versioning becomes a non problem.
I don't think you can hide schemas that way in practice. At Dropbox I saw that they had an abstract storage service in front of MySQL but every aspect of the underlying schema leaked out through the API in some way.
first: I didn't say rdbms, I said database. that covers nosql as well.
second: even if you don't have an explicit schema in the database, you still have to have a way to know which data goes in which column. just because it isn't written in a definition language doesn't mean the concept doesn't exist and doesn't require to be maintained as the application domain evolves.
This is a big reason why I like Datomic so much. I get the benefits of a schema, but the schema is basically append-only, so it's very easy to reason about. Updating the schema is instant.
The last time I tried a project with a no-schema DB, it was a huge PITA. It sucks having a function fail on one single document because it had a number instead of a string. I love flexibility, but it is very valuable to have the right constraints in the right places. I quickly learned that a schema is just that.
I don’t see how you’d have a number instead of a spring in a document in a typed language. Do you just directly dump user supplied data in your database without parsing/processing it? Otherwise, how is this not caught when it enters your system?
It is caught by the schema and never stored in the first place.
Or you can solve it for every column/field in your system manually, and probably introduce subtle bugs and inconsistencies along the way.
I'm for schemas, but not necessarily RDBMS for everything.
Most document databases have JSON editors, which are very useful at times but don't have any schema checks.
By "typed language" I suppose you mean "statically typed language". This isn't relevant because A) JSON types exist and B) I've never had a problem with unexpected types from user data in a dynamic language, not even in PHP.
> Otherwise, how is this not caught when it enters your system?
It should be caught, and it's very important that it be caught all of the time and not have any issues due to bugs or version mismatches. That's why the schema should live in the DB.
And everything I have worked on would have been in far, far better shape if they had just reached for RDBMS instead of following the NoSQL hype and reimplementing what the RDBMS gives you in each backend service.
Well, it's nice that the author is considering the costs compared to the benefits drawn from doing something "just because".
I don't think the article does a convincing job of arguing that releasing behind feature flags would be much easier to maintain than a staging deployment, though.
For making this consideration, it'd be worth seeing why the costs can't be lowered, why the effectiveness of staging can't be increased, or what cases might be difficult to handle with feature flags.
There's the production cluster then there's the development environments one of which happens to be the staging branch but it is not different in configuration from any web-1234 feature or bugfix branch. It doesn't add anything in costs as I doubt we could use a smaller machine (or is it two machines? can't remember but doesn't matter) than the current one just because we allow one less env... I thought this is what virtualization and all that was about?? And I absolutely can't see how could we drop the environment for feature branches -- not only QA is necessary but also if you want to talk to a stakeholder, how are you going to?
Staging is so important to me that I moved our startup from Paddle to Stripe as payment processor because Paddle did not (and I think still does not) offer any sort "dev" mode that I could use to test our payment integrations in dev/stag.
Literally their solution was "just make a product that costs $0 and test it against our production service".
The real kicker for me is when there’s no staging environment and production is kittens-not-cattle, and yet there’s no strong way to wonder how this situation came to be.
$100k for a staging environment is definitely worth it IMO. It sounds like this person hasn't setup a good one, but in my experience a good $100k staging environment will catch more than $100k worth of bugs and downtime over a year. One hour of downtime can be worth more than that.
I've not had a staging environment for a good 6 years, if not more.
Thats not to say that a staging env is bad, they can be very useful, if setup properly. It also might be the case that you really can't test in prod. (stuff that provides safety)
I am very keen on feature flags/magic routers. They have the advantage that you can test the inputs of the system with _real_ data. But that can also be a drawback.
Be it an infra change or application. It should always be like:
Staging -> Canary -> Production
If you invest time in making the above pipeline automated it will cause less outages.
P.S we use feature flags but it won't solve many other challenges around infra and application changes.
Some examples:
- Changing from Classic ELBs to ALBs (We care about latencies so ALBs served 1% traffic in the start)
- Zero downtime single mysql master upgrade. (ProxySql)
- New k8s controllers.
- New metrics adapters.
Staging should be like a playground to test all edge cases before you go to canary and then to prod.
So to "solve" her staging problem, she pushes the complexity of multiple feature flags and branching unto developers? Sounds a bit short-sighted given how quickly those complexities is known to multiply. Who is going to pay down those growing technical debts?
As others have said, staging is useful for certain types of changes that may be high-risk and not time-sensitive. For other changes that are time-sensitive and are low risk (like changing some incorrect/offensive copy), being able to bypass staging is also useful. I don't mean to disrespect the author but it feels like the article was designed to take an extreme stance for views. It even reads like clickbait. I bet this article leaves a bunch of people believing that using an intermediate testing environment can never be useful.
I set up a staging instance for my previous startup, then never used it. I don't really understand what the value is unless you have dedicated QA staff that run some manual tests against staging. If the tests are automated, they should run in a dev environment. If you don't have QA, just test in prod. This seems to work well for the largest tech companies, who (in)famously don't really use branches and ship directly to prod with feature flags.
The answer obviously varies wildly for every product and company.
Not everything can be feature flagged.
Not all tests can be automated.
Not all features can be safely tested in production.
You can't always roll-back code.
Scenarios where I've found staging environments to be invaluable:
- Features that make wide-scale changes to the application
- Changes that require user acceptance testing. Management/Customers might ask for Feature X, you have designers mock up what it'll look like, you might even build prototypes of it... but until they see it and play with it, they might not fully realise that what they actually asked for isn't what they actually intended.
- Integrating changes from multiple developers and seeing how they play together
- Testing that your feature flagging actually works
- Testing how the system responds to dumping large amounts of data/requests at it - a half-day outage in production might easily exceed the cost of a whole year of running a staging environment.
- Testing how the system behaves after data migrations/changes (things where there's no roll-back without losing data)
- Testing integrations with external systems.
- Detecting poor performance, memory leaks, deadlocks, and other stuff that isn't possible on a dev's machine
Testing code on your machine is fine as a smoke test, but your laptop probably has multiple times the capacity of a single node on production, for a single user. So your change that results in 1GB/sec of full table scans on the database might not even be noticed because hey, you have an NVME drive. Your tests also run for maybe a few minutes, but that staging instance might be up for days or hours, and deals with hundreds
I've picked up numerous issues because someone (I'm including myself here) committed some code that resulted in Staging going down or behaving poorly.
I've even overheard conversations that were along the lines of "Yeah, this feature isn't working right at the moment in testing, but that's because staging is running slow today It's fine, we can push this out to production and it'll be better there." if it wern't for me overhearing that, it'd have resulted in a multi-GB video file being sent for every visitor to the site, costing a fortune in CDN charges and still running like crap because it was a 30 second video.
Why need a development code base when you can just directly edit and save to prod? I remember opening tomcat prod directories, edit the html and saving it live in prod years back. Maybe that’s where we are headed.
Hi. We open sourced our feature flagging product https://github.com/BulletTrainHQ not long ago and would love any feedback from HN.
I still have concerns with deploying production upgrades behind flags due to the inherent issues surrounding things like database schema mismatches, terraform/infra upgrades that are not compatible bidirectionally etc. Has anyone worked on ways to manage these with feature flags?
It is possible but you need to design versioning in message and storage schemas from the start. It's a non-trivial thing, but can be done. I'm not aware of any general solution available, so need a bespoken design effort. You need to map out requirements for versioning beforehand, and the complexity and future limitations will follow exactly from requirements.
If instead of converging to versioning, you want full-on branching that never merge, this sounds like very chaotic design. You'd then maybe want to design everything around feature isolation (see: UNIX philosophy), but comes with own caveats around new requirements and cross-cutting concerns.
If staging is broken, the CI should break too due to the impossibility to deploy.
I don't get the "nobody will care to fix staging"; unless nobody cares of releasing in prod.
I suspect this DevOps is working in loneliness, without enough communication with devs. Devs ought to be educated, it's part of DevOps job.
Personally, I have worked in an environment where staging was a pain, full of cobwebs and issues and data mismatches. I sympathize because I know how painful it can be.
But if you respect your end-users, you cannot delegate the duties of a staging environment to them.
The amount of bugs we catch in staging is absolutely staggering.
From services that suddenly require new ports to be open through VPNs (development happens without intercontinental VPN’s of course) — to commandline options not correctly propagating to our hardened variant of configs — to even latency or bandwidth issues and timeouts due to assumptions.
There’s a difficult balance to strike when it comes to ease of development and not letting issues (security or issues with distributed systems) propagate to the customer.
Staging is also the place where you can safely let the chaos monkey reign supreme without fear of losing customer data! It’s the last place to test the “fit” of your system.
Staging has a place, and if puff pieces like this cause the company as a whole to downplay funding for it then I’m going to be mighty displeased.
Good catch, i didn’t notice that. I was about to write something along the lines of, « wait, there are cases where you absolutely need a staging environment. not everyone is a social network of cat videos », but now i know this piece isn’t worth critizing. It should be flagged.
This is typical. There is no recipe for all. For those that this is essential, they accept the costs and do it anyway.
Those who do this by fashion... I hope they earn enough ;)
99 comments
[ 5.2 ms ] story [ 152 ms ] threadThis sounds like more of a problem with the author's staging environment than a general issue with the concept of staging environments.
The idea is staging and production should mirror eachother as closely as possible. If your staging environment is wildly different, of course it will be less useful. But it's also your responsibility to maintain a useful-enough staging environment. Why not just say, "it's too much effort for me to maintain a separate, identical environment" rather than dismiss the concept entirely?
The longer you are here, the more weird shit you see, and the definition of “know” gets a lot murkier.
You aren’t going to “know” if it works in production, even if you put it into production. Which is something that only makes sense about the fourth time you encounter a piece of shipping code that is so fundamentally rotten that you are not sure how it ever came up with the right answer.
You don’t know if the whole car is working. You only know it’s moving the direction you wanted it to move and it hasn’t exploded, made horrible sounds, or smells. So it’s mostly working.
Every tool and process should be working to make you less afraid of trying things (like deploying to prod). Otherwise they’re just helping you build a bureaucracy instead of software.
There will always be known differences between staging and production, e.g. service names, networking.
The point about staging is that it lets you run risk-free tests on a set of things, in a way that give you confidence about the things that are the same as production. The fact that a set of things works in staging is never an absolute guarantee that it will work in production.
If you are in the position of needing to ship releases rapidly, and you can't run meaningful tests in the available time, then no, it's not useful.
If we have something fail in production, the first question we ask ourselves is: Why didn't we catch this in staging? 90% of the time, the answer is: "We didn't test if properly but thought we did."
Our rate of patching for defect is rare.
As far as the cost, our staging is run in hardware retired from production, with some upgrades (mostly RAM and SSDs). I'm not sure we've spent the authors $100K over 2 decades, let alone in a single year. But YMMV.
And are you really going to run your entire staging environment in spot instances? What happens on peak days?
https://www.ibm.com/blogs/cloud-computing/2019/03/05/20-perc...
(My recollection was they loved everything about AWS except the price tag, although creating a personal staging environment for me to debug their issue in would have only made a marginal difference to that total cost.)
Than blaming staging env as a whole, I would blame archaic devops practices for SaaS.
That's not true at all. It is a common misconception that feature flags have no risk.
Common approach is to just plaster the codebase with "if feature_flag(x)", so you complicate dozens of files/modules/functions, make them harder to reason about and debug. And cleaning it up is commonly a mess.
Another approach is to duplicate code which frequently leaves one branch untested.
FWIW, I do like feature flags, but as a solution for a different problem: long-lived feature branches (forks) of the master. And even then the complexity they introduce is sometimes a burden, but overall it's worth it over alternative approaches.
And sure, you can start implementing stuff iteratively, in small chunks and perusing feature flags to hide bits of unfinished user flow, but the benefits you are getting there are from the former (iterative, small changes), rather than from feature flags. It is still no substitution for testing in a prod-like environment, at the very least that you are not breaking any of the integration places which are not covered by tests (and integration and system tests are slow, which always puts a boundary on how many you can have).
But isn't this case what staging originally tried to solve? "this big feature isn't ready for production but we need to regression test in a setup as similar to production as possible" ?
For the last 15 years I've been at teams using a "staging", it was simply a single production-like environment (well, my first project was exactly like production on slightly less beefy hardware with sanitized production DB import nightly) that gets the "trunk" (or "main") deployed to automatically (on my current team, it can be used to run non-production code as part of the deployment process which you bail out of).
I am pretty sure there is no one true definition of staging, so it's a spectrum rather than an exact way to do things.
Feature flags is about multiple branching. Staging doesn't really give you that in full, just by hacking the environment. It's then that you end up with environments that don't properly reflect future production and start missing stuff.
The author sounds like someone familiar with testing, that needs to learn about CI/CD. Sure, for trivial front end stuff, you can often get away with doing it in production. But without synchronization between environments, you end up in trouble and take on more risk, just for the luxury of treating your paying customers as beta-testers.
It's a valid opinion depending on what one is working on, but I bet it doesn't apply for most outside unimportant side-projects.
Implementing a flag takes fewer lines of code than the feature, and so while the odds are very high that the bug will be within a feature, there is a non zero chance that the bug is in the flag.
A bug might break a new feature. It might break the old behavior. It may turn the system into a latch (you can launch it darkly, turn it on, and then not be able to turn it off).
And then you can break the system while retiring the flag and making the new path permanent.
I have to admit that I often find it funny when some talk about unit tests being the end of all. No manual QA needed, no staging needed, tests passed and everything is good to roll to production.
Unit tests are important, no question but it’s still a piece of code which could have bugs or not enough scenarios covered.
https://dougseven.com/2014/04/17/knightmare-a-devops-caution...
A practice you can adopt which can prevent this: suppose you use JIRA (the practice works the same for any such system, I'm just using JIRA as an example), and for every new feature you must have a JIRA issue, e.g. FEAT-1234. Now you have the rule that the feature flag name must embed both (1) the year and quarter the feature was merged and (2) the JIRA issue. So as part of FEAT-1234 I create feature flag "2020q3-FEAT-1234-split-big-trades". And then you have a custom field in JIRA called "feature flag name", and you record "2020q3-FEAT-1234-split-big-trades" in that field for FEAT-1234. And then you need to have this as a standard check in processes such as code review and release management. It might even be useful to have a report which compares JIRA to the feature flag system to check this rule is being followed. If they'd adopted this practice, that particular bug would likely have been avoided. Adding a new feature, you would not be allowed to reuse an old feature flag from years ago, for the year and quarter and the JIRA ticket would all be different.
We had some issues with Akamai WAF blocking things in PROD which only got caught when pushing to PROD so we added staging to Akamai now its closer to 1:1!
For example oracle e-business suite runs completely different with 900GB of prod data vs 100gb of staging data.
ASP.Net applications are slower talking to databases with 1600gb of MSSQL data in prod vs 40gb in staging vs 3gb in test etc.
DR performance different again because its scaled down version of PROD but has the same amount of data and traffic loads when it takes full production load.
Blog writer is taking the piss.
If you spend 1M on production, you might have a staging environment 1/10th of that size. Say 10 machines in a cluster instead of 100. But some technologies also have minimum implementation cost requirements per tier of usage. Like if you have a multiple data centers in production, you will need to test that in staging. You can’t just have 1 machine in a cluster, so your staging environment needs at least 3 to make a quorum. Therefore, to have 5 data centers in staging, you now need 15 machines... which is more than the base of 1/10th of production.
Issues like that can balloon staging costs if you try to properly implement it.
my preproduction uses a three machines setup so it can catch clustering problems, but the code itself can run just as well on a single machine interacting with a local single instance db, and devs run it like that.
Everything I've worked on in the last few years would have been in far, far better shape if they had just started with a no-schema data management system like Bigtable and just adapted their backend code around its limitations.
Imagining you are safe because you are smoke-checking DML statements in staging is just begging for a week-long outage.
Heck, I don't need to imagine, I have seen it time and time again.
After you’re onboard with this philosophy then I’m ready to sell you the fact that you don’t need joins, indexes, or transactions. If a giant complicated service like gmail doesn’t need any of those things then very likely that your thing doesn’t need them either.
second: even if you don't have an explicit schema in the database, you still have to have a way to know which data goes in which column. just because it isn't written in a definition language doesn't mean the concept doesn't exist and doesn't require to be maintained as the application domain evolves.
The last time I tried a project with a no-schema DB, it was a huge PITA. It sucks having a function fail on one single document because it had a number instead of a string. I love flexibility, but it is very valuable to have the right constraints in the right places. I quickly learned that a schema is just that.
By "typed language" I suppose you mean "statically typed language". This isn't relevant because A) JSON types exist and B) I've never had a problem with unexpected types from user data in a dynamic language, not even in PHP.
> Otherwise, how is this not caught when it enters your system?
It should be caught, and it's very important that it be caught all of the time and not have any issues due to bugs or version mismatches. That's why the schema should live in the DB.
YMMV I guess.
I don't think the article does a convincing job of arguing that releasing behind feature flags would be much easier to maintain than a staging deployment, though.
For making this consideration, it'd be worth seeing why the costs can't be lowered, why the effectiveness of staging can't be increased, or what cases might be difficult to handle with feature flags.
Literally their solution was "just make a product that costs $0 and test it against our production service".
To me, that was a very red flag.
You lovingly hand-provisioned your infra rather than writing infra as code. This makes it non-trivial to set up a stag environment, so you just don't.
Thats not to say that a staging env is bad, they can be very useful, if setup properly. It also might be the case that you really can't test in prod. (stuff that provides safety)
I am very keen on feature flags/magic routers. They have the advantage that you can test the inputs of the system with _real_ data. But that can also be a drawback.
Be it an infra change or application. It should always be like:
Staging -> Canary -> Production
If you invest time in making the above pipeline automated it will cause less outages.
P.S we use feature flags but it won't solve many other challenges around infra and application changes.
Some examples: - Changing from Classic ELBs to ALBs (We care about latencies so ALBs served 1% traffic in the start) - Zero downtime single mysql master upgrade. (ProxySql) - New k8s controllers. - New metrics adapters.
Staging should be like a playground to test all edge cases before you go to canary and then to prod.
> Staging -> Canary -> Production
Ironically, the author says exactly this when she presents on it. Why she wrote this article in this fashion is beyond me.
https://www.youtube.com/watch?v=adPQCuotAr4
As others have said, staging is useful for certain types of changes that may be high-risk and not time-sensitive. For other changes that are time-sensitive and are low risk (like changing some incorrect/offensive copy), being able to bypass staging is also useful. I don't mean to disrespect the author but it feels like the article was designed to take an extreme stance for views. It even reads like clickbait. I bet this article leaves a bunch of people believing that using an intermediate testing environment can never be useful.
Not everything can be feature flagged. Not all tests can be automated. Not all features can be safely tested in production. You can't always roll-back code.
Scenarios where I've found staging environments to be invaluable:
- Features that make wide-scale changes to the application
- Changes that require user acceptance testing. Management/Customers might ask for Feature X, you have designers mock up what it'll look like, you might even build prototypes of it... but until they see it and play with it, they might not fully realise that what they actually asked for isn't what they actually intended.
- Integrating changes from multiple developers and seeing how they play together
- Testing that your feature flagging actually works
- Testing how the system responds to dumping large amounts of data/requests at it - a half-day outage in production might easily exceed the cost of a whole year of running a staging environment.
- Testing how the system behaves after data migrations/changes (things where there's no roll-back without losing data)
- Testing integrations with external systems.
- Detecting poor performance, memory leaks, deadlocks, and other stuff that isn't possible on a dev's machine
Testing code on your machine is fine as a smoke test, but your laptop probably has multiple times the capacity of a single node on production, for a single user. So your change that results in 1GB/sec of full table scans on the database might not even be noticed because hey, you have an NVME drive. Your tests also run for maybe a few minutes, but that staging instance might be up for days or hours, and deals with hundreds
I've picked up numerous issues because someone (I'm including myself here) committed some code that resulted in Staging going down or behaving poorly.
I've even overheard conversations that were along the lines of "Yeah, this feature isn't working right at the moment in testing, but that's because staging is running slow today It's fine, we can push this out to production and it'll be better there." if it wern't for me overhearing that, it'd have resulted in a multi-GB video file being sent for every visitor to the site, costing a fortune in CDN charges and still running like crap because it was a 30 second video.
I still have concerns with deploying production upgrades behind flags due to the inherent issues surrounding things like database schema mismatches, terraform/infra upgrades that are not compatible bidirectionally etc. Has anyone worked on ways to manage these with feature flags?
If instead of converging to versioning, you want full-on branching that never merge, this sounds like very chaotic design. You'd then maybe want to design everything around feature isolation (see: UNIX philosophy), but comes with own caveats around new requirements and cross-cutting concerns.
I suspect this DevOps is working in loneliness, without enough communication with devs. Devs ought to be educated, it's part of DevOps job.
Personally, I have worked in an environment where staging was a pain, full of cobwebs and issues and data mismatches. I sympathize because I know how painful it can be.
But if you respect your end-users, you cannot delegate the duties of a staging environment to them.
From services that suddenly require new ports to be open through VPNs (development happens without intercontinental VPN’s of course) — to commandline options not correctly propagating to our hardened variant of configs — to even latency or bandwidth issues and timeouts due to assumptions.
There’s a difficult balance to strike when it comes to ease of development and not letting issues (security or issues with distributed systems) propagate to the customer.
Staging is also the place where you can safely let the chaos monkey reign supreme without fear of losing customer data! It’s the last place to test the “fit” of your system.
Staging has a place, and if puff pieces like this cause the company as a whole to downplay funding for it then I’m going to be mighty displeased.
These kinds of “articles” are a blight