I’m not sure I agree with this viewpoint. Assuming nothing critical lived on the service, and the worst side effect of the deploy was the client failing once and reconnecting, I’m not sure there’s anything wrong with just restarting the world.
Obviously if you know in advance you should not even start, but if you notice after you are halfway done? Meh, chances are that a rollback would take as long as finishing the push, and either way the service disruption continues until it’s finished.
You assume that is the only problem in the deploy. That assumption seems to have been correct. However, what if another bug had been found that requires a rollback from B to A? Now your rollback will result in a bunch of dead services until they are all on version A.
The best option when a defect in B was detected would have been to rollback so that you aren’t forced into a scenario where every fix has to be a fix forward.
When I hear "It's one of those things that talks to other instances of itself" I think of a redundant, high-availability database.
Assuming every machine sends data to all its peers regularly, if I have a cluster of 10 A machines and upgrade 1 machine to B, and the A machines crash, that means I lose 90% of my capacity - likely making the sole survivor get overwhelmed too.
"Outages are OK, clients can always reconnect or try again later" isn't the usual standard HA databases are operated to :D
At that point the A machines are dead though. There is no point in restarting them as A when you can also restart them as B, so I assume they don’t all die in one go.
But I guess we just have too little information to make a really informed decision about what you should do.
> There is no point in restarting them as A when you can also restart them as B, so I assume they don’t all die in one go.
A is “baked in”, so it should be the default.
There are simply too many classes of bugs which take time to manifest. Memory leaks, cache staleness, etc. So, for very good reasons, you consider version A preferable to version B.
When I hear "It's one of those things that talks to other instances of itself" I think of a redundant, high-availability database.
Oh, I thought of an ancient copy of Windows for Workgroups taking down the Windows NT 4 domain servers every single day. Real story. But that case was version A crashing version B, not the other way around.
My rule of thumb once you're already in a bad state is to figure out the fastest way to get to a good enough stable state. Deploys/roll-forwards are very commonly worn paths, so it's usually pretty easy to speed up a deploy or just do nothing and let it run its course. Rollbacks tend to be special, unique, and rare which makes it difficult, slow, and high risk. Odds are you'll be able to finish the deploy and stop the damage way faster than you can rollback _and_ undo the damage.
Obviously, avoiding deploying broken stuff is the best case, but preaching about fire prevention in the middle of a firefight is quite a bit too late and not terribly helpful.
If someone is paying for a service, you are obligated to provide that service. It may be nothing more than another (for example) random point-of-sale system to you, but to the business on the other end, it could be fatal.
Corrupting persistent data or creating ambiguous and irreconcilable (or expensively-reconcilable) states is a Very Bad Thing and something which one should avoid with extreme prejudice.
The updated version of the program, when pushed to servers, crashed the older version of the program on other servers which had yet to be updated. Rather than rolling back to diagnose and understand the cause, they charged on updating all the servers.
I don’t subscribe to the blog so perhaps it’s a selection bias, but the tone in those posts often feels passive aggressive.
In particular, I don’t like when folks write blog posts like that instead of talking to their colleagues. I remember being on a receiving end of such post as an ‘anonymous moron from someone’s work’ and as you can imagine, there was infinitely more nuance to the matter than my colleague ever cared to mention.
You misread that "many things", which refers to hypothetical actions the team could have taken (and which the post immediately describes), not hypothetical outcomes of their actual actions.
This article fails to mention the consequences of the rush to push version B, rather than rollback to resolve. As far as I can tell the result met the needs of the business?
If it did that's pure luck. It could have also had another bug. Rolling back keeps your service up. Pushing forward faster maybe gets your service back up, or maybe breaks it some other way.
When dealing with a complex system, roll back first, then figure out why it broke and how to fix it for the next attempt.
I should clarify - I don't disagree that rolling back and fixing the bug (or in a continuous delivery world - fixing forward as a priority) is the right thing to do. I'm just saying the article leaves out a lot of details and fails to articulate the reasons why the road taken was bad.
I thought the same thing. It's obviously a bad situation, but getting A out of service and getting every instance on version B seems like pulling victory out of the jaws of defeat.
My guess is there is some detail missing that clears up why what happened was wrong. Sounds like A had a bug that rose to the level of being a security problem of the denial of service variety, so redeploying it seems like a terrible idea.
> As far as I can tell the result met the needs of the business?
They have burnt their bridges by making it impossible to revert to older version. In effect, if someone can not run a newer version, they will be forced to completely abandon the software.
Of course, that still counts as "meeting needs of the business" if the only thing your business needs is surviving for a couple more days. But that can be accomplished a lot more easily by doing nothing — your business is unlikely to vanish in puff of smoke tomorrow, as long as you simply sit on your hands.
IMO, they could have halted distribution of buggy newer version, released minor fix for older one to stop it from crashing, then finally a backwards-compatible major release. That's more or less same effort, but with less damage to reputation and users of software.
Knowing the code changes well and making the decision to push on or revert back cant be second guessed by arm chair observers.
And of course it shouldnt happen in the first place but hey shit does happen. No one has ever managed to figure that out, and testing isnt a silver bullet.
It absolutely can be second-guessed, because this is a very common scenario and there is a consensus about the default way to deal with it—roll back first, ask questions later. It’s easier to fix bugs while production services are not on fire.
If you are seeing failures and want to roll forward, you should be able to clearly articulate why this is better than rolling back, and what makes this particular scenario different.
Otherwise, I’m going to be on the side of the armchair observers calling for rollbacks.
In this case, it sounds like a rollback (killing B, forcing all B-clients to reconnect to A) is just as damaging as continuing to deploy B (forcing all A-clients to reconnect to B). The tiebreaker for favoring of B would be A is already dead and you'll have to bring some As back. Tiebreaker favoring A would be other unknowns introduced in B that may be problematic.
Plus, we don't know if the bug resides with A or B. Maybe B is triggering a previous unknown bug in A instead of B being buggy.
Both A and B are known to be buggy, it is a fact. A is crashing. B is causing A to crash. A should not crash in the presence of B, that is a bug. B should not cause A to crash, that is a bug.
You cannot make the decision to roll forward on the basis that you guess that rolling forward to B is stable. You don’t have any evidence for that—just a guess. However, there is solid evidence that rolling backward to A is stable—it has been running that way for a while.
You must make the decision with imperfect knowledge. The correct answer is to roll back.
Except client A never crashed when receiving data from server A.
You can rightly argue that client A was irresponsible in not protecting itself more intelligently against invalid or unexpected inputs (dumping core is the crudest, bluntest protection there is; not the best choice in a distributed environment where transmission errors are a fact of life); but the system as a whole worked.
Had server B continued to supply client A with data in the format that client A expected to receive, no crash would have occurred and the entire rolling upgrade would have gone without a hitch. But no, the lazy irresponsible corner-cutting assholes couldn’t be arsed doing that; they just start pushing the new data to everyone, and then blame everyone but themselves when that goes sideways.
“The correct answer is to roll back.”
The correct answer is never to get into a state where rollback becomes necessary. Though having failed to do that, and so ended up in exactly this state, immediate rollback of B to A may well have been the next-best response, followed by system audit to determine what integrity/data loss has occurred and post-mortem of the procedures used, and subsequent corrections so that it doesn’t happen again.
But if you think a bunch of cowboys who were only too happy to shirk their responsibilities during the (private) development phase are suddenly going to own up and accept personal liability when it blows up in the (public) rollout phase, then boy, do I have an eight-figure Enterprise-y grade bridge to sell you.
Honestly, as a profession we very rarely roll out anything else. It’s one of the reasons we should be designing systems and procedures that are fault-tolerant from the very start.
In this particular case study, a very small, simple, obvious, cheap step (enabling server B to talk to clients A as well as clients B) was not taken during design and development, nor caught addressed during testing, resulting in very large, complicated, costly failure which the culprits then tried to mask instead of owning it like the professionals they’re supposed to be.
That so many other techies should automatically switch into CYA mode when it’s not even their own fuckup that we’re talking about here is a damning indictment of modern developer culture’s attitude toward professional responsibility and personal liability.
Better yet: build code and processes appropriate for long-lived massively distributed systems that will be incrementally upgraded over time. If the system is architected right, it will never get into a state where a rollback recovery becomes necessary. This is why we have Content Negotiation. This is why we have Erlang. This is not a new challenge by decades, and there is a huge body of expert knowledge and tools upon which to draw when implementing such systems, so any such complete catastrophic basic failures now are entirely down to PEBKAC, and remedied by a swift clue-by-four with a pink slip nailed to the end.
There is a very simple principle underpinning distributed communication: servers should never make assumptions about who their clients are and what they need. Talk to the client, find out what format(s) it’s willing/able to accept, and serve it the best match. A client should never need to know, nor care, if it’s talking to a version A server or a version B server: if the client says “I only understand version A data” then it’s the server’s job to serve up data in that exact format, not to pique and whine about how old and out of date the client is, push it version B data instead, and then blame the client for choking on it.
Indolent developers who approach IPC the same as local messaging and then blame everything but themselves when it barfs all over the place are the absolute bane of this industry, and this shit is entirely on them. And shame on the equally inept management culture that continues to let such incompetent amateurs get away with it.
There will be bad rollouts. I know of no set of practices which prevent bad rollouts. You talk about “indolent web developers”, well, that’s not productive and pointing fingers doesn’t make your software work. Your software will, in spite of your best practices, in spite of hiring the best people, in spite of experience, sometimes fall over.
My software shits itself all the time. What matters is that it does so safely. And when it doesn’t, I can tell you why, because I know what corners have been cut and why, and I’m not afraid to accept and acknowledge my responsibilities in such fuckups.
And yeah, I count on the fingers of one hand the number of web developers I’ve dealt with over the last decade who I’d be willing to cross the road to piss on were they on fire, and still have fingers to spare. They’re just the worst of the worst.
There was NO excuse for the failure described in the article. There was NO excuse for the described response to that failure. Yet such base incompetence and gross irresponsibility is not only systemic but entrenched, rationalized, and embraced in this industry. With responses like yours, it’s not hard to tell why. Buncha Children.
> There was NO excuse for the failure described in the article.
In this case, right. In general, stuff happens. There’s a tradeoff between reliability and effort. The correct reliability target is not 100%, because you can’t get 100% anyway, and as you approach 100% reliability the cost increases without bound.
I’m not sure what the rest of your comment is about besides taking a big shit on web developers and talking about how awful they are.
There is a precious small percentage of developers who are really good at making reliable systems and they have the burden / responsibility of spreading their knowledge. They work with the other actual developers you hire, those beautiful imperfect developers who cut corners, test in production, and don’t write tests.
You make changes to your culture and your practices. You build monitoring and rollout automation. You increase test coverage.
If you just call people children you’re going to be there, on the sidelines, watching other people build real products. You don’t teach people by making fun of them.
Sounds like a salty QA worker who is mad the developers "got away" with doing things the way they shouldn't be done. I mean it sounds like version A with the bugs unable to handle inputs without dying.
Having bug-free code in production is secondary to having a working system.
So it's arguably irrelevant that A was the buggy version, because A was also the only known-good battle-tested version.
As a developer who's also in charge of rollouts and has zero QA support I would also rollback in this (admittedly somewhat vague / hypothetical) situation.
Too many details missing here to make any sort of conclusions. Yeah, of course bugs aren't supposed to make it to production. But they do. Is it easier to roll back than it is to roll forward? Usually it's not. How far along are things before you recognize what's happening? How critical is the service?
You can't draw life lessons from some vague description of "something" that happened to "someone". Continuing the rollout could very well have been the best option. Or not. Who knows.
> Is it easier to roll back than it is to roll forward? Usually it's not.
If downtime has consequences then you have every incentive to make rollbacks happen at the touch of a button. From personal experience I know that plenty of teams simply are not at that point for many reasons.
If rollbacks are “hard” in any way for you, then you should know the known consequences of difficult rollbacks—they incentivize you to push forward even when the new release is kind of broken.
For example, let’s say that your team doesn’t test rollbacks and doesn’t have procedures in place. You know, in the corner of your mind, that a version rollout could potentially take your service completely down for days while people hunt down the bug or figure out how to do rollbacks on the fly. This is a predictable scenario to end up in, so at some point I recommend addressing it, or justifying why you don’t.
> Continuing the rollout could very well have been the best option. Or not. Who knows.
Rollback should be the default response in the face of uncertainty when you might have a bad rollout. Ideally. It should be the safe, one-button-press reset back to the way things were before.
Again, I know that’s often not the case. Schema changes are the biggest sticking point, usually. I often see schema changes in two phases—first you update the schema, and then later you roll out functionality which uses the new schema, in a separate rollout. This is tricky, but most updates are not schema changes.
Teams generally learn these things as object lessons (as in “we had downtime, wrote a postmortem, and it won’t happen again”) but that’s not the way things need to be. As the team evolves and the product matures, the release tooling gets better. You read articles like this and you can make the tooling better without the object lesson. No one-size-fits-all solution, obviously, but… painless, one-button rollbacks are one-size-fits-most for sure!
Someday, someone writes an integration test that sets up a new and old version and makes them talk to each other, checking that they both work correctly.
Another day, somebody writes an integration test that does a rollout from the previous version, followed by a rollback.
And someone else adds more monitoring signals so if they see service failures above some threshold it automatically rolls back.
These are all evolutionary steps. Yes, they don’t apply to all services. But—most of the services I run are somehow making money for the company, or people depend on them for critical pieces of functionality related to the company’s bottom line.
> Teams generally learn these things as object lessons (as in “we had downtime, wrote a postmortem, and it won’t happen again”) but that’s not the way things need to be. As the team evolves and the product matures, the release tooling gets better. You read articles like this and you can make the tooling better without the object lesson. No one-size-fits-all solution, obviously, but… painless, one-button rollbacks are one-size-fits-most for sure!
One advantage of 'learning as you go' is that oftentimes you can afford to be messy and cut some corners when you're just starting out without it impacting people too much.
I don't think anyone would disagree that having a rollout process that can be easily rolled back, utilizes things like feature flags, incremental rollout, and care around schema changes is a better process, but it's undeniably slower than "just roll it out and deal with the crashes", particularly when there's not plug and play standards for a lot of those processes and you need to build them yourself.
And if you have a tiny amount of users who are the early adopter type, messiness around rollouts and the occasional outage might not be the worst thing if you can move much faster.
> …messiness around rollouts and the occasional outage might not be the worst thing if you can move much faster.
Unfortunately, my personal experience with software engineers pushing to “roll forward” on failure is that they want to meet arbitrary deadlines. Metrics that are good for the SWE (e.g. if we roll forward, I complete my Q4 OKRs) but not good for the team—total velocity isn’t negatively impacted by a single rollback, at least not much, but reliability is positively impacted by rollback policy.
> If downtime has consequences then you have every incentive to make rollbacks happen at the touch of a button.
Not all rollbacks are the same. Even in a theoretical scenario where you can roll back the code to a previous version with a push of a button, it's not always possible due to many factors.
Sometimes new code creates and persists state that the old code can't handle and rolling back would just make the old version blow up because of it. Sometimes clients can't talk to the old code after talking to new code and expecting new protocol.
The point is, there's a lot of factors that could make one or another remediation strategy more or less viable, and sometimes rolling forward is the best way to proceed.
The article tries to advocate that rolling back is ALWAYS the correct thing to do, which is demonstrably not true for certain circumstances.
> Sometimes new code creates and persists state that the old code can't handle and rolling back would just make the old version blow up because of it. Sometimes clients can't talk to the old code after talking to new code and expecting new protocol.
These are both severe bugs that should have blocked deployment of the old version. Software should gracefully ignore state or data file content it doesn’t expect—not blow up. Well known techniques like fuzzing can help catch this kind of defect. Clients should be able to probe which version they are talking to and negotiate the protocol. This is kind of basic versioning cleanliness.
Sometimes new code creates and persists state that the old code can't handle and rolling back would just make the old version blow up because of it.
Or worse, the new feature involved the tracking of something in the real world, such as inventory or money, and rolling back after the feature had been used by a real customer would lose that.
The number of times I’ve had to explain to webdevs that when a customer asks where their money is “in a temp table” is not an acceptable answer.
Add to that standing orders to 'move fast and break things' -- this is just the same corporate shit warmed over that keeps people comfortably numb and confused about whether to lead follow or eat a donut
From experience with an app that doesn't auto-update, I know that the only thing that gets people to upgrade is a critical bug that affects them.
People generally keep using old versions of software for years. 90% of users don't care about new features or performance improvements or bug fixes that don't directly affect them.
So I understand what they did. If they indeed introduced a version C that had a workaround to avoid crashing A, then they would have never gotten rid of A. No matter how hard they pushed C, some people would just continue to use A.
But if A starts crashing, and there's an update available that fixes the issue, that's a really good motivator to get people to upgrade.
Sure, it'll suck for everyone for a short while, but it'll soon be over once everyone has upgraded.
And, as a dev, I also know that crashes are not the most serious kinds of bugs. They are usually easy to diagnose, and people realise that something is wrong right away -- much better than a bug that silently corrupts data, or a bug that causes subtly incorrect behavior.
If your goal is to get rid of A by any means necessary, make the business decision to turn off A at some point.
Not having the backbone to make that decision and allowing the call to forced at some arbitrary time by a bug and concluding the bug was a good thing (or at least had a silver lining) because it allowed the org to move away from A suggests a possible leadership decisiveness problem to me.
There is no organization of more than 500 employees that can take decisions like that.
Developers of one service simply don't have the power to force other developer in another part of the organization to do anything. That's assuming they're even aware of the existence of that service and who is supposed to own it.
Someone has the ability to make that decision. CTO or CEO in the worst case.
Having the decision "made for you" and being happy/relieved about it doesn't seem like the healthiest or firmest hand on the tiller way to run a company.
the team I'm working on has successfully turned off ~10 systems with/without replacements, and got all of our clients within a much larger than 500 person org to migrate fully.
There aren't enough details... I feel like the author needed to at least make a token effort to support the level of conviction she has that the developers in question are so terribly wrong.
It sounds like A was buggy (it dies when it receives a particular sort of request that it doesn't understand).
Maybe the devs knew about the bug in A? Maybe they decided it was better to make version B work as intended rather than try to throw in a one time hack to be backwards compatible with a broken version that was going to be a thing of the past in... How long does deployment take? Is it even long enough to matter? Maybe the most efficient solution was to allow some of the instances to die during deployment? What kind of application is it? Possibly a few failed requests while you wait for B to fully deploy isn't a big deal. Maybe the client auto-retries quickly enough that the dropped connection is negligible. Maybe there isn't a client at all.
The possibilities are pretty much endless given what we know.
People are saying there isn't enough info but I think the author definitely made the right analysis here.
The purpose of partial rollouts is to observe if there are bugs. The unusual case here is that version B's bug is forgetting backward compatibility, causing it to rapidly take down version A too. This means you can't simply rollback B as usual. After you rollback you still need to fix broken A instances. It's a lot of work but it would be the right thing to do.
Instead, they went all in on version B to avoid the bug they just introduced. This is completely reckless. You're skipping your usual process so you never get a chance to observe for other bugs in B. You should actually expect B to have even more bugs in it given you know it already has one major production breaking bug.
Going all in on a version that you're not confident in just to fix the one bug you know about is stupid.
With the info given, it's entirely possible that the bug was actually in version A. Perhaps sending it a payload that is perfectly acceptable by the spec of how the API should work caused A to crash.
In that world, "fixing" B could involve sending invalid or unintended data to work around the problem in A, or patching A before rolling out B (which, when you're at the point of rolling out A.2, you may as well just roll out B)
If this would be our situation, it would be a firmware update for our smart outlets in the field.
You definitely want to make sure that updates are backwards compatible. We use a mesh and once there was an incompatibility on the mesh protocol. You want to make sure this will never cause trouble. We decided to use another channel all together.
Our next update will update the bootloader and radio stack as well. Scary! This is an update that uses multiple intermediate bootloaders so we fall back to a working system at each step. We definitely do not want to have bricked devices out there. https://github.com/naveenspace7/bluenet-bootloader/blob/inte...
We almost didn't find a bug nevertheless. In our older devices there were some programmed with a binary generated by objcopy without the --gap-fill 0xFF flag. It lead to an hardfault in initialization of the new file system. Brrr. https://github.com/crownstone/bluenet/issues/73
The purpose of slow rollouts is to catch bugs before they affect everyone. We already have evidence this is buggy software, so slow rollouts are even more important.
And even if there aren't any other bugs, this scenario must have caused a lot of stress for the people in charge of the rollout.
ITT: people who don't do incremental rollouts or run reliable software stacks.
Sure, there are scenarios where you can justify rolling forward when you're spraying packets of death everywhere. None of those scenarios are the default - the default is roll back. (Imagine if there was some latent bug in the new release corrupting data in transit that you didn't notice until 100%, and you're not sure where it is, just that it's happening - now you literally can't roll back, because your current release will hose the previous one. At this point all you can do is go ¯\_(ツ)_/¯.)
Also, the title of the post screams pretty loudly to me that the people making the call to push the backwards-incompatible release didn't seriously think through the implications of their decision.
It seems odd to me that they do not version their exchange protocols, file formats, APIs etc. And then check if both participants have compatible versions when interacting..
It always amuses me how the HN collective tears apart whatever rachelbythebay writes. Maybe they should change their online identity to rossbythebay.
The tale they've shared should be obvious: When a deploy is causing problems, stop. And here are some generic techniques that could have been used to avoid ending up in a situation where one might choose to plow ahead instead of rolling back.
How is this controversial to anyone? This is Formalized Change Process 101.
In this case, the new version is meant to replace all instances of the old version. As an upgrade strategy, a kill switch, deliberate or otherwise, is rather effective. Whether it’s a good idea or not really depends on the situation.
I think that any post that is technical and related to software* go through some process of being tear apart collectively by HN.
I have the opposite impression that HN collectively appreciate Rachel posts very much, judging by the sheer number of her blog's posts that are upvoted to front page recurrently. Not many personal blogs reach that number of upvotes in HN.
*Maybe even all kind of posts, but the ones that are technical and about software are just the sweet spot because HN is full of software engineers.
It’s not clear whether their strategy was a success or a failure. If it succeeded, the old bromides of “don’t rush pushes” doesn’t actually apply. I mean it worked didn’t it? The reality of production software is that it’s always pushed in somewhat of a rush, to meet a myriad of internal and external deadlines.
I would counter the “don’t rush” bromide with “anything worth doing is worth doing adequately.”
95 comments
[ 3.3 ms ] story [ 156 ms ] threadObviously if you know in advance you should not even start, but if you notice after you are halfway done? Meh, chances are that a rollback would take as long as finishing the push, and either way the service disruption continues until it’s finished.
The best option when a defect in B was detected would have been to rollback so that you aren’t forced into a scenario where every fix has to be a fix forward.
Assuming every machine sends data to all its peers regularly, if I have a cluster of 10 A machines and upgrade 1 machine to B, and the A machines crash, that means I lose 90% of my capacity - likely making the sole survivor get overwhelmed too.
"Outages are OK, clients can always reconnect or try again later" isn't the usual standard HA databases are operated to :D
But I guess we just have too little information to make a really informed decision about what you should do.
A is “baked in”, so it should be the default.
There are simply too many classes of bugs which take time to manifest. Memory leaks, cache staleness, etc. So, for very good reasons, you consider version A preferable to version B.
Oh, I thought of an ancient copy of Windows for Workgroups taking down the Windows NT 4 domain servers every single day. Real story. But that case was version A crashing version B, not the other way around.
Obviously, avoiding deploying broken stuff is the best case, but preaching about fire prevention in the middle of a firefight is quite a bit too late and not terribly helpful.
Why would you make that assumption?
If someone is paying for a service, you are obligated to provide that service. It may be nothing more than another (for example) random point-of-sale system to you, but to the business on the other end, it could be fatal.
In particular, I don’t like when folks write blog posts like that instead of talking to their colleagues. I remember being on a receiving end of such post as an ‘anonymous moron from someone’s work’ and as you can imagine, there was infinitely more nuance to the matter than my colleague ever cared to mention.
Yes, yes, many things, certainly. Obviously, fix-forward is very bad because of these many things.
As a highly compensated software engineer, of course I know what these things are, but, just checking, what would you say they are?
When dealing with a complex system, roll back first, then figure out why it broke and how to fix it for the next attempt.
My guess is there is some detail missing that clears up why what happened was wrong. Sounds like A had a bug that rose to the level of being a security problem of the denial of service variety, so redeploying it seems like a terrible idea.
They have burnt their bridges by making it impossible to revert to older version. In effect, if someone can not run a newer version, they will be forced to completely abandon the software.
Of course, that still counts as "meeting needs of the business" if the only thing your business needs is surviving for a couple more days. But that can be accomplished a lot more easily by doing nothing — your business is unlikely to vanish in puff of smoke tomorrow, as long as you simply sit on your hands.
IMO, they could have halted distribution of buggy newer version, released minor fix for older one to stop it from crashing, then finally a backwards-compatible major release. That's more or less same effort, but with less damage to reputation and users of software.
And of course it shouldnt happen in the first place but hey shit does happen. No one has ever managed to figure that out, and testing isnt a silver bullet.
Not a place from which you can reasonably judge unless judgement is your only goal.
If you are seeing failures and want to roll forward, you should be able to clearly articulate why this is better than rolling back, and what makes this particular scenario different.
Otherwise, I’m going to be on the side of the armchair observers calling for rollbacks.
Plus, we don't know if the bug resides with A or B. Maybe B is triggering a previous unknown bug in A instead of B being buggy.
You cannot make the decision to roll forward on the basis that you guess that rolling forward to B is stable. You don’t have any evidence for that—just a guess. However, there is solid evidence that rolling backward to A is stable—it has been running that way for a while.
You must make the decision with imperfect knowledge. The correct answer is to roll back.
You can rightly argue that client A was irresponsible in not protecting itself more intelligently against invalid or unexpected inputs (dumping core is the crudest, bluntest protection there is; not the best choice in a distributed environment where transmission errors are a fact of life); but the system as a whole worked.
Had server B continued to supply client A with data in the format that client A expected to receive, no crash would have occurred and the entire rolling upgrade would have gone without a hitch. But no, the lazy irresponsible corner-cutting assholes couldn’t be arsed doing that; they just start pushing the new data to everyone, and then blame everyone but themselves when that goes sideways.
“The correct answer is to roll back.”
The correct answer is never to get into a state where rollback becomes necessary. Though having failed to do that, and so ended up in exactly this state, immediate rollback of B to A may well have been the next-best response, followed by system audit to determine what integrity/data loss has occurred and post-mortem of the procedures used, and subsequent corrections so that it doesn’t happen again.
But if you think a bunch of cowboys who were only too happy to shirk their responsibilities during the (private) development phase are suddenly going to own up and accept personal liability when it blows up in the (public) rollout phase, then boy, do I have an eight-figure Enterprise-y grade bridge to sell you.
We occasionally roll out bad software. I know of no reasonable set of practices which can avoid it.
I honestly don’t understand how you would expect to make this possible without an obscene budget + insanely slow pace of development.
Honestly, as a profession we very rarely roll out anything else. It’s one of the reasons we should be designing systems and procedures that are fault-tolerant from the very start.
In this particular case study, a very small, simple, obvious, cheap step (enabling server B to talk to clients A as well as clients B) was not taken during design and development, nor caught addressed during testing, resulting in very large, complicated, costly failure which the culprits then tried to mask instead of owning it like the professionals they’re supposed to be.
That so many other techies should automatically switch into CYA mode when it’s not even their own fuckup that we’re talking about here is a damning indictment of modern developer culture’s attitude toward professional responsibility and personal liability.
It’s excuses all the way down.
Better yet: build code and processes appropriate for long-lived massively distributed systems that will be incrementally upgraded over time. If the system is architected right, it will never get into a state where a rollback recovery becomes necessary. This is why we have Content Negotiation. This is why we have Erlang. This is not a new challenge by decades, and there is a huge body of expert knowledge and tools upon which to draw when implementing such systems, so any such complete catastrophic basic failures now are entirely down to PEBKAC, and remedied by a swift clue-by-four with a pink slip nailed to the end.
There is a very simple principle underpinning distributed communication: servers should never make assumptions about who their clients are and what they need. Talk to the client, find out what format(s) it’s willing/able to accept, and serve it the best match. A client should never need to know, nor care, if it’s talking to a version A server or a version B server: if the client says “I only understand version A data” then it’s the server’s job to serve up data in that exact format, not to pique and whine about how old and out of date the client is, push it version B data instead, and then blame the client for choking on it.
Indolent developers who approach IPC the same as local messaging and then blame everything but themselves when it barfs all over the place are the absolute bane of this industry, and this shit is entirely on them. And shame on the equally inept management culture that continues to let such incompetent amateurs get away with it.
There will be bad rollouts. I know of no set of practices which prevent bad rollouts. You talk about “indolent web developers”, well, that’s not productive and pointing fingers doesn’t make your software work. Your software will, in spite of your best practices, in spite of hiring the best people, in spite of experience, sometimes fall over.
Yes, it will sometimes segfault.
And yeah, I count on the fingers of one hand the number of web developers I’ve dealt with over the last decade who I’d be willing to cross the road to piss on were they on fire, and still have fingers to spare. They’re just the worst of the worst.
There was NO excuse for the failure described in the article. There was NO excuse for the described response to that failure. Yet such base incompetence and gross irresponsibility is not only systemic but entrenched, rationalized, and embraced in this industry. With responses like yours, it’s not hard to tell why. Buncha Children.
In this case, right. In general, stuff happens. There’s a tradeoff between reliability and effort. The correct reliability target is not 100%, because you can’t get 100% anyway, and as you approach 100% reliability the cost increases without bound.
I’m not sure what the rest of your comment is about besides taking a big shit on web developers and talking about how awful they are.
There is a precious small percentage of developers who are really good at making reliable systems and they have the burden / responsibility of spreading their knowledge. They work with the other actual developers you hire, those beautiful imperfect developers who cut corners, test in production, and don’t write tests.
You make changes to your culture and your practices. You build monitoring and rollout automation. You increase test coverage.
If you just call people children you’re going to be there, on the sidelines, watching other people build real products. You don’t teach people by making fun of them.
So many words to non precisely say one to three (assuming a five finger hand).
So it's arguably irrelevant that A was the buggy version, because A was also the only known-good battle-tested version.
As a developer who's also in charge of rollouts and has zero QA support I would also rollback in this (admittedly somewhat vague / hypothetical) situation.
You can't draw life lessons from some vague description of "something" that happened to "someone". Continuing the rollout could very well have been the best option. Or not. Who knows.
If downtime has consequences then you have every incentive to make rollbacks happen at the touch of a button. From personal experience I know that plenty of teams simply are not at that point for many reasons.
If rollbacks are “hard” in any way for you, then you should know the known consequences of difficult rollbacks—they incentivize you to push forward even when the new release is kind of broken.
For example, let’s say that your team doesn’t test rollbacks and doesn’t have procedures in place. You know, in the corner of your mind, that a version rollout could potentially take your service completely down for days while people hunt down the bug or figure out how to do rollbacks on the fly. This is a predictable scenario to end up in, so at some point I recommend addressing it, or justifying why you don’t.
> Continuing the rollout could very well have been the best option. Or not. Who knows.
Rollback should be the default response in the face of uncertainty when you might have a bad rollout. Ideally. It should be the safe, one-button-press reset back to the way things were before.
Again, I know that’s often not the case. Schema changes are the biggest sticking point, usually. I often see schema changes in two phases—first you update the schema, and then later you roll out functionality which uses the new schema, in a separate rollout. This is tricky, but most updates are not schema changes.
Teams generally learn these things as object lessons (as in “we had downtime, wrote a postmortem, and it won’t happen again”) but that’s not the way things need to be. As the team evolves and the product matures, the release tooling gets better. You read articles like this and you can make the tooling better without the object lesson. No one-size-fits-all solution, obviously, but… painless, one-button rollbacks are one-size-fits-most for sure!
Someday, someone writes an integration test that sets up a new and old version and makes them talk to each other, checking that they both work correctly.
Another day, somebody writes an integration test that does a rollout from the previous version, followed by a rollback.
And someone else adds more monitoring signals so if they see service failures above some threshold it automatically rolls back.
These are all evolutionary steps. Yes, they don’t apply to all services. But—most of the services I run are somehow making money for the company, or people depend on them for critical pieces of functionality related to the company’s bottom line.
One advantage of 'learning as you go' is that oftentimes you can afford to be messy and cut some corners when you're just starting out without it impacting people too much.
I don't think anyone would disagree that having a rollout process that can be easily rolled back, utilizes things like feature flags, incremental rollout, and care around schema changes is a better process, but it's undeniably slower than "just roll it out and deal with the crashes", particularly when there's not plug and play standards for a lot of those processes and you need to build them yourself.
And if you have a tiny amount of users who are the early adopter type, messiness around rollouts and the occasional outage might not be the worst thing if you can move much faster.
Unfortunately, my personal experience with software engineers pushing to “roll forward” on failure is that they want to meet arbitrary deadlines. Metrics that are good for the SWE (e.g. if we roll forward, I complete my Q4 OKRs) but not good for the team—total velocity isn’t negatively impacted by a single rollback, at least not much, but reliability is positively impacted by rollback policy.
Not all rollbacks are the same. Even in a theoretical scenario where you can roll back the code to a previous version with a push of a button, it's not always possible due to many factors.
Sometimes new code creates and persists state that the old code can't handle and rolling back would just make the old version blow up because of it. Sometimes clients can't talk to the old code after talking to new code and expecting new protocol.
The point is, there's a lot of factors that could make one or another remediation strategy more or less viable, and sometimes rolling forward is the best way to proceed.
The article tries to advocate that rolling back is ALWAYS the correct thing to do, which is demonstrably not true for certain circumstances.
These are both severe bugs that should have blocked deployment of the old version. Software should gracefully ignore state or data file content it doesn’t expect—not blow up. Well known techniques like fuzzing can help catch this kind of defect. Clients should be able to probe which version they are talking to and negotiate the protocol. This is kind of basic versioning cleanliness.
Or worse, the new feature involved the tracking of something in the real world, such as inventory or money, and rolling back after the feature had been used by a real customer would lose that.
The number of times I’ve had to explain to webdevs that when a customer asks where their money is “in a temp table” is not an acceptable answer.
Toggle the flag. Feature is disabled.
People generally keep using old versions of software for years. 90% of users don't care about new features or performance improvements or bug fixes that don't directly affect them.
So I understand what they did. If they indeed introduced a version C that had a workaround to avoid crashing A, then they would have never gotten rid of A. No matter how hard they pushed C, some people would just continue to use A.
But if A starts crashing, and there's an update available that fixes the issue, that's a really good motivator to get people to upgrade.
Sure, it'll suck for everyone for a short while, but it'll soon be over once everyone has upgraded.
And, as a dev, I also know that crashes are not the most serious kinds of bugs. They are usually easy to diagnose, and people realise that something is wrong right away -- much better than a bug that silently corrupts data, or a bug that causes subtly incorrect behavior.
Not having the backbone to make that decision and allowing the call to forced at some arbitrary time by a bug and concluding the bug was a good thing (or at least had a silver lining) because it allowed the org to move away from A suggests a possible leadership decisiveness problem to me.
Developers of one service simply don't have the power to force other developer in another part of the organization to do anything. That's assuming they're even aware of the existence of that service and who is supposed to own it.
Having the decision "made for you" and being happy/relieved about it doesn't seem like the healthiest or firmest hand on the tiller way to run a company.
the team I'm working on has successfully turned off ~10 systems with/without replacements, and got all of our clients within a much larger than 500 person org to migrate fully.
Fact is, it's a nightmware to handle. It just takes one client who can't or won't upgrade and it's blocked forever.
Except this time they did seem to dig their way out of the hole :D
It sounds like A was buggy (it dies when it receives a particular sort of request that it doesn't understand).
Maybe the devs knew about the bug in A? Maybe they decided it was better to make version B work as intended rather than try to throw in a one time hack to be backwards compatible with a broken version that was going to be a thing of the past in... How long does deployment take? Is it even long enough to matter? Maybe the most efficient solution was to allow some of the instances to die during deployment? What kind of application is it? Possibly a few failed requests while you wait for B to fully deploy isn't a big deal. Maybe the client auto-retries quickly enough that the dropped connection is negligible. Maybe there isn't a client at all.
The possibilities are pretty much endless given what we know.
And while if age with the practice of roll back first ask questions later the fatalistic tone does not match the documented impact.
Of course well-thought release policy will help immensely (how to test, when to release, staged rollouts, etc...)
The purpose of partial rollouts is to observe if there are bugs. The unusual case here is that version B's bug is forgetting backward compatibility, causing it to rapidly take down version A too. This means you can't simply rollback B as usual. After you rollback you still need to fix broken A instances. It's a lot of work but it would be the right thing to do.
Instead, they went all in on version B to avoid the bug they just introduced. This is completely reckless. You're skipping your usual process so you never get a chance to observe for other bugs in B. You should actually expect B to have even more bugs in it given you know it already has one major production breaking bug.
Going all in on a version that you're not confident in just to fix the one bug you know about is stupid.
In that world, "fixing" B could involve sending invalid or unintended data to work around the problem in A, or patching A before rolling out B (which, when you're at the point of rolling out A.2, you may as well just roll out B)
You definitely want to make sure that updates are backwards compatible. We use a mesh and once there was an incompatibility on the mesh protocol. You want to make sure this will never cause trouble. We decided to use another channel all together.
Our next update will update the bootloader and radio stack as well. Scary! This is an update that uses multiple intermediate bootloaders so we fall back to a working system at each step. We definitely do not want to have bricked devices out there. https://github.com/naveenspace7/bluenet-bootloader/blob/inte...
We almost didn't find a bug nevertheless. In our older devices there were some programmed with a binary generated by objcopy without the --gap-fill 0xFF flag. It lead to an hardfault in initialization of the new file system. Brrr. https://github.com/crownstone/bluenet/issues/73
And even if there aren't any other bugs, this scenario must have caused a lot of stress for the people in charge of the rollout.
Sure, there are scenarios where you can justify rolling forward when you're spraying packets of death everywhere. None of those scenarios are the default - the default is roll back. (Imagine if there was some latent bug in the new release corrupting data in transit that you didn't notice until 100%, and you're not sure where it is, just that it's happening - now you literally can't roll back, because your current release will hose the previous one. At this point all you can do is go ¯\_(ツ)_/¯.)
Also, the title of the post screams pretty loudly to me that the people making the call to push the backwards-incompatible release didn't seriously think through the implications of their decision.
https://github.com/bitcoin/bips/blob/master/bip-0050.mediawi...
In that case they decided to rollback.
The tale they've shared should be obvious: When a deploy is causing problems, stop. And here are some generic techniques that could have been used to avoid ending up in a situation where one might choose to plow ahead instead of rolling back.
How is this controversial to anyone? This is Formalized Change Process 101.
https://news.ycombinator.com/item?id=9224
"Rossbythebay" is good though.
I have the opposite impression that HN collectively appreciate Rachel posts very much, judging by the sheer number of her blog's posts that are upvoted to front page recurrently. Not many personal blogs reach that number of upvotes in HN.
*Maybe even all kind of posts, but the ones that are technical and about software are just the sweet spot because HN is full of software engineers.
I would counter the “don’t rush” bromide with “anything worth doing is worth doing adequately.”