Ask HN: Advice for leading a software migration?
I'm about to take lead of a decent sized software migration at work. (From V1 of some subsystem, to v2, both in house. We want to deprecated and eventually remove V1 totally) For 8 of our clients, totalling about 16 million customers.
I don't have too many details to share, as I don't know what's relevant. But I'm asking if anyone has any advice or recommended reading regarding such?
One book that is really inspiring me about it is "how big things get done" by Bent Flyvbjerg and Dan Gardner. In it, there's some key bits of advice such as
* Think slow, ask fast, and mitigate long tailed risks.
* Compartmentalize and stick to repeated processes. "Build with LEGOs"
* Look around at other projects of similar nature.
The last point is why I'm here, as I know some of you have been in the game for longer than I have, so feel free to share experiences that you might think is relevant, if you'd like.
79 comments
[ 3.0 ms ] story [ 125 ms ] threadWhat is motivating the move politically?
What is motivating the move psychologically?
Be clear regarding each.
They are all there in the decision.
Don't pretend they aren't.
Only one of them is technical.
And it is not most of success.
Good luck.
So, to add to the comment above:
- does your migration affect the clients and the way clients work in any way? No matter how small, if the answer is “yes” then you need to ensure full buy-in from the clients. Even if your migration went flawlessly from a technical perspective, if a large enough client didn’t realise that V2 comes with some change that he doesn’t like, and when the change hits him after the migration, he raises this as a problem and escalates the problem high enough up the food chain with the message “this is not working for us” then you are going to be rolling back, regardless of the technical stuff. So, realise that the clients are big stakeholders and they need to be managed from the beginning of the project until some time after your V2 go live. In my experience the best results come from bringing them close to the project early and get some buy in by having them e.g. do some end-to-end testing if V2 and get them to accept the V2 before the go live. Preferably in an email for if things get ugly at some point (it happens, is sucks).
- also as the comment above says, don’t ignore the political. You should know what every important stakeholder gets out of this? Don’t forget personal ambition, ego, promotions etc as possible motivators for stakeholders. Who of the stakeholders are supporting your project now, and who is not? And just as important, what may change for a stakeholder to “switch camp” from supporter to not. Maybe the stakeholder is a mid-level manager who is measured on some KPI and V2 will make his KPI look better. So he’s a supporter. But then his company gets a new ceo and the KPIs change. Now he is no longer a supporter because V2 doesn’t give him anything he wants. And he’s actually now against your project because he has to commit some resources to it, but doesn’t get anything, so actually if your project is killed he frees up resources and doesn’t loose anything.
From one developer to another; The tech part is the easier part I’m sorry to say.
Technical and political components are external. Career aspirations and mitigating boredom/stagnation by pursuing complicated work create motivations to invent interesting projects.
And there’s the ability to claim integration from v1 to v2 as progress. Rather than only change.
To put it another way there is always some degree of change for the sake of change motivating our desires for change. Particularly when a big chunk of our time must be accounted for. Typically, playing video games, sleeping, and walking a dog through the woods instead are not viable alternatives in contexts where data migrations are being considered.
If migration was something the OP didn’t want to do, the question would be about finding a new job.
Write tests if you can, and set up a staging environment for V2 that you can setup and tear down easily for battle testing way before going live.
From there, break the tasks up from above into their business domains, and abstract those into new api services that the v1 system can use without any downtime.
For a frontend migration, that’s a whole different story and you would have to provide more details such as “moving from legacy Angular 1 to React 18 while it’s running”.
> Write tests if you can, and set up a staging environment for V2 that you can setup and tear down easily for battle testing way before going live.
I've successfully helped migrate a critical project and followed exactly this strategy. Older versions were being developed and run 1:1 in parallel to the newer ones until the customers got only a small downtime due the change of IP Addresses were the system was running
https://github.blog/2018-09-28-upgrading-github-from-rails-3...
They show some details how to migrate on large scale yet be safe while doing so.
- migrations always run longer than expected. In my case, leadership estimates were off by a factor of 10. What the eng manager originally said would take 3 months ended up taking a couple years.
- try to deliver quick wins and incremental value. This is often hard though. But it’s worth a try.
- Try to avoid this becoming the project everybody attaches their pet projects too. It’s too easy for people to make this the project where they use that new framework, test well, set up a design system, and make lots of little changes.
- that being said: migrations are easiest if you keep the design (visually and engineering) exactly the same. There will be lots of pressure to “just redo it while you’re already having to rewrite it”, but the uncertainty of a redesign really slows things down. Having a reference implementation means you don’t have to invent tons of acceptance criteria from first principles.
- as soon as things start getting delayed, which they will, try offering to cut corners or cancel the project. You want somebody else in corporate to stick their neck out to extend the project.
- Try seeding the team with more veteran ICs internally. You’ll need their help as you uncover dragons or need to get other teams to help run or integrate your new code.
- Among projects I’ve seen like this, the person running them gets fired or quits partway through at least half of the time. This is often because some middle manager made a promise they couldn’t keep to executives, and needs a scapegoat to save their own job. (It’s often that kind of middle manager who switches jobs every two years and keeps failing up silently and the project delay happens halfway through their stay at the company and they’re just trying to get to the two year mark and quit before anybody realizes what is going on internally.)
This is a good point. Or the migration appears to have been very successful to management (before it's actually complete from an engineering perspective) and they get promoted / moved onto higher priority work.
Either way: make sure you are keeping the rest of the relevant engineering organization informed about how the new system works and how the migration is going to work.
After more than a decade at large sw companies, I can count on one hand the number of migrations where the legacy system was ever able to be turned down. I’ve seen migrations drag on for years, to the point where most of the team has turned over. I’ve seen them become a three-way migration because the second version was deemed insufficient so a third solution was introduced.
Absolutely put your most senior devs on this; maintain as much support from management as possible; budget for much, much more time than you think; you need full commitment or you are going to be maintaining both systems indefinitely.
If part of the plan wasn't to run a v1 shim on top of v2 to handle legacy users that won't migrate, v2 almost certainly doesn't meet the needs of v1 customers and it's not a question of 'migration' it's a question of ending a product and releasing a similar product.
Sometimes that's what's wanted and needed, but often it's not, and then it's a surprise that the v1 users want their needs met and it's hard to say no to paying customers, but nobody signed up to run two products forever.
Migrations are painful, thankless and always run over budget and time. Unless I've been at the company long enough, have enough confidence and rapport with my reporting head and skip level, I'd rather not do it.
I'm never taking any big (more than 2-3 month) migrations. Only small predictable subsystems that i can rollback or run both v1 and v2 in parallel. First one third time is for discovering by making changes and seeing where things break and possibly come up with fast tests (manual or automated). Last one third is for actual testing, trying out small pieces in production and fixing unexpected issues. So take your dev estimate and multiply by 3.
Even then, you have to shoot down any demands to use new frameworks, new processes and new dependencies. And resist your own temptation. Remember no one gives a shit about migrations.
You will be asked a thousand times on the progress by people incapable of fathoming the complexity. They expect a percentage. Have one ready with a small roadmap, of cornerstones and publish as a report or something. Everytime someone asks, point to the report. No one ever opens that report.
Redesigns almost always result in a decrease in metrics/KPIs. The redesign just lacks the learned improvements that were baked into the old product. So, the initial launch almost always seems like a failure - and requires leadership to expect this dip before problems can be patched.
Large projects have lots of vulnerabilities, but I've seen many get sucked into "v2 is going to fix all the problems and mistakes of v1." Without a solid technical plan, goals and deliverables, it's easy for that effort to devolve into a years-long architect astronaut-style arguments about nanoseconds saved by something over something else. Halfway through somebody will suggest all problems with this approach will be solved by $newLanguage. If it doesn't serve the goals and deliver meaningful value, avoid getting stuck in those traps. Know what you're trying to solve.
There will probably be a v3 and somebody will complain about your version someday, too. It's the way of progress. As long as it's an improvement over the old and lays the right groundwork, continue moving in the right direction.
Find ways to ship smaller versions of the migration first. If possible: isolate features that can be migrated on their own.
If possible silently run v2 in parallel with v1 for as long as it takes to be comfortable with v2.
Assume that at some point you are going to have to completely halt the migration, go back to v1-only, fix something, and restart the migration.
I'd bet it's going to take 2-3x longer than you think to completely deprecate v1.
I don't blame people. Usually the offenders are in a culture where telling the truth is unpopular. It just depends on if you want to have a successful transition, or make people feel good about a project that takes 6 years to not finish.
If you're migrating data, can you take counts of things, so you can get quickly verify, i.e. we have 2.32M records before, and we have a way to prove we have 2.32M records after.
Mostly though, all migrations take longer than you think.
"But that means if I have a 1 month project and I have to involve 10 other teams it would take 10 years or so"
Yes that is another way of saying it will fail.
If you can figure out how to get up front sign off from all teams so you can just do it all within your own team you will make things go a lot faster.
Separately figure out how to cake slice things. If you have Dev and prod for instance and 10 applications, don't migrate all 10 in Dev first. Migrate 1 app in Dev, then the same app in prod, then go onto the next app. That way wherever you stop at least something will be delivered to the custoner.
After a couple iterations your estimation will be much better and you will see the light in the end of the tunnnel.
EDIT - almost forgot the most important part: write small backwards compatible prs that are deploy to production constantly. Don’t write few big PRs, they will bite your back.
Management rarely wants to wait years for before seeing any pay off from a big dramatic cutover, and big sweeping changes are disruptive to clients.
This will likely create more work. Maybe some layer has to be built to allow v1 and v2 subsystems to both operate with the other parts of the app. But it should ultimately make it less stressful.
If you can allow some friendly departments from friendly clients to test and provide feedback before rolling it out to the whole company or the full set of companies, that would probably go a long way to help identify blind spots.
Most importantly, listen to your team and the people who know the systems well. The projects I’ve seen that have really gone sideways are ones where the people who know the true issues are never consulted, or completely ignored when they try to raise an alarm.
If you can't do it incrementally, it's going to fail. Corporations rarely have the attention span and staff tenure to make that kind of migration work.
Even if it takes a year of pre-work to get to a point where it can be done incrementally, it will be the only way it gets done.
Integration tests for behavior verification.
But the incremental migration is key.
Require anyone who reports to you during the planning process to do the same; provide the most accurate estimate possible, then be transparent about their padding.
If there's a a 'known unknown' call it out. Mitigate risk with high-level executive check-ins. Be candid with your status lights, and tell them what you're doing to mitigate any risk on a regular basis.
Migration is about managing up to the org not just to a boss; the more candid you are the more you deflate the rage that comes with unexpected downtime, rollbacks, etc.
Sorry to say it but it smells a bit of "we're migrating because microservices or kafka or whatever" - don't. Grow organically into it. Do this kind of stuff because you have to, not because you can.
If you said you're struggling/something doesn't work and you can't anymore – it would be easier to advice and it wouldn't feel like a step in the wrong direction.
If you have a big bang V2 that is incompatible with V1, you've already lost. There should be a V1.0.1, V1.0.2, etc that incrementally gets you to what you would've already gotten with V2 without losing the ability to do each individual piece in stepwise succession. That's essentially what the "strangler pattern" is.
The strangler pattern is helpful, because it forces you to focus on what you piece-wise need to "strangle" -- which usually isn't as much as it looks like at first blush.
The hardest part of most migrations is data model migrations, and the best approach here is to start writing to the new model before you start reading it from the core business logic. By the time that works as expected, much of the pain is done. This takes a long time because it requires a lot of repairing the ship as you steer it, so for the sake of the business, it is best doing it in small pieces aligned with chunks of business value or new feature iteration velocity.
The second part of many migrations is adding sufficient test coverage -- in a lot of cases, this will already be present, but if it's not, you're in for a world of pain. If you don't have enough test coverage of the V1, add that before you try and do anything fancy or you'll end up testing "the long way" (through production outages and late night scrambles to hotfix and inevitable rollbacks).
The first thing I would say is take all advice you get with a huge grain of salt. Details matter, and the particular details that matter the most vary tremendously from project to project. That said, here's my advice:
- Be clear on the goals up front and along the way. It's already a red flag that you don't lead with the goal and say things like "I don't have many details to share as I don't know what's relevant". In the heady early days of a big project, there will be many rose-tinted ideas of problems that can be solved, and people will keep tacking them on without the burden of knowing the stumbling blocks that will inevitably come. You need to keep the goal in mind at all times so you can ruthlessly make tradeoffs every step of the way. It's even okay if the goal changes, but be explicit about it.
- Make sure you find a way to do it incrementally. If you find you have code accumulating that is not being exercised in the running system for more than a few weeks at a time, that's a huge red flag. Kent Becks Trough of Despair [1] from a few days ago is relevant to this point. You need to be very careful that your trough doesn't grow wider than you can handle. It's surprisingly easy for that to happen given the nature of software system complexity growth. The risk is even greater if you have a lot of resources at your disposal because more cooks in the kitchen means hire risk of losing cohesion.
- There's no substitute for seniority up and down the chain. One or two weak links can really derail the entire effort. And it's not just about technical strength, communication and social aspects are equally important. Every single front line engineer will likely run into issues that will be relevant outside of their scope, but will they recognize that for areas they are not focused on? When a project is too big for any one individual to understand all the details, you need a critical mass of big picture thinkers, and some lightweight ways for informal conversations to be sparked and escalated (or de-escalated) as the importance comes into focus
- If you ever ask an engineer why they're doing something and the answer is "because XXX told me to" or "because that's the plan", it's time for a quick sit-down. Engineers who don't know why they're doing something will not make good choices when the unforeseen arises (which it always does).
- Know your clients. Are they internal, external? Are there ghost or second-order clients due to leaking internal details or other encapsulation violations? Will you still support all the features they need? What actions will they need to take to support you? What rate of change can they support? You can have the perfect end-state in mind, and then get tripped up by mundane constraints on your clients that you were not fully aware of.
Assuming V1 and V2 offer users the same functionality, there’s a bunch of tests you can offer. The best one IMO is oracle testing where you do something on v1 and v2 and check they do the same thing. Preferably roll out to a subset of users such as via a canary deployment and make sure you have a rollback plan.
It's not clear whether v2 is already in production somewhere else. If it is not, you better wait until 1) the v2 data model has really been finalized and in prod and 2) key resources can be made available to the migration team. We were forced to begin the migration before the new product was complete and it was just plain impossible. We had to start all over every quarter.
- Migrations are very difficult to estimate. Any optimistic estimate will bite back. Hold off as much as you can, and ensure appropriate buffers if you really have to.
- ensure that the 8 clients have an identical v1 data model (tables, constraints, etc). If that is not the case, remember you will run n migrations, not 1.
- You need a team with knowledge of both v1 and v2 data models, as well as business domain know-how. There are many decisions that need to be made and you need the right people to be around.
- Not everything has to be migrated. Trying to migrate 100% is a common mistake: engage with the customers to understand what's the minimum that legally and operationally has to be migrated, especially if the v1 system has been in production for many years.
- Data migration is a iterative process, and the last thing you want to is to manually QA every iteration. You need to develop tests that will provide a reasonable data integrity assurance.
- Dashboards showing data migrated, failing/ok tests, remaining tables, etc. help communicate status and track progress.
- Customers will need to be involved during the whole project. You need them to commit to making people available that can quickly answer questions to unlock you dev teams. ideally, you want to create a single team. Make sure that decisions are traced and versioned.
- Performance matters. Discuss the performance requirements upfront. Our process was very, very slow and we found out a bit too late that the customer would not tolerate such down time. Also, discuss "when" ok to migrate, how to rollback in case of failure, etc.