These stories are great, and totally resonate with me at the moment. I'm managing a team working on a 30 year old codebase. It is in technical debt, like many codebases that old. That being said, the codebase is full of so much good stuff from hard learned lessons over 30 years, and that's where its value lies.
I'm convinced that the way forward is not to refactor or rewrite it, but to produce something new and different that draws on the 30 years of experience learned during the development of the original product. And, like Basecamp, run them side-by-side and allow users to chose what they want.
I think the hardest conversation circles around language choice. One problem with choosing innovative languages is that you can end up on a dead end. Zipcar, for example, used Tcl. While I am a fan of Tcl, it became obvious around 2012ish (IIRC) that this was not the future. There was a first attempt at a migration using rails, but that ended up being ditched for a JVM stack. (I don't know if most of it is still in groovy, but there were also pieces in ruby, kotlin, and java.) We also had a large number of performance and billing problems we needed to solve, and as we were guaranteeing compatibility and interoperability (a big bang switchover was too risky for many reasons), we had a lot of performance and usability issues for years.
It cost far more than management expected in the outset for many reasons, but at the same time, staying on Tcl was unsustainable for developer recruitment and retention purposes. It's something that biases me toward blub languages now as a manager unless you have a very good reason to innovate there.
> … staying on Tcl was unsustainable for developer recruitment and retention purposes…
I agree that this is a real phenomenon, and I find fascinating because it illustrates the role of fashion in software development. TCL may have been quirky compared to other languages, but actual working software was successfully developed in it by more-or-less typical developers. Yet it failed anyway because it wasn’t “cool”.
Not being trendy meant that fewer brains were thinking about it, evolving it by improving tooling and writing libraries, producing training materials and evangelizing it. Ultimately it just stagnates.
I don't even know if it's "fashion" as much as an industry that only knew that java 1.2 wasn't going to be the path forward if you wanted to develop with speed. There was no guarantee python or php were going to be the winners.
I think it does speak to the amount of labor that goes into making a language viable as an option, though. Java had paid labor. Tcl didn't have enough unpaid labor to succeed while python did.
> staying on Tcl was unsustainable for developer recruitment and retention purposes
I'm not sure I get this. Any competent programmer can learn Tcl. So find one versed in scripting languages like Python or Ruby, and they can learn how Tcl works in a week and learn how the program is structured over a few weeks and start making small changes. I'm skeptical that this training overhead would really cost more than replacing the whole system if it really was working well.
I doubt training was the issue. There are many languages that developers can work with, but would rather not. They might have a personal dislike of the language, be persuaded by industry fashions, or worry that they'll be harming their future career prospects.
You're looking at it soley from the PoV of the company.
> Any competent programmer can learn Tcl. So find one versed in scripting languages like Python or Ruby, and they can learn how Tcl works in a week and learn how the program is structured over a few weeks and start making small changes.
Sure, but why would they want to? They'd be wasting time on a skill that adds exactly $0 to their market value.
The longer they stay with you, spending time getting experienced with Tcl (and not with some other popular language), the less they are worth.
> I'm skeptical that this training overhead would really cost more than replacing the whole system if it really was working well.
For the company, no. For the developer, most developers tend to shy away from dead-end tech stacks. If you get someone close to retirement (10 years away), they'd be happy to use your dead-end tech stack, because to them their current market value gets more irrelevant with each passing day.
> You're looking at it soley from the PoV of the company.
No, I'm not. I personally am not at all concerned about "dead-end tech stacks" because the kind of companies that care that you have specific skills rather than overall engineering competence are not the ones I want to work for. I'm sure I'm not the only one.
I already agreed that "people in general" don't want that. I don't see how that refutes my argument that enough people don't think that way that it might have been cheaper to pay such a person.
People were leaving the company, and recruiters were having difficulty bringing great programmers in at the price that they were willing to pay. (I could argue it would have been cheaper to pay programmers a premium to work in the technology, given how much money it cost to rebuild the system and how much more in infrastructure costs the new microservices system cost relative to one production server and one (Oracle) database.)
And as people left, the theory of the program was being lost, turning it into "legacy code." There were other problems as well - in particular, the initial data structures that were coupled to billing made it difficult to try other business models, and the industry was starting to innovate with point to point in other markets. Those were solvable in Tcl, though. The real problem was people.
Great article. I think the key differentiator in all of these stories is the "stickiness" of their customers. Netscape's customers could switch at no cost (apart from migrating bookmarks, maybe?) and did. Basecamp's were much more invested in the product and 37Signals never abandoned them during the time they starting building its replacement.
I get the impression (having never used it) that Fogbugz's users liked it but that Fog Creek couldn't win new larger customers once Jira arrived on the scene. And since it was a shrink-wrapped one-time purchase FC couldn't fund a paydown of their tech debt.
Netscape will forever be remember though for this:
"Less than two years after it was released, the company’s $3-billion IPO launched the dot-com era."
This. This is why many of us are here reading this today.
It's interesting to now be at a point where we see a generation of SaaS make way for the next generation of SaaS.
I've come to realise that realistically, platforms -do- change, and eventually you will need to either commit to a rewrite (even if you choose to do it by spinning up a whole new company like Freshbooks did).
As an example, a SaaS app from 2010 might have been barely responsive, might have been put together with jQuery, and the backend might've been written in a framework that died off in the meantime. And while you can do a lot of little improvements and refactors here and there, ultimately the foundation on which you've built your system gets to a point where it slows you down more than it benefits you, and there's no real way forward.
The stories in the article make me think that the old adage of "never rewrite" is really more like "eventually you will either rewrite or die".
The old adage is more “never do a big bang rewrite”, where the launch of v2 is one big step. An incremental rewrite, where perhaps you change a single component or bit of the UI, is always necessary. Although even there people moan about “change fatigue”.
In a large enough system a full rewrite becomes impractical anyway, because the team is sized to incrementally grow not replace whole cloth. The team simply isn’t capable of rewriting the whole codebase in one big bang release in anything less than years.
You can bring up a larger team for the rebuild but that introduces its own risk, or you can leave customers out in the cold for years, like netscape did.
Continuous rewrites are the only way to handle that kind of system, just like in a large enough city just to keep up with natural decay it becomes necessary to always have construction and road works projects going on.
I’m a Tech Lead for a SaaS product that’s been around since ~2010. Four or so years ago we stared piecemeal transitioning away from jQuery/Backbone to React. Nearly all of our main UI is React now, but it’s still pretty coupled to Backbone. There’s a lot more risk removing that, so we’ve not had enough of a reason to yet.
Basically the high touch (from a developer point) areas of the app are always the most modern and the first place to get whatever treatment we transition to (React, typescript etc) and there’s some really untouched areas that might remain prehistoric for years, and that’s fine.
Its worked pretty well for us, and I can’t see why it wouldn’t continue that way.
I'd call that a continuous rewrite, which is also the strategy that worked best for me. If you're not willing to make these investments as you go, you may well end up in a full-rewrite-or-die situation, I suppose. Comes down to how aggressively management prioritises short term results. I personally have never worked on a code base that far gone, but I can imagine they're out there.
The big risk of letting the high touch areas be what get migrated to the modern stack, is that for many companies this is largely driven by the fashion of new hires. And letting new hires drive the technical choices isn't necessarily bad, but you do have a major risk of them not understanding why things were there.
I'd say, historically, you were also lucky in the timing. The last 4 years have been absolutely rock solid for frontend development, compared to the time before. Still way more fluid than I personally think it should be, but I'm willing to ack that things have gotten a lot more stable.
Oddly, the backend has gotten worse in these years. The amount of service related churn that has happened in the past decade has been impressive. And mostly not in a good way. I can hope that we are going to be a bit more stable now.
High touch in the sense of being driven by product development. We let everyone make technical choices, sometimes autonomously but sometimes as a pitch. A part of technical leadership in smaller organisations is mitigating a lot of that “new hire/shiny thing” risk.
We used react a lot externally to the main product as early as 2014, but didn’t introduce it into the main product until we were comfortable.
Apologies if it felt like I was questioning your choice. I meant my post as an addition.
Using things outside the critical path is definitely the right choice. Making things to throw away is also great.
I do get worried, sometimes, that it is also correct to let the new coders pick the styles and frameworks. Letting them choose is a huge boon to their productivity. And letting them advance on a bet like this can payoff really well. Just expect losses along the way.
Great article! Something I was wondering about when reading this is how these lessons could apply to software that depends on its userbase.
As an example, let’s use Discord; what if in 10 years the Discord codebase is seen as obsolete and requires a re-write? You can’t really make a “Discord 2”, because now you split up your userbase between Discord 1 and Discord 2.
In this re-write you would have to make sure that you aren’t splitting up the community that you rely on, since it is part of the product offering (it is basically a social medium).
I briefly worked in the (Swedish) public sector, was on a team tasked to replace an old system written partially in Cobol and partially in some IBM-tech nobody understood anymore.
The system was huge and poorly understood, and a lot of people used it to do most of their work. Extremely critical system for the very fabric of society.
There had been several previous attempts at replacing it. They'd sort of fizzled out along the way, so they were stuck with several half-finished iterations of replacements that were in used in parallel with the old system.
This was it though. Hundreds of people. Micro services. Modern JS frameworks. C4 architecture and hexagonal design and cloud native and every goddamn buzzword that's appeared in a programming blog the last 10 years. Just get it all in there! Everything the cool start-up kids were using needed to be apart of this. It wasn't clear exactly how it was going to better, but it sure was gonna be modern.
I didn't see the project through but I could see where it was heading clear as day. It was only a year in and maintenance was already becoming a problem. Shit was falling apart. It was a nightmare to debug. Nobody knew how the heck all the services quite communicated.
I'll eat my shoe if they don't spend the next 25 years fixing leaky pipes with this microservice mess bolted onto the previous iterations as an even more grotesque programming-cronenberg.
I think part of the problem was they needed to pitch the project to the government to get it funded. Probably easier to sell the idea of the Homer Simpson Model S[1] than what they actually needed.
That, and everyone with any sort of deciding power were technology-illiterate bureaucrats who waste no opportunity to try to make us somehow use AI to somehow enhance this system in some fashion, largely so they could announce to the world that their system use AI somehow.
I think the problem is not with the latest technology, the problem is with the inappropriate one.
Have they chose golang, or (God forbid) Rust, they will most likely be fine.
There are choices, which are very dangerous, as they increase complexity dramatically, like micro-services, or "event-driven" architectures, or Kubernetes. Add all three and good luck lifting project off the ground.
I had a similar experience in the Norwegian (hello, neighbour!) public sector, and it was _also_ a rewrite from a partly COBOL-based solution. The difference is that we succeeded, which is - if you're Norwegian - almost unheard of. :)
The whole thing was rebuilt using new tech; Java for the most part, cloud-ish hosting (our "own" bare metal servers and OpenShift via a Norwegian company), and this stack was never a problem. In fact, not much was a problem, because:
The old system was _riddled_ with unit tests that could easily be replicated in any language. All of the bugs and/or minor issues over the last 30 years had been documented and had unit tests covering them. This was what saved the rewrite. Our rewrite could rely a lot on TDD based on those tests, which made it possible for us to trap problems early on.
Without those tests - written by programmers who probably are long dead - the rewrite would never have succeeded, or at least not in the time frame given, and at least not on budget.
Since then, whenever our consulting company is involved in anything that has to do with existing software, we ask the client about how their test coverage is;
* None, but you will give us $1 million per hour? Nah. Doesn't matter. Find someone else.
* Lots, and you'll pay us $1,000 per hour? Tell us more.
So: create (unit) tests for _everything_. Fixed a bug? Create a test for it. Changed some behaviour? Create a test for it. Removed a feature? Create a test for it.
EDIT: Final stack looked like this:
* Self-ish-hosted servers (non-Norwegian cloud providers was not an alternative because of privacy issues).
* Java as main programming language on the backend, but many services was written in C#.
* Frontend didn't have to be fancy, so some simple JS-stuff there; think auto-complete, no fancy DOM-handling or anything like that. Application as a whole was very server-centric.
From what I've heard, they are now trying to get away from Java, and move more and more over to C#. I just hope they keep the unit tests. :)
This is such a good approach. A former colleague of mine built an entire scaffolding of integration tests surrounding a test-free system that needed to have a large fraction rewritten. Mocking clients, collaborators and everything.
The tests were only ever used once for that rewrite (the customer terminated the contract shortly after the rewrite was complete, and the entire codebase was thrown into the rubbish bin) but I still think it was worth it. The tests didn't even run that many times because most of them passed at the first attempt but I suspect the effort of enumerating cases had a strong impact on the successful design of the rewrite.
Spend the first four hours sharpening the axe, and so on.
The first thing I do with any bug report is check for or write a test reproducing it. I’m entirely bewildered when programmers don’t do this. It saves time in so many ways, from explaining PRs to refactors or getting future colleagues up to speed.
> The old system was _riddled_ with unit tests that could easily be replicated in any language. All of the bugs and/or minor issues over the last 30 years had been documented and had unit tests covering them. This was what saved the rewrite. Our rewrite could rely a lot on TDD based on those tests, which made it possible for us to trap problems early on.
This is the crux of it. The hard part of a rewrite or major refactor is not building out the new stack from scratch using all the latest tech. The hard part is getting the initial system in a place where it can be rewritten. This includes -
- Adding robust testing, both unit and integration/system/functional.
- Working, functional CI/CD pipelines that execute these tests. I have encountered so many legacy projects without CI pipelines, some of which even had tests but were just expected to fail. Not marked in the code as expected to fail, mind you, just expected to fail by developers, who shrugged and only ran tests they thought were relevant to their code change.
- Code cleanup, particularly if you're doing a slow migration by functionality (which you should be). If you're making a "user microservice," you need to structure the code such that all user-related functionality is isolated and written in a way that works as a microservice, before you write a single line of the microservice itself.
- Infrastructure to do parallel implementation[0], including comparison and reporting of differences in results.
The main problem with all of this is that it's boring. Rewrites are almost always, of course, advocated by developers who have to deal with the code on a regular basis. If you tell developers that they can do a rewrite, but first they have to spend 6 months writing an integration testing framework and fix all of the broken unit tests, most won't go for it. They want to jump directly into using Kubernetes on the cloud with Erlang.
The problem compounds when you look at the wider industry, because honestly, it's often a great career move for the developer who starts up a rewrite with Kubernetes and Erlang. They can stick around for 6-18 months, put "Rewrote legacy system using $HYPEDTECH" on their resume, and start shopping around. Doesn't matter if the rewrite ends up being a dumpster fire, which it normally is. That's a much better story for your career, unfortunately, than talking about how you deeply understood the domain and the quirks of the classic ASP code powering your current company's most critical software.
One of the things I've had my nose rubbed in several times is just how very effective some of of those "legacy" languages - like Cobol or the various 4GL's - are for writing business systems apps.
It clicked for me when I started thinking of them as DSL's, where the "domain" is "business stuff" ( or equivalent, government data processing etc )
Of course they have limitations - but they're also often part of a full platform and very opinionated as to app and data model, and within those constraints programmers can be crazy productive.
So many of these rewrites fail, and I think a big part of it is people don't fully realise the ramifications of the fact that they're replacing both a platform and a system built on that platform.
Sure, there's probably a wizzbang new architecture, which well intentioned smart people will be able to prove improves on that platform in every way - with reams of documentation about the choices made and the way things work... but they're still choices, and sometime they'll get misunderstood, or ignored, or subverted by people who think they know better or who just need a short cut - and it's often big enough project that staff turnover is actually a significant factor too, so opinions may shift.
But many of the architecture/data model decisions are just built in to those older languages.
The architecture isn't subject to the same drift pressures because they're not choices, and that makes the system less complicated overall, and especially over time.
Of course limitations of the various language implementations, as well as the very human distaste of working within a constrained system is probably a big part of the reason that these legacy languages effectively died.
But I think we lost something valuable, without many of us ever realising.
I think the question reveals a secret that nobody talks about.
Your software is not understandable to others, so it's impossible to adapt.
Your software needs high quality documentation about the mental models used to create and think of the software. But too many developers think the code is the documentation, meaning their code is legacy as written.
can't agree more as somebody who joined a place for the sole purpose of maintaining already profitable products and continuously pay the tech debt there.
I keep emphasizing it, looking at this project is like looking at a complied code, you can understand what this portion does by reading the program but You can't understand what the dev/architect had in mind when they did it, architectural diagrams + proper high level docs can do wonders. I can understand what those modules are doing, but it will take me years before I can grasp the entire system to know where to look for every fix/new increment.
That is so, so true. So many open source software that I'd like to be able to contribute to, if only they would put a Markdown file documenting at a high level how the code is organized, how data flows through it... Instead all we have is the code itself, sometimes some instructions for running the tests and that's it. Then every contributor needs to go through all of the code and reverse engineer it to even start.
One of the great software rewrite battles that I know of is computer graphics sector.
It was mid-90s and Silicone Graphics (it hasnt rebranded to SGI yet) was the dominant hardware, and Wintel were too slow.
In hollywood the flagship products were Alias PowerAnimator, Wavefront Visualiser, Thomson Digital Exploer, Softimage 3D for animation, Sidefx Prisms for particles/effects. PowerAnimator had the best modeling tools like NURBS. Softimage had best animation tools. None of these ran on Windows and all were running on SGI's own IRIX. Microsoft bought Softimage in 94 and ported it to Windows NT. Suddenly Silicon Graphics saw Wintel as serious threat. So they bought and merged Alias,Wavefront, TDI to form Alias|Wavefront and began to write Maya which would combine the technologies from all three. Softimage also started to write its new one codenamed at that time Sumatra which would eventually release as XSI. The race was on, and the two big giants SGI and Microsoft were having a proxy war now. Everyone was keen on seeing who is going to come out first and best product.
Meanwhile Sidefx was also writing something new from scratc which they eventually released as Houdini in Dec 96. Sidefx has always been a smaller niche player and has stayed true to that till today.
Maya 1.0 came out in Feb 98 and it was immense success. A lot of hollywood studios migrated to Maya because Sumatra was nowhwere near release. Eventually XSI came out around 2000 I think and it was too late, but not too little. XSI still had more powerful animation capabilities but they had lost a lot of ground to Maya interms of mindshare and customers.
Eventually Microsoft sold Softimage/XSI to Avid. In the mean time 3D Studio from Autodesk was not a serious player when it came to hollywood stuff. So Autodesk bought Softimage from Avid. Unfortunately SGI also had real financial trouble due to Intel finally catching up interms of speed. They also sold Alias|Wavefront to Autodesk. So now Autodesk had the two major 3D graphics packages used in hollywood. Autodesk eventually killed XSI and continued Maya at their high end, and 3D Studio Max as their lower/medium end product.
I recall reading an article about one of the database engines (maybe MySQL or MariaDB) and how it was created as a drop in replacement using test driven development. Can’t find it… anyone know of an article like that?
I personally don’t have a great handle on how to reason about anticipated maintenance costs for a given project, yet would like to. Some simple notions: core libraries should be written with few dependencies so as to reduce update impetus surface area, simplicity of method is often better. But that’s as far as I’ve got. If building something to last (the dream) probably would be better to solidify this further.
A few times in my career I have been in situation where decision to rewrite was made.
Worst case was where decision was made, resources had been invested and then it was an ongoing theme for years to come (this particular company I have in mind is still talking about next version 6 years after initial resources were invested, I was the one to build foundation system which works to date even though I left 4 years ago)
Best case was in the company I was contracting for, they had huge and almost 20 years old pearl codebase which they decided to replace with python ecosystem. In 6 months the company was already actively switching to the new system, exposing bugs early etc.
In my view each time management is where it can result in good or bad outcome.
I really wonder Spolsky didn't just do what Golang did and use the transpiled code as a basis for the continued work. Of course this predates the success of Golang but it's not a very far fetched idea, I bet Jeff Atwood would've suggested it.
Of course happy they didn't because if they did Fogbugz would've had an incremental upgrade path on PHP all the way into modernity and would have been an acceptable thing to work on, preventing Trello from ever existing most likely.
One thing to note is that you absolutely can fully replace a legacy codebase a piece at a time, the big risk is when you have a completely incompatible “rewritten version” of the product that you will cut over to at some later time.
If you do something like build a facade in front of the old service, and gradually move functionality into the new layer, this can be seamless for users.
But of course, this doesn’t give you the ability to revisit product functionality, like the Basecamp rewrites. I think those are really great examples of how to continue to innovate and experiment, while keeping your old customers happy, and the key point there would be “rewrites can be done safely if you commit to keeping older versions around”. Basically treating the versions as distinct products, instead of the rewrite replacing the old version.
57 comments
[ 3.0 ms ] story [ 120 ms ] threadWhen you should rewrite software from scratch and what should be kept in mind doing it.
[0]: https://www.joelonsoftware.com/2000/04/06/things-you-should-...
I'm convinced that the way forward is not to refactor or rewrite it, but to produce something new and different that draws on the 30 years of experience learned during the development of the original product. And, like Basecamp, run them side-by-side and allow users to chose what they want.
Are the original authors still there or do you have insanely good documentation?
I think the hardest conversation circles around language choice. One problem with choosing innovative languages is that you can end up on a dead end. Zipcar, for example, used Tcl. While I am a fan of Tcl, it became obvious around 2012ish (IIRC) that this was not the future. There was a first attempt at a migration using rails, but that ended up being ditched for a JVM stack. (I don't know if most of it is still in groovy, but there were also pieces in ruby, kotlin, and java.) We also had a large number of performance and billing problems we needed to solve, and as we were guaranteeing compatibility and interoperability (a big bang switchover was too risky for many reasons), we had a lot of performance and usability issues for years.
It cost far more than management expected in the outset for many reasons, but at the same time, staying on Tcl was unsustainable for developer recruitment and retention purposes. It's something that biases me toward blub languages now as a manager unless you have a very good reason to innovate there.
I agree that this is a real phenomenon, and I find fascinating because it illustrates the role of fashion in software development. TCL may have been quirky compared to other languages, but actual working software was successfully developed in it by more-or-less typical developers. Yet it failed anyway because it wasn’t “cool”.
Not being trendy meant that fewer brains were thinking about it, evolving it by improving tooling and writing libraries, producing training materials and evangelizing it. Ultimately it just stagnates.
I think it does speak to the amount of labor that goes into making a language viable as an option, though. Java had paid labor. Tcl didn't have enough unpaid labor to succeed while python did.
First time I’ve heard of tcl being used outside of an FPGA or EDA toolchain. Far out.
I'm not sure I get this. Any competent programmer can learn Tcl. So find one versed in scripting languages like Python or Ruby, and they can learn how Tcl works in a week and learn how the program is structured over a few weeks and start making small changes. I'm skeptical that this training overhead would really cost more than replacing the whole system if it really was working well.
You're looking at it soley from the PoV of the company.
> Any competent programmer can learn Tcl. So find one versed in scripting languages like Python or Ruby, and they can learn how Tcl works in a week and learn how the program is structured over a few weeks and start making small changes.
Sure, but why would they want to? They'd be wasting time on a skill that adds exactly $0 to their market value.
The longer they stay with you, spending time getting experienced with Tcl (and not with some other popular language), the less they are worth.
> I'm skeptical that this training overhead would really cost more than replacing the whole system if it really was working well.
For the company, no. For the developer, most developers tend to shy away from dead-end tech stacks. If you get someone close to retirement (10 years away), they'd be happy to use your dead-end tech stack, because to them their current market value gets more irrelevant with each passing day.
No, I'm not. I personally am not at all concerned about "dead-end tech stacks" because the kind of companies that care that you have specific skills rather than overall engineering competence are not the ones I want to work for. I'm sure I'm not the only one.
And as people left, the theory of the program was being lost, turning it into "legacy code." There were other problems as well - in particular, the initial data structures that were coupled to billing made it difficult to try other business models, and the industry was starting to innovate with point to point in other markets. Those were solvable in Tcl, though. The real problem was people.
Right, it's always something like that. Thanks for clearing that up.
I get the impression (having never used it) that Fogbugz's users liked it but that Fog Creek couldn't win new larger customers once Jira arrived on the scene. And since it was a shrink-wrapped one-time purchase FC couldn't fund a paydown of their tech debt.
Netscape will forever be remember though for this:
This. This is why many of us are here reading this today.I've come to realise that realistically, platforms -do- change, and eventually you will need to either commit to a rewrite (even if you choose to do it by spinning up a whole new company like Freshbooks did).
As an example, a SaaS app from 2010 might have been barely responsive, might have been put together with jQuery, and the backend might've been written in a framework that died off in the meantime. And while you can do a lot of little improvements and refactors here and there, ultimately the foundation on which you've built your system gets to a point where it slows you down more than it benefits you, and there's no real way forward.
The stories in the article make me think that the old adage of "never rewrite" is really more like "eventually you will either rewrite or die".
You can bring up a larger team for the rebuild but that introduces its own risk, or you can leave customers out in the cold for years, like netscape did.
Continuous rewrites are the only way to handle that kind of system, just like in a large enough city just to keep up with natural decay it becomes necessary to always have construction and road works projects going on.
Basically the high touch (from a developer point) areas of the app are always the most modern and the first place to get whatever treatment we transition to (React, typescript etc) and there’s some really untouched areas that might remain prehistoric for years, and that’s fine.
Its worked pretty well for us, and I can’t see why it wouldn’t continue that way.
I'd say, historically, you were also lucky in the timing. The last 4 years have been absolutely rock solid for frontend development, compared to the time before. Still way more fluid than I personally think it should be, but I'm willing to ack that things have gotten a lot more stable.
Oddly, the backend has gotten worse in these years. The amount of service related churn that has happened in the past decade has been impressive. And mostly not in a good way. I can hope that we are going to be a bit more stable now.
We used react a lot externally to the main product as early as 2014, but didn’t introduce it into the main product until we were comfortable.
Using things outside the critical path is definitely the right choice. Making things to throw away is also great.
I do get worried, sometimes, that it is also correct to let the new coders pick the styles and frameworks. Letting them choose is a huge boon to their productivity. And letting them advance on a bet like this can payoff really well. Just expect losses along the way.
As an example, let’s use Discord; what if in 10 years the Discord codebase is seen as obsolete and requires a re-write? You can’t really make a “Discord 2”, because now you split up your userbase between Discord 1 and Discord 2.
In this re-write you would have to make sure that you aren’t splitting up the community that you rely on, since it is part of the product offering (it is basically a social medium).
The system was huge and poorly understood, and a lot of people used it to do most of their work. Extremely critical system for the very fabric of society.
There had been several previous attempts at replacing it. They'd sort of fizzled out along the way, so they were stuck with several half-finished iterations of replacements that were in used in parallel with the old system.
This was it though. Hundreds of people. Micro services. Modern JS frameworks. C4 architecture and hexagonal design and cloud native and every goddamn buzzword that's appeared in a programming blog the last 10 years. Just get it all in there! Everything the cool start-up kids were using needed to be apart of this. It wasn't clear exactly how it was going to better, but it sure was gonna be modern.
I didn't see the project through but I could see where it was heading clear as day. It was only a year in and maintenance was already becoming a problem. Shit was falling apart. It was a nightmare to debug. Nobody knew how the heck all the services quite communicated.
I'll eat my shoe if they don't spend the next 25 years fixing leaky pipes with this microservice mess bolted onto the previous iterations as an even more grotesque programming-cronenberg.
If you're going to replace a creaky old legacy system, never ever do it with the latest technology. Do it with tried and boring technology.
Otherwise soon enough you will have two problems.
That, and everyone with any sort of deciding power were technology-illiterate bureaucrats who waste no opportunity to try to make us somehow use AI to somehow enhance this system in some fashion, largely so they could announce to the world that their system use AI somehow.
[1] https://imgur.com/97lhWVR
Have they chose golang, or (God forbid) Rust, they will most likely be fine.
There are choices, which are very dangerous, as they increase complexity dramatically, like micro-services, or "event-driven" architectures, or Kubernetes. Add all three and good luck lifting project off the ground.
The whole thing was rebuilt using new tech; Java for the most part, cloud-ish hosting (our "own" bare metal servers and OpenShift via a Norwegian company), and this stack was never a problem. In fact, not much was a problem, because:
The old system was _riddled_ with unit tests that could easily be replicated in any language. All of the bugs and/or minor issues over the last 30 years had been documented and had unit tests covering them. This was what saved the rewrite. Our rewrite could rely a lot on TDD based on those tests, which made it possible for us to trap problems early on.
Without those tests - written by programmers who probably are long dead - the rewrite would never have succeeded, or at least not in the time frame given, and at least not on budget.
Since then, whenever our consulting company is involved in anything that has to do with existing software, we ask the client about how their test coverage is;
* None, but you will give us $1 million per hour? Nah. Doesn't matter. Find someone else.
* Lots, and you'll pay us $1,000 per hour? Tell us more.
So: create (unit) tests for _everything_. Fixed a bug? Create a test for it. Changed some behaviour? Create a test for it. Removed a feature? Create a test for it.
EDIT: Final stack looked like this:
* Self-ish-hosted servers (non-Norwegian cloud providers was not an alternative because of privacy issues).
* Java as main programming language on the backend, but many services was written in C#.
* Frontend didn't have to be fancy, so some simple JS-stuff there; think auto-complete, no fancy DOM-handling or anything like that. Application as a whole was very server-centric.
From what I've heard, they are now trying to get away from Java, and move more and more over to C#. I just hope they keep the unit tests. :)
The tests were only ever used once for that rewrite (the customer terminated the contract shortly after the rewrite was complete, and the entire codebase was thrown into the rubbish bin) but I still think it was worth it. The tests didn't even run that many times because most of them passed at the first attempt but I suspect the effort of enumerating cases had a strong impact on the successful design of the rewrite.
Spend the first four hours sharpening the axe, and so on.
Great analogy. I'll remember that one.
Also, after some DNS-digging just now, it seems like they have changed to a different (Norwegian) provider.
This is the crux of it. The hard part of a rewrite or major refactor is not building out the new stack from scratch using all the latest tech. The hard part is getting the initial system in a place where it can be rewritten. This includes -
- Adding robust testing, both unit and integration/system/functional.
- Working, functional CI/CD pipelines that execute these tests. I have encountered so many legacy projects without CI pipelines, some of which even had tests but were just expected to fail. Not marked in the code as expected to fail, mind you, just expected to fail by developers, who shrugged and only ran tests they thought were relevant to their code change.
- Code cleanup, particularly if you're doing a slow migration by functionality (which you should be). If you're making a "user microservice," you need to structure the code such that all user-related functionality is isolated and written in a way that works as a microservice, before you write a single line of the microservice itself.
- Infrastructure to do parallel implementation[0], including comparison and reporting of differences in results.
The main problem with all of this is that it's boring. Rewrites are almost always, of course, advocated by developers who have to deal with the code on a regular basis. If you tell developers that they can do a rewrite, but first they have to spend 6 months writing an integration testing framework and fix all of the broken unit tests, most won't go for it. They want to jump directly into using Kubernetes on the cloud with Erlang.
The problem compounds when you look at the wider industry, because honestly, it's often a great career move for the developer who starts up a rewrite with Kubernetes and Erlang. They can stick around for 6-18 months, put "Rewrote legacy system using $HYPEDTECH" on their resume, and start shopping around. Doesn't matter if the rewrite ends up being a dumpster fire, which it normally is. That's a much better story for your career, unfortunately, than talking about how you deeply understood the domain and the quirks of the classic ASP code powering your current company's most critical software.
[0] http://sevangelatos.com/john-carmack-on-parallel-implementat...
It clicked for me when I started thinking of them as DSL's, where the "domain" is "business stuff" ( or equivalent, government data processing etc ) Of course they have limitations - but they're also often part of a full platform and very opinionated as to app and data model, and within those constraints programmers can be crazy productive.
So many of these rewrites fail, and I think a big part of it is people don't fully realise the ramifications of the fact that they're replacing both a platform and a system built on that platform. Sure, there's probably a wizzbang new architecture, which well intentioned smart people will be able to prove improves on that platform in every way - with reams of documentation about the choices made and the way things work... but they're still choices, and sometime they'll get misunderstood, or ignored, or subverted by people who think they know better or who just need a short cut - and it's often big enough project that staff turnover is actually a significant factor too, so opinions may shift. But many of the architecture/data model decisions are just built in to those older languages. The architecture isn't subject to the same drift pressures because they're not choices, and that makes the system less complicated overall, and especially over time.
Of course limitations of the various language implementations, as well as the very human distaste of working within a constrained system is probably a big part of the reason that these legacy languages effectively died.
But I think we lost something valuable, without many of us ever realising.
Your software is not understandable to others, so it's impossible to adapt.
Your software needs high quality documentation about the mental models used to create and think of the software. But too many developers think the code is the documentation, meaning their code is legacy as written.
I keep emphasizing it, looking at this project is like looking at a complied code, you can understand what this portion does by reading the program but You can't understand what the dev/architect had in mind when they did it, architectural diagrams + proper high level docs can do wonders. I can understand what those modules are doing, but it will take me years before I can grasp the entire system to know where to look for every fix/new increment.
In hollywood the flagship products were Alias PowerAnimator, Wavefront Visualiser, Thomson Digital Exploer, Softimage 3D for animation, Sidefx Prisms for particles/effects. PowerAnimator had the best modeling tools like NURBS. Softimage had best animation tools. None of these ran on Windows and all were running on SGI's own IRIX. Microsoft bought Softimage in 94 and ported it to Windows NT. Suddenly Silicon Graphics saw Wintel as serious threat. So they bought and merged Alias,Wavefront, TDI to form Alias|Wavefront and began to write Maya which would combine the technologies from all three. Softimage also started to write its new one codenamed at that time Sumatra which would eventually release as XSI. The race was on, and the two big giants SGI and Microsoft were having a proxy war now. Everyone was keen on seeing who is going to come out first and best product.
Meanwhile Sidefx was also writing something new from scratc which they eventually released as Houdini in Dec 96. Sidefx has always been a smaller niche player and has stayed true to that till today.
Maya 1.0 came out in Feb 98 and it was immense success. A lot of hollywood studios migrated to Maya because Sumatra was nowhwere near release. Eventually XSI came out around 2000 I think and it was too late, but not too little. XSI still had more powerful animation capabilities but they had lost a lot of ground to Maya interms of mindshare and customers.
Eventually Microsoft sold Softimage/XSI to Avid. In the mean time 3D Studio from Autodesk was not a serious player when it came to hollywood stuff. So Autodesk bought Softimage from Avid. Unfortunately SGI also had real financial trouble due to Intel finally catching up interms of speed. They also sold Alias|Wavefront to Autodesk. So now Autodesk had the two major 3D graphics packages used in hollywood. Autodesk eventually killed XSI and continued Maya at their high end, and 3D Studio Max as their lower/medium end product.
No 'e'. Silicon is for computers, Silicone[1] is for breast enlargements.
[1] Insert jokes about Silicone Valley here.
Vesting.
That is a bit harsh on the new team that just inherited someone else's mess and the One Guy who understood it is not there any more.
Worst case was where decision was made, resources had been invested and then it was an ongoing theme for years to come (this particular company I have in mind is still talking about next version 6 years after initial resources were invested, I was the one to build foundation system which works to date even though I left 4 years ago)
Best case was in the company I was contracting for, they had huge and almost 20 years old pearl codebase which they decided to replace with python ecosystem. In 6 months the company was already actively switching to the new system, exposing bugs early etc.
In my view each time management is where it can result in good or bad outcome.
Of course happy they didn't because if they did Fogbugz would've had an incremental upgrade path on PHP all the way into modernity and would have been an acceptable thing to work on, preventing Trello from ever existing most likely.
If you do something like build a facade in front of the old service, and gradually move functionality into the new layer, this can be seamless for users.
But of course, this doesn’t give you the ability to revisit product functionality, like the Basecamp rewrites. I think those are really great examples of how to continue to innovate and experiment, while keeping your old customers happy, and the key point there would be “rewrites can be done safely if you commit to keeping older versions around”. Basically treating the versions as distinct products, instead of the rewrite replacing the old version.