Corollary: pick a starting point, and add a shit-ton of logging to trace what is actually going on. Hide it under #if DEBUG if necessary, and hope that you don't get a Schrodinger's Code situation where observing the code changes how it behaves.
I've had a few more-or-less realtime multithreaded projects that I've worked on where you can't really put a debugger on a system and halt it, without breaking things in interesting and misleading ways, and the only good option is to fall back to ye olde printf debugging.
Many times. Usually it wasn't a whole system but a component or library licensed from someone else, where "someone else" was unresponsive or out of business. In one case, the guy who'd written it used to go on month-long wilderness expeditions during which he was completely unreachable.
The solution almost always involved a certain amount of reverse engineering, and damn sure I always tried to advocate getting away from software we couldn't maintain effectively. The third leg of that tripod is isolation. Reduce the number of things that depend on the code, and the number of ways that they use it. If nothing else, that will reduce your exposure. It will also tell you what code paths are important to understand and which are not. Finally, it can help guide implementation of tests, or of a replacement.
If you're really stuck with such a piece of code, some of the advice in a blog post I wrote about a similar challenge might apply.
That's about learning a codebase that's notable mainly for being large, even if the original developers are as available and helpful as could be, but looking at it now I see quite a bit that applies to this case as well.
Maybe a little more clarification? When you say that nobody on the team can understand it, are you primarily talking about code complexity, code style, or language?
Many years ago, I became the designated maintainer for a legacy inventory system that was used for internal audit purposes. The system ran on a PDP-11/70, and was written in a combination of Basic-Plus (this was a long time ago) and COBOL. I only had a passing familiarity with either of those languages. I find the challenges of learning a code base usually boil down to a few recurring problem areas:
* Problem domain knowledge: The system was used to track leased telecommunication facilities, and had a lot of obscure business logic built into it. At least half the challenge was reverse engineering business logic from the code.
* Coding style: I always find it challenging to get comfortable with another developer's style. Mismatches in assumptions/preferred approaches can make it really hard to get comfortable with someone else's code. This particular system had some truly weird programming choices, including a "screen driver" (similar to curses) written in COBOL.
* Code complexity: I was pretty lucky that the code was not very complicated. With all the other challenges, if the code had been complex it would have probably been an impossible challenge.
* Language knowledge: the original developers had used some features of Basic-Plus and COBOL that were a little obscure, which made understanding the code base that much harder.
Yes. It was a vital subsystem in a Smalltalk program. While most of the project had passable Object Oriented organization, this one subsystem had zero instance methods and zero instance variables. Instead, it was copy-pasta after slightly modified copy-pasta of these long methods that called each other recursively. Each one of these methods used a "merging" style algorithm that incremented 4 indexes into arrays, all the while executing deeply nested conditional logic.
There were some smart cookies on the team, but we were all in fear of this code. The person who wrote it spent her days sitting in the cafe downstairs, reading novels. She'd check some logs, and occasionally come and yell at us for doing something wrong, which she would never explain. It turns out that the system had objects, but they were all embodied by consecutive spans of entries in those arrays. We went on a trek through the bowels of this corporation, asking around for documentation of the 3rd party software that used those sequences in those arrays, but no one had it, and that company no longer existed. If you tried to explain to her that Object Oriented code would have instance methods, she'd always bring up her PhD in math.
A coworker of mine spent a week charting out one of those methods, and managed to rewrite it in 1/6th the line count, with no errors. However, that didn't really help, as it still implemented the same weird merging algorithm.
Cool. I don't think I've ever heard a war story about a SmallTalk codebase before -- I have always gotten the impression that SmallTalk was a research/toy language similar to Haskell that is held in high regard but rarely used in production.
I’ve always found that if you stare at something long enough you can eventually understand it, and break it down into its core components. If your business depends on it, don’t be so quick to just rewrite it and abandon because it likely will have a ton of hardened edge cases that are baked into it that the business relies upon.
You must study the code and document what you can. Reverse engineering it is sometimes the only option in your quest to understand it. Then you’ll need to make decisions about how to move forward given the constraints of the business whether it be time, resources, other priorities. There is no one size fits all answer but don’t be shy about vocalizing the risks and making sure the business knows the pain points. Sometimes you have to sell the problem you now have and get buy-in to really do something to fix the mess.
One thing I’ve realized is that when people don’t take the time to document and write clean code sometimes it boils down to their idea of job security.
Other times it turns out the problem is there is no single owner of this code base and it’s been hacked to death.
Lastly when something is ugly, enough and messy enough...then you can just call it proprietary technology...I kid I kid.
I’m a consultant and make a living saving bad projects. That’s literally why I get phone calls for work. Keep in mind that I work in high level modern languages, I’m sure there’s some crazy proprietary cpu running a robot in a Detroit factory.
In any case there’s never been something that I’ve run into that I’ve not figured out. It takes time, and the hard part usually is not figuring out what it does but the weird edge case of the moon aligning with Venus and then the output suddenly changes. This is why understanding the requirements is more important than the code. I don’t care if the code is bad if I can more or less write a test case against it and make sure it does that.
That said complete rewrites never happen. It usually is only rewriting portions when that is cheaper than fixing. It is the if it isn’t broke don’t fix it adage.
The only time ever I’ve been stuck was when I saw a proprietary software implantation on top of a custom software package ontop of Solr (technically ontop of the JVM) create a large object heap issue ontop of a proprietary OS (Windows). It wasn’t code related I diagnosed that it was a GC issue, but it wasn’t in any code I had source to. A Windows update ended up fixing it. And this is why working with enterprise software is hard.
Have you used input/output proxy logging shims in these situations? They've been so invaluable to be in finding those weird edge cases that come up in 'orphaned software' projects as well as 'ancient legacy platform' migrations ;)
Do you have any specialized tools (code navigation tools, for example) that you use when first encountering these large piles of code? I'd love to hear some recommendations; I have to deal with large (only sometimes bad, but always large) piles of vendor code. I'm currently staring a pile of 900kloc of pretty nice code but it's a /lot/ of code.
Not OP, but I imagine this is fairly language specific. This is where Java shines. Keep in mind I mostly work with services not applications.
My process looks like this:
Step one: Identify sources of reflection, this is the triskyest. Hopefully the only dependencies are open source, so you generally know what they do and grep can usually find the rest.
Step Two: go code spelunking. Find your entry points. Find your main() or framework equivalent. Find callsites for rest endpoints, rpc, jmx, etc.
Step three: find other "external request processing" endpoints. Do you have timer threads? Reading a Kafka stream and acting per record? Etc.
Once you understand those, you can interpret where most any stacktrace is from. Good old Intellij or Eclipse can give you all the callsites for a functions as you root around. You should slowly get a feel for which part of the code things get called from.
Now start asking questions like: what data is shared between these entry points? What's mutable? Is it all done safely?
Hopefully this wasn't too narrow an example. I'd imagine it'll apply to any services.
I'm working in Linux wifi drivers, all in C. Giant complicated protocol with giant complicated code. I've been tinkering with Microsoft VSCode, CLion, and Sourcetrail. Vim + Ctags seems to work well in the beginning but only gives pinpoint answers (can find trees, but little view of the forest). Still experimenting.
It depends upon the platform, the language and the toolsets. For instance, for C, a trick I've used is to (if I can) use different C compilers and crank the warnings/errors to 11 and fix every complaint (or try to---it can be daunting to attempt this all at once).
Other tricks---run the code through linters or other stylistic nit-pickers and fix those too. I haven't yet used a code reformatter, but that's a quick way to get code into a consistent style.
I'm all in C, too: kernel drivers from wifi chipset vendors. There's one C compiler we can use (gcc) and we're even restricted to working with a specific version (because cross compiling). We have to be very careful about changing any of the code because we have to integrate any changes into the next drop of the vendors' code. Static analysis is our best bet. Is a really interesting problem.
Yeah bit of an odd request for me to send you my business contacts. It really is being in an industry long enough to build a reputation to get things done. And these are enterprise projects, there’s no sexiness here. This is webforms in .NET type of stuff. Learning to ignore that the codebase is imperfect and will never be reasonable is one of the reasons I get calls.
That's what I've been doing for the past 7 or so years. Turning around struggling products, gradually improving and then sometimes help a re-launch as "greenfield based on lessons learned" when the time is right.
It's not the most sexy work in the early stages but very rewarding when you help turn a failing situation around.
Sometimes it's bugs and bad algorithms or data structures. Sometimes it's misunderstood requirements and the fix has been surgical rewrites. Most often a mix.
> It's not the most sexy work in the early stages but very rewarding when you help turn a failing situation around.
And most often that struggling but working product still brings money and real value to people or business, not like most of greenfield shiny start-up..[cough!].. throw-away projects.
That's really interesting, and definitely agree knowing the requirements is hugely important. I've heard from people at BigCorps about inheriting projects where no one at the company really knows how or why a piece of code behaves the way it does. That seems like one of the more terrifying scenarios, where you can't even ask someone what a particular piece of code is supposed to do, and you have no way of knowing how it's supposed to work (and then you are tasked with fixing bugs when they crop up). But I guess those three conditions are unlikely to overlap probably (hopefully).
Our team inherited a tangled spaghetti mess of a client facing API. There were some additional requirements that needed to be added and we quickly discovered some serious security issues.
We considered a full rewrite, but this was too time consuming and did nothing to solve the immediate issues. It was also risky, we had a "working" production app.
We ended up writing an extremely thorough integration test suite. Making changes is still painful but we know for sure we aren't breaking anything. If we have the time/drive to rewrite the test suite could be re-utilized.
Not so much on a team, but when working on a difficult and unfamiliar codebase I usually start by taking a chunk and reformatting it to suit my style as though I wrote it. Spacing, indentation, bracketing, ect.
Once I've got it to a point where I can read it with minimal cognitive load (I like condensed code with little to no white space and no orphan brackets) I'll make sure it still works and pick a spot or feature in the finished application and try to find its code. Work backwards until I've figured out what makes it tick.
In the process of doing that I usually see how a lot of other things tick and get a sense of how and where the rube goldberg machine starts.
The hardest part is understanding the rationale of the developer. Many don't impart such details in the comments. For that I'll try rewriting sections with less code than the original and see what the adverse effects are.
That is essentially what I was hired out of school to do. I walked into a massive heap of ASP.NET (with VB) and MSSQL stored procedures that didn't really work at all, and had been through the wringer of a few cut-rate outsourcing groups. I struggled along with it for a few months figuring out how it was supposed to work and trying to duck-tape it together, doing a lot of support with customers that were trying to use it, and talking to them about what they were trying to do.
Then eventually I decided I wanted to learn some newer tech, so I started playing with Linq-to-SQL and ASP.NET MVC and Razor and Bootstrap, and over the course of three or four weekends and evenings I did a ground-up rewrite of the whole thing for fun. After a a bit more time flailing away with the old mess, I showed my side-version of it off to my boss, and it wasn't that hard a sell, being much prettier and less buggy.
It helped that there was nobody around who was invested in the old code base.
Generally, I've found it is a lot easier to effect this kind of change if you just do it stealthily and present it as a fait-accompli, because otherwise people get so bogged down in debate and fear that any impetus to actually take a risk and do something evaporates.
I'm doing something similar for a project I inherited from an outsourced company. Did you end up just giving it to your employer? Did you ask for compensation for all the non-work time you spent building it?
I was 23 and had a lot of time on my hands, and I was learning stuff and getting away from having to be on endless support calls with that gawdawful mess that I replaced. It was a pretty good trade, all things considered - I think I got a 15k raise that year, and built a lot of trust that I could do good work independently without a lot of oversight. On balance, I think I've been more than compensated for that time since then.
The problem with going this route is you have to sacrifice personal time for it. Now in your case you taught yourself something new, but that's not always going to work.
I do agree with your assessment that it is much easier to drop a working thing in people's laps and get buy-in than ask for permission. This is also what causes pork barrelling in projects.
Or, they are actually good at their craft and rewrote it. That's not an unreasonable timeline. Taking code that does something and writing another version in another design pattern is much, much faster than starting from scratch. Even with complicated code (especially with complicated code)
It really wasn't very complicated... basically just some queries to search a database and a front-end to display results.
I still don't know how the preceding mess got to the point of being such a baroque cobbled-together monstrosity, but it was enough copy-pasta to keep an Olive Garden supplied for a year, and a raft of incomprehensible stored procedures that did SELECT * FROM table WHERE xyz in bizarrely complicated ways involving multiple casts and nested temp results.
I've worked on similar projects that could have been re-written relatively quickly but were only complicated because the developers made it complicated. Some devs just want to solve hard problems in "elegant" ways and will throw in elaborate inheritance chains, meta-programming, code generation, service oriented architectures, layers upon layers, frameworks, rules engines and anything else that's not boring business logic.
IME these projects are usually more complicated than all but the most wild of business logic.
I know this isn't what most would consider a "codebase" but at a college job doing mostly CNC programming I had to troubleshoot problems with startup and maintenance G Code for a 2.5 axis CNC machine that the owners didn't want to pay to have the manufacturer consult on (smart move). The kicker was it was entirely documented in Italian. It took a lot of meticulous documentation and patience. It was honestly a great learning experience in terms of both reading code and thinking through all of the outcomes of a change you made (considering a mistake could have damaged the machine).
I'll open by saying I've only ever had bad experiences with complete re-writes and these experiences have impacted my aversion to them.
"[Working Effectively with Legacy Code]" by Michael Feathers really helped me get through a situation like this.
My recommendation is not to try to understand the code per se, but understand the business that the code was being used in/by.
From there, over time, just start writing really high level end-to-end tests to represent what the business expects the codebase to do (i.e. starting at the top of the [test pyramid]). This ends up acting as your safety net (your "test harness").
Then it's less a matter of trying to understand what the code does, and becomes a question of what the code should do. You can iterate level by level into the test pyramid, documenting the code with tests and refactoring/improving the code as you go.
It's a long process (I'm about 4.5 years into it and still going strong), but it allowed us to move fast while developing new features with a by-product of continually improving the code base as we went.
>> My recommendation is not to try to understand the code per se, but understand the business that the code was being used in/by.
I strongly agree with this. I've done at least 4 or 5 successful complete rewrites of old code bases, and I have found, rather than even 'business' the word for this might be 'context'.
If you can contextualize a piece of software, it's functionality and operations, you can have a much better understanding of an existing codebase.
What would you do if the codebase was actually 5 codebases absorbed from 5 different smaller companies? Assume that zero institutional knowledge about the code / business have been passed on.
I have to assume someone is using the software therefore there is some tribal knowledge of what it does? Otherwise this is maybe SAAS software that users use and some functionality is exposed that would allow you to begin decomposing backwards toward expected input/output. You're almost black-boxing at that point.
I will admit that I have, on very rare occasion, scream tested a piece of software running on a server that nobody would claim ownership or knowledge of either on the eng. team or within the org.
There's a surface level understanding of what it does but nobody really understands how many of the large features really work, or what the actual rules are that govern them. Yes, much of this is black box. Example: yesterday I had to try to figure out what branch of code was compiled and deployed to our server. Everyone had assumed it was the Master branch, but no...deploying that branch fubared everything. I finally found the "working" branch of code.
Part of the problem is that the people who owned tribal knowledge were all fired / quit without documenting anything. Every member of the existing team has been there around a year or less.
>> Assume that zero institutional knowledge about the code / business have been passed on.
Who is, in that case, using the software? They obviously understand the context by which the software is at least going to work, otherwise, why is the software being rewritten?
Who is requesting the rewrite? Do they know what it is supposed to do? Is there an executable build of it that exists somewhere?
These are ecommerce systems. It's astonishing because no-one in the company truly has a complete understanding of the business, as far as I can tell. The code is running in production and serving customers.
Rewrite is being pushed by certain parties because we're unable to meet feature requests quickly with the existing system, and it's being assumed that a rewrite will fix that problem. The team is barely functional though (from the top down). I've seen a few failed projects now and I don't think the rewrite will ever be accomplished. If we manage to rewrite, it's far from certain that we'll do a better job than the last guys did.
I...yeah. The picture wasn't completely clear until very recently and now the anxiety has kicked in. I'm trying to stick around a little because I've been through too many jobs in too short a time and I think I need to show some "commitment" on my resume.
I love that book. I can't recommend it highly enough.
Approval Tests (http://approvaltests.com) can be a huge timesaver when you're getting that initial black box characterization put together.
Besides being an important part of getting your bearings, talking to everyone who relies on the software to get a better understanding of how they interact with it can be a great time saver, too. It's amazing how quickly you can clean up legacy code with the delete key, provided you can confirm nobody's using it anymore.
The wholesale rewrite is a will-o-the-wisp. Very, very attractive, yes. But usually when people chase after it, they end up drowning in a quagmire. That isn't to say that you shouldn't strive to get rid of all the bad code, but do it as a long-term, component-wise, in-place rewrite.
> My recommendation is not to try to understand the code per se, but understand the business that the code was being used in/by.
You're absolutely right, but the problem comes when the code itself is the only authoritative documentation of what the code does, and in a lot of cases, the only authoritative documentation (or even the only documentation, period) of what the code is supposed to do!
Often enough to consider Software Archaeologist as a role.
- Big picture:
try to identify the integration points with other systems or entry/exit points into the code.
See if the code is logically (and hopefully actually) divided into separate smaller parts. If it is, try to work out the main purpose of each part, its integration points, and if there are any obvious side-effects.
- Detail:
Is it building, clean-building, testing etc? That will make it a lot safer to explore and experiment.
For a single source file people have different approaches to "reading" it. Some people add notes as comments as they go through the file, they don't have to be permanent well formatted "Comments", rather just things to reduce the memory strain. Other people remove blank lines, comments and extra whitespace to try to compact as much actual code into a single screen to look at the code paths.
- Repository:
Is the code checked into a source control system with a log history? If so look in there for clues as to WHY things were changed, this gives a good indicator of changes to requirements and also can explain why some parts of code may "feel" different to others (they may have had to shoe-horn in a new change to an existing codebase).
- Pragmatic:
The previous people (just like you) probably never had a chance to refactor or clean up any tech debt.
A coworker really wanted to be a technical lead on a firmware project so our boss gave him one. Part way through our boss asked me to help the coworker out but I had a tough time understanding his code. Soon enough he admits that his is leaving the company. He created too much of a mess and wanted to bail out. Suddenly it was all my responsibility. And this was with an major customer we had multiple partnerships with. So somehow I had to salvage everything without making them aware the mess we were in. So over the next 3 months we would give them weekly engineering builds while totally rewriting the code piece by piece. Once we were back on track it was much smoother. It helped that our management didn't micromanage me and the customer engineers were brilliant and a breeze to work with. It was all about the code, requirements and doing the right things. Our progress meetings were literally 15 minutes week. Everything else was technical discussions and development.
I contracted for a company that had no software team, and had outsourced the development of their embedded product to the lowest bidder. The original firm had delivered code that met most requirements, but were not willing/able to resolve issues with random crashes or implement additional features. The product manager reached out to my employer at the time for help.
When I took over the code base, my initial attempts to modify the function of the code rendered the device completely non functional, so I focused on restructuring the code without changing its behavior. I moved code into functions, functions into libraries. I added parameters to existing functions so that global variables would have to be injected, rather than accessed directly (this helped make it clear what the inputs and outputs to the function were.)
Eventually, I modeled state external to the system with state machines so that it would be clear when code was trying to manipulate a resource that hadn't been initialized yet. (This helped make some bugs stand out like a sore thumb.)
Through incremental changes and testing at every step, this refactoring made the structure and flow of the program much easier to understand. After only 2 weeks of refactoring, I was able to identify and fix the bugs that had been causing the random crashes. I was also able to add new functionality to the well structured program in a fraction of the time it had taken to do the initial refactoring.
The best part about restructuring/refactoring code is that even after totally reorganizing the entire codebase, I still only had a high-level understanding of how it all worked; I didn't have to personally grok every requirement or fine detail as I would have needed to do if I'd rewritten the code from the ground up. Refactoring was slow going at first, but it really saved the day.
Large insurance business paid a previous contractor to write up a simple web app to consolidate public rates.
So this business could go and answer questions like "how much is my competition charging for XYZ?"
This "simple web app" turned into the previous contractor writing his own insane web framework from scratch in Python, because I guess Django or something was not good enough...?
Anyway the result was something that was almost impossible to read, had who knows how many security vulnerabilities, and was an awful experience for the insurance company.
Lots of times implementing a new feature meant changing the web framework so you could actually implement it.
Company already spent who knows how much on the previous contractor, so after a small cost working with me to evaluate what else was needed to complete the project they decided to go another direction.
I planned out how we could migrate the app slowly over to Django (it was a SPA as well, but he did at least use React there not something crazy he wrote himself) but they didn't have the budget.
Unfortunate. Could have been a really cool tool for not a lot of money, and I would love for businesses to be more eager about developing such products. The concept was the perfect example of a business-specific use of software to give the corporation an edge.
This sounds like a great opportunity to just go build the thing they needed then offer it back to them as a service or for a licensing fee. Then you have the option to offer it to other companies as well.
You might not have access to the data sources, you might not have the context to interpret them in a meaningful, and then the company you are targeting is probably your only potential customer (or you have to cold sell to every competitor, so now you have a sales job, right?)
Also, after a project fails the business is always close to 0% likely to work with you on it, even if it isn't your fault (I was brought in as a clean-up man and my billing was insignificant).
I have not fond memories of the "alite" CPU simulator. It was widely used as a basis for scientific papers on CPU design back in the early/mid 90s. The source code of alite was absolutely incomprehensible, which did make me wonder about the validity of the published results.
It's very sad, but understandable that you're getting down voted. It's understandable because most devs want to put their own stink on a project so rewriting code the "right way" is a way to do it. Problem is, they get 6, 8, 10 months in and figure out the same thing their predecessor did and leave. It's sad, because they don't realize that if a piece of code is out in the field and it's working, then you really shouldn't do a wholesale rewrite.
Exactly. Everyone thinks that the predecessor is an idiot and they can do it better. Every time you throw away existing code you lose business knowledge.
I agree with you in general, but I think a better rule would be to almost never rewrite code.
I know that everyone thinks their predecessor was an idiot. And I agree that full rewrites are usually a bad idea. But sometimes, the predecessor genuinely was an idiot, and the code really is that bad. If the use cases and inputs and desired outputs are well documented enough, a full rewrite can be the right choice.
It depends on the size of the system, too. A full rewrite of something as complex as Netscape is asking for trouble. If it's something that can be rewritten in a day, or even a few weeks, it might okay to go ahead and do it.
Of course, by the time you've got enough experience to accurately estimate how long a rewrite will take, you've probably got enough experience to just refactor the old code without going insane.
If the inputs and outputs and the use cases are understood and it’s akready working, create an anti corruption layer and treat it like a 3rd party binary blob.
Yes, it was a mega legacy codebase written by a single person over the span of a decade and was extremely “job secured”. It had Perl scripts that would system call to php scripts that would in turn do a curl request to another http perl script that would system call another php script that would output HTML, which then would get parsed by the calling scripts several ways. That was just one place. There were lots and lots of these problems.
Our team was handed the project to rewrite it. It was a secret project and we were very careful. We didn’t want to spook the original developer.
It was a lot of hair pulling and tracking of the code. Lots of gruelling work.
Oh I inherited a similar Perl/php/shell script mess a few years ago. A fun twist was that the codebase started off as a commercial groovy/grails application, and had this Frankenstein thing surgically attached, reading and writing to the same database tables.
Would love to know what happened once you were done, especially was the original developer still around and did they have a funny panic mode upon realizing the job security rug had been pulled out from under them?
Eventually the dev got a whiff of it. He was let go abc offered, I’m guessing, a lofty consulting position for a year. This was the exact scenario they were trying to avoid, but at least we pushed it back to as far as we could.
The project eventually went from horrible to interesting once all of the legacy was dealt with.
PostgreSQL allows C extensions. A function which used to work was now segfaulting occasionally so needed to be fixed. Original author was gone, nobody really understood the PostgreSQL extension system (which involves a ton of macros.) No version control, no useful comments, no spec. I read the code (the core of the logic was straight-forward C, it was just the interface to PostgreSQL that was hard to understand) and wrote a new version in Python (another language for which PostgreSQL supports extension functions.) That version was too slow, by two orders of magnitude. Went back to the C implementation. Did a little light refactoring to separate the core logic from the PostgreSQL interface. Then wrote a test harness program in C to drive call only the inner function. Wrote unit tests until I reproduced the segfault. Fixed the C code and tested it with my test harness. The PostgreSQL wrapper just called the (now correct) inner function so it now worked too. Checked the fixed code and unit tests into version control.
A PhD (no longer with the company) had fit some simple neural net models in R. He'd written his own code for this because, according to him, the standard packages in R didn't support a few of the bells and whistles he'd wanted like ReLU activation. Not only was the code in-house, but the models themselves were saved as serialized R objects. No specifications for any of this stuff. Apparently the company had been using this code to score medium size databases for several years. When we wanted to scale up to a much larger database (approximately 30 billion rows) the problems with the R implementation became apparent. Fairly slow, high memory usage, worse yet a slow memory leak on large jobs, and worst of all occasional silent crashes where it would simply stop and exit with a successful status of 0 and no error message. This time I took the approach of reading the code and de-serialized R objects. I re-wrote the implementation in Python using numpy arrays and wrote a small R program to read the serialized models in the .RData format and emit a cleaned up JSON object that could be easily read from Python. Luckily I didn't have to port any of the cross-validation/training/optimization stuff; just the prediction part. That meant 80% of the code could be safely ignored. However, the devil was in all the one-off special cases in the serialized R objects, many of which had behavior different from default. I would test by comparing the predictions by the two programs on small batches of a million. These predictions were floating point numbers between 0 and 1 so when both programs agreed to within 1e-5 for all million I knew it was correct. It took a week to track down all the special cases, though, and the special cases made it impossible to just use a standard neuralnet library like Keras. (We already had several Keras models in production and a set of tools to manage them; that would have been easy for us.) Proprietary code begets proprietary code I guess. At least the new implementation was much faster and didn't leak memory so could deal with the whole database in a single long running process. I pitched the idea of re-training all models from scratch in Keras using our more modern tools but management wanted 100% backwards compatibility and to preserve the value-add of the PhD.
221 comments
[ 2.3 ms ] story [ 265 ms ] threadI've had a few more-or-less realtime multithreaded projects that I've worked on where you can't really put a debugger on a system and halt it, without breaking things in interesting and misleading ways, and the only good option is to fall back to ye olde printf debugging.
The solution almost always involved a certain amount of reverse engineering, and damn sure I always tried to advocate getting away from software we couldn't maintain effectively. The third leg of that tripod is isolation. Reduce the number of things that depend on the code, and the number of ways that they use it. If nothing else, that will reduce your exposure. It will also tell you what code paths are important to understand and which are not. Finally, it can help guide implementation of tests, or of a replacement.
If you're really stuck with such a piece of code, some of the advice in a blog post I wrote about a similar challenge might apply.
http://obdurodon.silvrback.com/navigating-a-large-codebase
That's about learning a codebase that's notable mainly for being large, even if the original developers are as available and helpful as could be, but looking at it now I see quite a bit that applies to this case as well.
Many years ago, I became the designated maintainer for a legacy inventory system that was used for internal audit purposes. The system ran on a PDP-11/70, and was written in a combination of Basic-Plus (this was a long time ago) and COBOL. I only had a passing familiarity with either of those languages. I find the challenges of learning a code base usually boil down to a few recurring problem areas:
* Problem domain knowledge: The system was used to track leased telecommunication facilities, and had a lot of obscure business logic built into it. At least half the challenge was reverse engineering business logic from the code.
* Coding style: I always find it challenging to get comfortable with another developer's style. Mismatches in assumptions/preferred approaches can make it really hard to get comfortable with someone else's code. This particular system had some truly weird programming choices, including a "screen driver" (similar to curses) written in COBOL.
* Code complexity: I was pretty lucky that the code was not very complicated. With all the other challenges, if the code had been complex it would have probably been an impossible challenge.
* Language knowledge: the original developers had used some features of Basic-Plus and COBOL that were a little obscure, which made understanding the code base that much harder.
There were some smart cookies on the team, but we were all in fear of this code. The person who wrote it spent her days sitting in the cafe downstairs, reading novels. She'd check some logs, and occasionally come and yell at us for doing something wrong, which she would never explain. It turns out that the system had objects, but they were all embodied by consecutive spans of entries in those arrays. We went on a trek through the bowels of this corporation, asking around for documentation of the 3rd party software that used those sequences in those arrays, but no one had it, and that company no longer existed. If you tried to explain to her that Object Oriented code would have instance methods, she'd always bring up her PhD in math.
A coworker of mine spent a week charting out one of those methods, and managed to rewrite it in 1/6th the line count, with no errors. However, that didn't really help, as it still implemented the same weird merging algorithm.
You must study the code and document what you can. Reverse engineering it is sometimes the only option in your quest to understand it. Then you’ll need to make decisions about how to move forward given the constraints of the business whether it be time, resources, other priorities. There is no one size fits all answer but don’t be shy about vocalizing the risks and making sure the business knows the pain points. Sometimes you have to sell the problem you now have and get buy-in to really do something to fix the mess.
One thing I’ve realized is that when people don’t take the time to document and write clean code sometimes it boils down to their idea of job security.
Other times it turns out the problem is there is no single owner of this code base and it’s been hacked to death.
Lastly when something is ugly, enough and messy enough...then you can just call it proprietary technology...I kid I kid.
In any case there’s never been something that I’ve run into that I’ve not figured out. It takes time, and the hard part usually is not figuring out what it does but the weird edge case of the moon aligning with Venus and then the output suddenly changes. This is why understanding the requirements is more important than the code. I don’t care if the code is bad if I can more or less write a test case against it and make sure it does that.
That said complete rewrites never happen. It usually is only rewriting portions when that is cheaper than fixing. It is the if it isn’t broke don’t fix it adage.
The only time ever I’ve been stuck was when I saw a proprietary software implantation on top of a custom software package ontop of Solr (technically ontop of the JVM) create a large object heap issue ontop of a proprietary OS (Windows). It wasn’t code related I diagnosed that it was a GC issue, but it wasn’t in any code I had source to. A Windows update ended up fixing it. And this is why working with enterprise software is hard.
My process looks like this:
Step one: Identify sources of reflection, this is the triskyest. Hopefully the only dependencies are open source, so you generally know what they do and grep can usually find the rest.
Step Two: go code spelunking. Find your entry points. Find your main() or framework equivalent. Find callsites for rest endpoints, rpc, jmx, etc.
Step three: find other "external request processing" endpoints. Do you have timer threads? Reading a Kafka stream and acting per record? Etc.
Once you understand those, you can interpret where most any stacktrace is from. Good old Intellij or Eclipse can give you all the callsites for a functions as you root around. You should slowly get a feel for which part of the code things get called from.
Now start asking questions like: what data is shared between these entry points? What's mutable? Is it all done safely?
Hopefully this wasn't too narrow an example. I'd imagine it'll apply to any services.
Other tricks---run the code through linters or other stylistic nit-pickers and fix those too. I haven't yet used a code reformatter, but that's a quick way to get code into a consistent style.
It's not the most sexy work in the early stages but very rewarding when you help turn a failing situation around.
Sometimes it's bugs and bad algorithms or data structures. Sometimes it's misunderstood requirements and the fix has been surgical rewrites. Most often a mix.
And most often that struggling but working product still brings money and real value to people or business, not like most of greenfield shiny start-up..[cough!].. throw-away projects.
We considered a full rewrite, but this was too time consuming and did nothing to solve the immediate issues. It was also risky, we had a "working" production app.
We ended up writing an extremely thorough integration test suite. Making changes is still painful but we know for sure we aren't breaking anything. If we have the time/drive to rewrite the test suite could be re-utilized.
Once I've got it to a point where I can read it with minimal cognitive load (I like condensed code with little to no white space and no orphan brackets) I'll make sure it still works and pick a spot or feature in the finished application and try to find its code. Work backwards until I've figured out what makes it tick.
In the process of doing that I usually see how a lot of other things tick and get a sense of how and where the rube goldberg machine starts.
The hardest part is understanding the rationale of the developer. Many don't impart such details in the comments. For that I'll try rewriting sections with less code than the original and see what the adverse effects are.
Then eventually I decided I wanted to learn some newer tech, so I started playing with Linq-to-SQL and ASP.NET MVC and Razor and Bootstrap, and over the course of three or four weekends and evenings I did a ground-up rewrite of the whole thing for fun. After a a bit more time flailing away with the old mess, I showed my side-version of it off to my boss, and it wasn't that hard a sell, being much prettier and less buggy.
It helped that there was nobody around who was invested in the old code base.
Generally, I've found it is a lot easier to effect this kind of change if you just do it stealthily and present it as a fait-accompli, because otherwise people get so bogged down in debate and fear that any impetus to actually take a risk and do something evaporates.
I do agree with your assessment that it is much easier to drop a working thing in people's laps and get buy-in than ask for permission. This is also what causes pork barrelling in projects.
Either that or significant chunks of it were retained and your improvements were mostly cosmetic.
I still don't know how the preceding mess got to the point of being such a baroque cobbled-together monstrosity, but it was enough copy-pasta to keep an Olive Garden supplied for a year, and a raft of incomprehensible stored procedures that did SELECT * FROM table WHERE xyz in bizarrely complicated ways involving multiple casts and nested temp results.
IME these projects are usually more complicated than all but the most wild of business logic.
"[Working Effectively with Legacy Code]" by Michael Feathers really helped me get through a situation like this.
My recommendation is not to try to understand the code per se, but understand the business that the code was being used in/by.
From there, over time, just start writing really high level end-to-end tests to represent what the business expects the codebase to do (i.e. starting at the top of the [test pyramid]). This ends up acting as your safety net (your "test harness").
Then it's less a matter of trying to understand what the code does, and becomes a question of what the code should do. You can iterate level by level into the test pyramid, documenting the code with tests and refactoring/improving the code as you go.
It's a long process (I'm about 4.5 years into it and still going strong), but it allowed us to move fast while developing new features with a by-product of continually improving the code base as we went.
[test pyramid]: https://martinfowler.com/bliki/TestPyramid.html [Working Effectively with Legacy Code]: https://www.amazon.com/FEATHERS-WORK-EFFECT-LEG-CODE/dp/0131...
I strongly agree with this. I've done at least 4 or 5 successful complete rewrites of old code bases, and I have found, rather than even 'business' the word for this might be 'context'.
If you can contextualize a piece of software, it's functionality and operations, you can have a much better understanding of an existing codebase.
I have to assume someone is using the software therefore there is some tribal knowledge of what it does? Otherwise this is maybe SAAS software that users use and some functionality is exposed that would allow you to begin decomposing backwards toward expected input/output. You're almost black-boxing at that point.
I will admit that I have, on very rare occasion, scream tested a piece of software running on a server that nobody would claim ownership or knowledge of either on the eng. team or within the org.
Who is, in that case, using the software? They obviously understand the context by which the software is at least going to work, otherwise, why is the software being rewritten?
Who is requesting the rewrite? Do they know what it is supposed to do? Is there an executable build of it that exists somewhere?
Rewrite is being pushed by certain parties because we're unable to meet feature requests quickly with the existing system, and it's being assumed that a rewrite will fix that problem. The team is barely functional though (from the top down). I've seen a few failed projects now and I don't think the rewrite will ever be accomplished. If we manage to rewrite, it's far from certain that we'll do a better job than the last guys did.
Approval Tests (http://approvaltests.com) can be a huge timesaver when you're getting that initial black box characterization put together.
Besides being an important part of getting your bearings, talking to everyone who relies on the software to get a better understanding of how they interact with it can be a great time saver, too. It's amazing how quickly you can clean up legacy code with the delete key, provided you can confirm nobody's using it anymore.
The wholesale rewrite is a will-o-the-wisp. Very, very attractive, yes. But usually when people chase after it, they end up drowning in a quagmire. That isn't to say that you shouldn't strive to get rid of all the bad code, but do it as a long-term, component-wise, in-place rewrite.
You're absolutely right, but the problem comes when the code itself is the only authoritative documentation of what the code does, and in a lot of cases, the only authoritative documentation (or even the only documentation, period) of what the code is supposed to do!
I quit six months in.
I lost most of my hair six months in.
- Big picture:
try to identify the integration points with other systems or entry/exit points into the code.
See if the code is logically (and hopefully actually) divided into separate smaller parts. If it is, try to work out the main purpose of each part, its integration points, and if there are any obvious side-effects.
- Detail:
Is it building, clean-building, testing etc? That will make it a lot safer to explore and experiment.
For a single source file people have different approaches to "reading" it. Some people add notes as comments as they go through the file, they don't have to be permanent well formatted "Comments", rather just things to reduce the memory strain. Other people remove blank lines, comments and extra whitespace to try to compact as much actual code into a single screen to look at the code paths.
- Repository:
Is the code checked into a source control system with a log history? If so look in there for clues as to WHY things were changed, this gives a good indicator of changes to requirements and also can explain why some parts of code may "feel" different to others (they may have had to shoe-horn in a new change to an existing codebase).
- Pragmatic:
The previous people (just like you) probably never had a chance to refactor or clean up any tech debt.
https://news.ycombinator.com/item?id=18442637
(But the current question is different enough that it has seeded a different kind of thread.)
I contracted for a company that had no software team, and had outsourced the development of their embedded product to the lowest bidder. The original firm had delivered code that met most requirements, but were not willing/able to resolve issues with random crashes or implement additional features. The product manager reached out to my employer at the time for help.
When I took over the code base, my initial attempts to modify the function of the code rendered the device completely non functional, so I focused on restructuring the code without changing its behavior. I moved code into functions, functions into libraries. I added parameters to existing functions so that global variables would have to be injected, rather than accessed directly (this helped make it clear what the inputs and outputs to the function were.)
Eventually, I modeled state external to the system with state machines so that it would be clear when code was trying to manipulate a resource that hadn't been initialized yet. (This helped make some bugs stand out like a sore thumb.)
Through incremental changes and testing at every step, this refactoring made the structure and flow of the program much easier to understand. After only 2 weeks of refactoring, I was able to identify and fix the bugs that had been causing the random crashes. I was also able to add new functionality to the well structured program in a fraction of the time it had taken to do the initial refactoring.
The best part about restructuring/refactoring code is that even after totally reorganizing the entire codebase, I still only had a high-level understanding of how it all worked; I didn't have to personally grok every requirement or fine detail as I would have needed to do if I'd rewritten the code from the ground up. Refactoring was slow going at first, but it really saved the day.
Large insurance business paid a previous contractor to write up a simple web app to consolidate public rates.
So this business could go and answer questions like "how much is my competition charging for XYZ?"
This "simple web app" turned into the previous contractor writing his own insane web framework from scratch in Python, because I guess Django or something was not good enough...?
Anyway the result was something that was almost impossible to read, had who knows how many security vulnerabilities, and was an awful experience for the insurance company.
Lots of times implementing a new feature meant changing the web framework so you could actually implement it.
Company already spent who knows how much on the previous contractor, so after a small cost working with me to evaluate what else was needed to complete the project they decided to go another direction.
I planned out how we could migrate the app slowly over to Django (it was a SPA as well, but he did at least use React there not something crazy he wrote himself) but they didn't have the budget.
Unfortunate. Could have been a really cool tool for not a lot of money, and I would love for businesses to be more eager about developing such products. The concept was the perfect example of a business-specific use of software to give the corporation an edge.
You might not have access to the data sources, you might not have the context to interpret them in a meaningful, and then the company you are targeting is probably your only potential customer (or you have to cold sell to every competitor, so now you have a sales job, right?)
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
I know that everyone thinks their predecessor was an idiot. And I agree that full rewrites are usually a bad idea. But sometimes, the predecessor genuinely was an idiot, and the code really is that bad. If the use cases and inputs and desired outputs are well documented enough, a full rewrite can be the right choice.
It depends on the size of the system, too. A full rewrite of something as complex as Netscape is asking for trouble. If it's something that can be rewritten in a day, or even a few weeks, it might okay to go ahead and do it.
Of course, by the time you've got enough experience to accurately estimate how long a rewrite will take, you've probably got enough experience to just refactor the old code without going insane.
Our team was handed the project to rewrite it. It was a secret project and we were very careful. We didn’t want to spook the original developer.
It was a lot of hair pulling and tracking of the code. Lots of gruelling work.
Still have nightmares about it now.
The project eventually went from horrible to interesting once all of the legacy was dealt with.
PostgreSQL allows C extensions. A function which used to work was now segfaulting occasionally so needed to be fixed. Original author was gone, nobody really understood the PostgreSQL extension system (which involves a ton of macros.) No version control, no useful comments, no spec. I read the code (the core of the logic was straight-forward C, it was just the interface to PostgreSQL that was hard to understand) and wrote a new version in Python (another language for which PostgreSQL supports extension functions.) That version was too slow, by two orders of magnitude. Went back to the C implementation. Did a little light refactoring to separate the core logic from the PostgreSQL interface. Then wrote a test harness program in C to drive call only the inner function. Wrote unit tests until I reproduced the segfault. Fixed the C code and tested it with my test harness. The PostgreSQL wrapper just called the (now correct) inner function so it now worked too. Checked the fixed code and unit tests into version control.
A PhD (no longer with the company) had fit some simple neural net models in R. He'd written his own code for this because, according to him, the standard packages in R didn't support a few of the bells and whistles he'd wanted like ReLU activation. Not only was the code in-house, but the models themselves were saved as serialized R objects. No specifications for any of this stuff. Apparently the company had been using this code to score medium size databases for several years. When we wanted to scale up to a much larger database (approximately 30 billion rows) the problems with the R implementation became apparent. Fairly slow, high memory usage, worse yet a slow memory leak on large jobs, and worst of all occasional silent crashes where it would simply stop and exit with a successful status of 0 and no error message. This time I took the approach of reading the code and de-serialized R objects. I re-wrote the implementation in Python using numpy arrays and wrote a small R program to read the serialized models in the .RData format and emit a cleaned up JSON object that could be easily read from Python. Luckily I didn't have to port any of the cross-validation/training/optimization stuff; just the prediction part. That meant 80% of the code could be safely ignored. However, the devil was in all the one-off special cases in the serialized R objects, many of which had behavior different from default. I would test by comparing the predictions by the two programs on small batches of a million. These predictions were floating point numbers between 0 and 1 so when both programs agreed to within 1e-5 for all million I knew it was correct. It took a week to track down all the special cases, though, and the special cases made it impossible to just use a standard neuralnet library like Keras. (We already had several Keras models in production and a set of tools to manage them; that would have been easy for us.) Proprietary code begets proprietary code I guess. At least the new implementation was much faster and didn't leak memory so could deal with the whole database in a single long running process. I pitched the idea of re-training all models from scratch in Keras using our more modern tools but management wanted 100% backwards compatibility and to preserve the value-add of the PhD.