Ask HN: How to be productive with big existing code base

I recommend the book Working Effectively with Legacy Code by Michael Feathers. I've been reading it recently, and I've enjoyed the lessons and advice so far.

What is your problem with the code base actually?

Document everything as you explore it. I'm an advocate for literate programming but accept it's not going to be accepted by most organizations. So I use it as a personal tool.

Tools: emacs, org mode, org babel.

Create a parallel directory structure, hypothetical project:

  ./src
  ./project/src/main.js
  ./project/src/some-file.js

Create a new directory structure with one org file per source file and one index org file:

  ./project-org/src/main.org
  ./project-org/src/some-file.org
  ./project-org/index.org

(You can organize it differently, this has worked for me.)

index.org will be a simple tree view of the folder hierarchy:

  * Project Name
  Description
  ** Source Files
  *** src
  **** [file:src/main.org]
  **** [file:src/some-file.org]

You may add in some notes about the general purpose of each of those files.

main.org copy the entire code into the main.org file like:

  * main.js
  #+BEGIN_SRC js
  // all the code from main.js
  #+END_SRC
  * [[file:../index.org][Project Root]]

Start splitting the contents of main.js into separate snippets, I don't know Javascript very well so let me make some quick C example:

  * main.c
  #+BEGIN_SRC c :noweb yes :tangle yes
    <<includes>>
    <<structs>>
    <<functions>>
  #+END_SRC
  ** includes
  #+NAME: includes
  #+BEGIN_SRC c
    ,#include <stdio.h> // [0]
  #+END_SRC
  ** structs
  #+NAME: structs
  #+BEGIN_SRC c
    // structs
  #+END_SRC
  ** functions
  #+NAME: functions
  #+BEGIN_SRC c :noweb yes
    <<some-func>>
    <<main>>
  #+END_SRC
  *** main
  #+NAME: functions
  #+BEGIN_SRC c
    int main (...) {...}
  #+END_SRC
  *** some_func
  This function will initialize a block of memory
  to be used as a shared buffer between two processes.
  #+NAME: some_func
  #+BEGIN_SRC c
    void some_func (...) {...}
  #+END_SRC

As you go through this you can make cross-references to other files and functions/structures. Eventually you'll find a smallest reasonable unit. A long, but clear, function doesn't need to be dissected. But a short, complex one, may end up with each line broken out and analyzed.

I don't just import a massive code base and do this in one go. Instead I import parts of it and break down all the related files to a particular topic ("How does X happen?"). I trace it from start to end, and then repeat with the next question. Good, modular code makes this much, much easier. The more tightly coupled, the harder it is to understand no matter the method.

[0] The comma is inserted by org babel to distinguish from it's on #-prefixed content.

    // increments the counter
    i += 1

  * can_send: () -> bool
  =can_send= will signal =true= if the conditions are correct
  for transmitting a message over the radio. Otherwise, it'll
  transmit false. Here are the conditions that it checks:
  - Condition :: description
  - Condition :: description
  If any of these are true, then we can transmit.
  #+BEGIN_SRC C
    // body of can_send
  #+END_SRC

  - Condition :: description, which maps to Requirement SRD-1010.
  * Message Y
  // description of the message format
  // code for packing it or the class struct or whatever

  ** Structs
  #+NAME: structs
  #+BEGIN_SRC c
  // all the struct defs
  #+END_SRC

  ** Structs
  #+NAME: structs
  #+BEGIN_SRC c :now yes
    <<msg123>>
    // rest
  #+END_SRC
  *** msg123
  This struct holds messages of type 123. Here's the spec for it [some link].
  Here's a list of each component and their acceptable values:
  - type :: stored in the lower 5-bits of the first 16-bit word, should be 123
  - timetag :: stored in the second 16-bit word, represents time since midnight
    in seconds.
  #+NAME: msg123
  #+BEGIN_SRC c
    // code
  #+END_SRC

My #1 rule for existing codebases: Just because you wouldn't have done it the way they did doesn't mean they did it wrong.

I think it's developer nature to look at a huge pile of code that someone else wrote and immediately think: "This is a pile of crap. I can do better, so the first thing to do is rewrite all of this, my way (which just so happens to be _The Right Way_)."

Figure out what you're trying to do, and what is keeping you from doing it. Take an iterative approach to get things done. Realize that after 3 years, they have hopefully fixed a lot of bugs and got to a solution that is somewhat mature and better than you can do in a week.

Debuggers are your friend.

Do they have good test coverage? That's key. If they don't, start with that.

I have a similiar problem like you, except it's Java and more like 15 years old...

What helped? Using a debugger and stepping through the code was useful, it's more a less a REST-API here (build ontop of the system, before it was SOAP, etc.pp) and I've just used some heavily used endpoints and stepped through all the way...

Another huge boost in understanding was using flamegraphs (not sure what's hip for nodejs maybe this? https://github.com/davidmarkclements/0x)

This was really an eye opener because that app also used an external huge Java ECM and there was lot's of AOP magic, reading the flamegraphs and looking at the source was a big boost in understanding.

It's also a really useful tool to get visibility for performance problems that are not directly visible in the code.

If there are tests, reading them might also be worthwile.

And take your time... took me a few months to get a basic understanding how it's working (I'm more sysadmin, not really a dev there), so don't except to grasp everything in one week.

Ask your colleagues - maybe were to find documentation or if you don't understand something while reading the source.

This is super specific to each project but here things that worked for me in previous projects.

Two assumptions: You plan to work on this longer-term (not a 1month project stint) and there are things worth improving (eg barely used legacy app might not be worth your time)

#1 Get the team on board

if there are multiple people you need their buy-in and support for whatever approaches you want to do

#2 Plan for "health by a thousand small improvements"

it will be an iterative approach and you will refactor as you go.

#3 Don't assume different = bad

people might have done differently, consider using their approaches. you might do it differently. but it's better if you keep a consistency within the codebase. in codebase management consistency trumps cleverness

#4 Create space

Consider introducing a fix-it friday where everyone can work on little improvements

#5 Create non-blame culture

Stuff will break if people risk improving things. Avoid blame shifted to them. If bug trackers ping individual people consider pinging the whole team instead

#6 Consider automation

introduce linters, autoformating, codemods, danger.js, code complexity analysis, etc

#7 Introduce tests

This one is the most annoying. But worth doing: whenever you improve a feature a bit try adding a test - often in legacy apps there are no good tests. A lot of people recommend writing a test suite for the whole app before you do anything. If you are lucky enough to do this try it. I always found the iterative approach more realistic as you can also do feature work while refactoring.

When doing tests focus on integration (vertical/functional/etc) and not unit tests (unless the "unit" contains critical or complex logic). Your goal is to know "that you broke something" - you get by if you don't always know "what you broke"

#8 Acknowledge tech debt

not everything needs refactoring. If it's not critical and nobody needs to touch it consider acknowledging it as tech debt. Add larger notes above the problematic areas and explain why you aren't refactoring it, explain things worth knowing to understand the code better, etc. Whenever you leave comments remember that comments should explain "why" not "what" the code does.

hope that helps! good luck.

Same exact boat. Case sensitive search.

Reading Refactoring by Martin Fowler helped me a lot. The examples are in Java but many of the concepts apply across all languages. However I would say the examples makes the most sense for statically typed languages. I wonder if anyone knows of a book that covers the concepts in Refactoring but with examples in a dynamically typed language like javascript?

A lot of people are giving advice on changing/testing/refactoring etc, I'm not sure if that's what you are asking, as opposed to how to lean a new code base.

My technique for learning code bases, other than asking other devs questions like :-

"What architectural patterns are you using?" "How do you deal with testing?" "How is it deployed?" "how do you deal with data access?" "How do you deal with data migration" "how do you deal with scaling / concurrency / security / authentication" etc... broad stroke things.

If there are no other devs...look at the code and look around at some of these broadstroke things.

One thing you may find is they are using an architectural pattern you aren't familiar with, so be on the lookout for things that look odd but look very deliberate and duckduckgo to see if you can find any information around names in the code. Like if you saw FooActor BarActor, google for Actor / software etc.

Another thing to keep in mind when you find odd stuff is that quite a lot of devs copy paste stuff from the internet, and implement partial ideas they have learnt about, so duckduckgo for snippets you suspect.

Then the main thing I like to do is take a usecase from the software and follow it right through every layer, either through static inspection, or using a debugger. I then follow up on things that initially seem confusing. I then start sketch out a bit of an architectural diagram ( throwaway )

For Embedded systems code I tend to start from boot and draw out a bit of a flow diagram of what's happening.

I just happened to write a blog post about this a while ago:

http://obdurodon.silvrback.com/navigating-a-large-codebase

Short version: document everything as you learn, master your tools, look at message and data formats before code, follow some of the important code paths, change something and see how the system responds.

1. Build the code. Don't do anything else until you can build the code.

So simple, yet so many places get it wrong. Lazy devs check in code that is broken. Project structures that depend on you having things on your machine and in a particular place. Circular references. You name, I've cursed it.

If it's a project that you should be able to check out and build locally, then you should be able to check the code out in any directory and build it. Period.

Are you able to talk to the original author or the most recent maintainer? If so, I'd spend a couple hours reading the code on my own to get a basic familiarity, then sit down with the original author and ask them to give you an overview. They'll probably even still remember which parts they consider bad or hacky, and knowing about that stuff could save you a lot of trouble.

The things I do when starting on a new codebase:

1) Ask if there is onboarding documentation, or someone who can give you a high-level overview of the codebase. Typically finding a person with a lot of context on the code is the fastest and most thorough way to understand the responsibilities and layouts of a codebase. Ask if they can draw an ER diagram, it's extremely valuable documentation for any additional developers.

2) Read all the documentation possible, especially design documentation. This should hopefully give you some clues as to both function (what) and purpose (why). The discussion around this will also introduce you to the major players in the architecture of the codebase.

Note this does not necessarily mean formalized design docs, it could just be searching for any README's or relevant wiki pages. You're just gathering threads at this point, and documentation tends to be a lot more compact and easily digestible than foreign code.

3) Look at the models - there will be compact representations of data at some point. This gives good insight into the shared language of the code and can give a lot of clues about how things are done. They also tend to be a lot more human-readable than other pieces of code, so this is a plus.

4) Find and skim the largest files. Typically these perform the majority of the work, have the most responsibility, and introduce the most bugs. Knowing roughly where the major players are and what they do makes it a lot easier to read any individual file.

5) Run the application, find some small behavior (a single, simple endpoint) and debug it, stepping through the application code so you can see how a particular request flows through the system. This can show you how a lot of different concerns within the code are tied together and also ensures that you're set up for both running and debugging the codebase.

At this point you should have a fairly solid understanding of at least the most critical points of the codebase, and also be set up to run and debug it. You should also have at least one or two points of contact to ask questions. This gives you a good framework for figuring out how to modify the codebase moving forward.

Has anybody tried software intelligence tools? Seeing the architectural components and control flow of the code seems like a quick way to document the codebase and figure out which “clusters” to study. https://en.m.wikipedia.org/wiki/Software_intelligence

When improving existing code, only do gradual changes. Don't do rewrites, don't replace existing code with better solutions in one swoop.

This way, you won't be in a situation when the new solution doesn't work in some cases and you already thrown all the code it replaces under the bus. You'll always have a mostly working app.

OTOH, if you introduce an alternative solution, finish migrating to it before beginning improvements in other places in the codebase.

OpenGrok supports JS and helps to navigate the codebase easily, in turn helping you to reason about it. Good luck!

You'll come back to the same pieces of code over and over, forgetting most of the details and context each time. If it takes you a while to figure out, write it down. If you had to use the debugger to find out what's in a map, leave a comment with an example of what the keys and values look like and where they're populated from.

Once you have added comments, it lets you hover over a function to remind yourself "This does X to Y when the deposit is a check", so you never have to read the internals of that function again when you're not tracing a check.

When you have to go 12 levels deep in the call stack to find the source of parameter Y, make a note in a side wiki so you can recover that detective work the next time.

Your knowledge of the code base grows like compound interest when you only have to figure out what each piece of code does once and can skip over it after that.

aakybe How yus i do,you now pieng wued vase is just is code us fade1jsyeb ask?

That's nothing. Try a 10 year old PHP codebase with remnants of a failed GWT integration.

Step 1: investigate. find out where the program is breaking and/or delivering unacceptable performance. This means asking stakeholders "is anything broken? What is wrong with the app? what do you want to change?", reviewing exceptions, and crash logs. Keep in mind that often times stakeholders don't know what they want - determine the business objective and then work from that. Determine where the broken code is, and write narrative comments about what you think it does. If there is documentation, read it. There often isn't any documentation, and often times the comments are useless on a good day. Resign yourself to playing a combination garbage man/forensic psychologist for the next few months.

Step 2: triage. determine which broken parts you can get away with leaving alone for the short term, and which parts need immediate attention.

Step 3: Fix the most critical broken parts incrementally. If there are no tests (there are never tests...), write a test for each block of code you modify. Avoid wholesale redesigns if possible. make sure that you write tests. Try to avoid getting mad about the previous person's style - focus on getting things working to a borderline acceptable level, writing comments to explain your decisions so you or someone else has a frame of reference. The goal of doing this is to buy yourself time to clean the entire thing up.

Step 4: Once the app is working at a baseline acceptable level, examine the codebase and determine which areas (if any) require redesigns, and determine the cost/benefit of each redesign, based on what the stakeholders need, want and expect. If any redesign is necessary, negotiate with stakeholders to buy time for it - your bargaining chip should be an additional feature or two that the previous guy shat the bed on. Basic criteria for a redesign: is the current design impossible to understand? Does the current design impose unacceptable costs in terms of performance or development time? if yes to either, a redesign is probably worthwhile.

focus on outcomes. have an objective like better monitoring, better perf, better code coverage, and use that as your guiding principal. make sure any change benefits the customer or your team. avoid refactoring just to modernize, every change is a potential regression. and what is in style now will be legacy in due time.

once you decide on the objective, try to divide components up into interfaces so you can test and refactor a component with reduced side effects.

set a goal for each objective. like improve coverage 5% a month or no net neg.

long story short, be methodical and focus on outcomes

You haven't said much about your role here, but let's assume that programming figures prominently. Your immediate problem is that you will be given tasks that involve making changes to a codebase that you don't understand very well. There are many different ways to understand a codebase, and to be effective, you will need to learn some of all of them.

Firstly, there is the purpose of the system, which means getting to know what its users want from it and how they use it to achieve that. I put that first, because everything else follows from it, but that does not mean that you have to know all there is to know about that aspect before tackling anything else.

You will need to learn about your environment's process: whatever is used for task assignment, scheduling and tracking; version control; building; testing and verification, inspections, test setup and execution; integration and deployment. Of these, you will need to know how to get the source code and test any changes you make, before you can do any programming, and a significant landmark in getting to know the system is when, given nothing but a backup of the source and configuration files, you could resurrect it.

The more you know about the architecture, the better - it is the first step in understanding how the system meets (or fails to meet) the users' needs. The architecture often imposes requirements and constraints on how you approach completing a given task.

Understanding how everything works at the code level would be a desirable goal, but not one that can be achieved quickly (if at all), so you will have to be guided by what you need to understand in order to do do your assigned tasks.

It is also useful to know who knows what among the people you will be working with.

Fear large changes to existing code. If you come into a large existing code base with a mindset of a smaller one you will cause 3 bugs for every one you fix.

Instead always try to make the smallest possible change to achieve your end goals(bug fix/feature etc).