I am working on on a very large codebase for the first time,mainly on the front end code.
Needs some tips on how to trace down the code base effectively?
What you need, more than anything, is the mental model the authors used when writing it. In a company you want to sit down with the people who wrote it. If it's open source you'll have to do the same if the API is undocumented. Otherwise it's something like unbaking a cake. I've done it myself by first figuring out the dependency tree of the modules, then trying to get a high-level idea of the purpose of each component. From there, it should become easier. If the code is written with any sense at all it will get lower-level and less abstract as you work your way down to the function calls.
That's the most relevant comment. Analyzing code by reading is probably the worse way to understand it. Using preprocessed information in the form of a human being* is much faster and less error prone...
This is an incredibly broad question, and the answers will probably be all over the place. If you are working on the front end, try to find the entry points and work from there through various hypothetical scenarios. Hopefully there is documentation on various parts of the system, or architects/devs who built it, but that is unfortunately not always the case…
Find out the deployment process. Find out what to do if anything goes really, really wrong. Try to find patterns in the code. Just explore for a while and even try to create you own mental model of how things are structured.
A codebase has a time dimension, and utilizing this effectively is critical.
Not sure when a bug was introduced? Write a test then bisect (if you can't automatically bisect frontend code you have other problems, but manual bisects work in a pinch).
Not sure why something was written the way it was? Hey sometimes this stuff just evolves naturally, try looking back through blame history to see how different commits effected the code.
Not sure what needs touching to add a new feature? Try seeing if you can find a similar feature that someone else wrote in the past.
To find the frontend code associated with a feature, search the view code for text that you see in the app.
To see how it uses the backend, open the network tab in Chrome developer tools (or equiv in other browsers), look at what requests it makes, and then find corresponding controllers within the code.
You should be able to break down a front end project by documenting of all the pages and then components they depend on. Then document their flow (which pages link to which and what dialogs / components they open). Then you need to look at how data is managed among all the pages and components. Do they use something like redux for data management? Then how is the business logic is composed and how backend API's are used. Also for a front end project how are styles managed through out the application.
These are not rote advice, it’s literally the parts that have been successful as I’ve inherited maintainership of several large projects.
If it’s not already TypeScript, add a tsconfig.json, with allowJs: true, checkJs: true. You’ll probably need some tweaks from there. But merely adding the config is enough to kick VSCode or other language service providers into finding references and a lot more.
Apart from that:
- Keep notes as you uncover how things flow and interact.
- The best way to keep those notes is as part of your project’s documentation. Use JSDoc and TypeScript declaration files. Learn how these docs interact and aid navigation.
- @see is a particularly useful JSDOC tag!
- Speed up the build if it’s slow. No really, it’s going to help with discoverability in the codebase! Breaking flow to wait for a build is a sure way to lose track of your journey.
- Backfill tests before you touch implementation. Even if you suspect the tests might be redundant. Odds are they’re not, but even if they are, (1) that’s more documentation you can reference and (2) any other maintainers/contributors seeing redundancy can use that as a signal to shortcut your familiarization.
- Git blame is your friend. Some days it’s your best friend.
- Look in weird places! Sometimes that old version of a dependency is pinned for a reason. Sometimes it’s pinned to a fork, again for a reason.
Logging to console, to build a mental model of the codepaths data and events take.
Git grep (or ripgrep) to find usage - useful when refactoring and to see how data is used/accessed and where it's passed around. Also useful to note where certain data doesn't show up: you can infer some structure from this.
Looking at when something was last changed with git blame can be useful. Is something suddenly broken, but hasn't been changed in 5 years? Could give an indication of where not to look on a first pass.
Break some things (locally) on purpose. Get a feel for how errors bubble up through the application and how dependant code behaves when something is wrong.
Look over the last handful of PRs/merged patches. It can be helpful to see these smaller pieces of code, the changeset, and their associated context - whether it was a feature, a bugfix, and what the code was supposed to achieve.
Use existing code in the codebase as a styleguide. Most work on large codebases isn't groundbreaking or innovative, so you're likely to find existing code similar to what you're currently trying to achieve that you can use to guide you.
If possible, make use of code reviews with colleagues.
- Search for unique-looking strings from whatever view you're trying to modify to find the associated html/js file.
- Pull up the commit history for the file to see what other files were modified along with it the last few times. This will give you dependencies and linkages.
- Make your change and then ask your ide/command line to find all typing/lint errors in your project which will help you find other dependencies you may have missed.
- If you get stuck, reach out to the authors or reviewers of previous PRs. (Hint: you may want to include them as reviewers. They'll give good feedback and you'll engender good will by keeping them in the loop when you touch their corner of the codebase)
- Write a few solid unit tests. Maybe even clean up the testing code a little bit while you're there.
- Write a concise but informative description of your changes in your PR. If you made two or more logical changes, split your PR and stack them. Your teammates will appreciate the shorter PRs and you will get feedback more quickly.
- Land the PR in a timely manner and keep an eye on it until it hits prod.
- Once in prod, test it yourself and keep an eye on the logs for a day or three.
- Bonus: put all changes behind feature flags to do slow rollouts and so you can quickly revert without waiting for a deploy. Make a task to remind yourself to remove the dead code behind the flag in month or so when you're pretty sure it's stable.
Git blame is also super helpful when reading individual files, as it gives you a line by line commit history.
I recently used it in a major re factor of a file (I didn't write) as it gave me background info (commit messages) on why the code ended up looking like that.
If it's C, cflow (example usage: cflow .c .h) and doxygen (needs some config, but there's a GUI) could come in handy. Doxygen is capable of generating call graphs if you install some external dependencies.
If it's a different language, you could try looking around in the docs if anyone's generated callgraphs, or you could look up ways to do so.
To trace the code you could use a debugger -- e.g. if it's gdb just issue the command 'start' and then step through from the main function to see how things go. Or (assuming it's something like a C program) you could get an strace (example usage: strace -vvvttf -o strace.log ./program) and maybe get a feeling for the config/etc. files read or written to, network services accessed, etc.
It would help if you could tell us what kind of program it is, or what kind of programming language it's written in.
Assuming sourcemap pipelines work, open your browser and put debuggers on anything you don't understand tracing requests / invocations. Start with something conceptual like a particular page / component and work your way out beginning at the request entrypoint. Along the way you will likely figure out things like authentication management, routing, view hierarchy and design patterns leveraged, etc. Allow a particular issue you need to resolve guide you if needed, but don't sweat the learning curve, as it is required and should be expected.
Find out how to run unit tests in your project and then find interesting unit tests to run. Set break points preferably at some launch points of your code and debug through unit tests. The process will take you through relevant parts of the code, the decisions being made etc.
This method has also worked for me when building features in a large codebase. Write a unit test first and then keep checking where the code breaks and fix those until your test succeeds. This is effectively TDD. Note that you might have to refactor the code for better design but it gets you started towards understanding the flow.
For example, let say I'm working on an ecommerce system and I try to understand what happens when a buyer adds an item to its cart. I'm going to put logs / debugger's prompts on the important steps of that operation. Next to those, add a comment that describes why this step is important. The important thing is to label the comment with incrementing numbers. Those numbers allow me to keep track of the order of execution.
Finally, commit this to a dummy branch and share it with your co-workers if both of you are new to this code base.
That's one of the first thing I do when I jump in a new codebase. Pick something that interests you, and log the whole operation.
Start by adding breakpoints for some key actions of the app. Then step through the flows with the debugger. First pass you can just step over functions to get the high-level idea. In subsequent passes, you can step into functions that seem important. Rinse and repeat until you understand those actions well. Then move on to other areas in the app.
This works because you can see the actual end-to-end execution flow (not always clear from reading code), inspect runtime data (impossible by reading code) and even change the runtime data (variables, DOM) to validate assumptions about how the code works.
I suggest you try JetBrains IDEs depending on your tech stack. They have got an incredible search capabilities. You can search for a word/function where it has been used across project, you can find and replace across the project, etc. It's amazing!
Wouldn't Vscode alone can do the trick?
I have been using Vscode previously and could easily find implementions/definitions inside the project by searching.
Never used JB webstorm before, there will definitely be a learning curve with this new IDE. I am not sure the benefits weigh more than the time i have to get used to new IDE.
VSCode is pretty barebones. You'll need to setup a language server and figure out how to search everywhere and also learn grep in the process if you don't know it already.
In Webstorm, you just press on Shift twice and a popup appears that allows you to search for everything you need. Since the IDE indexed all symbols in your project already, you'll be able to quickly find whatever you need.
If the project you are working on has enough time to burn, and the codebase has no (or near to) unit tests, add them.
Find large utilities that are not coupled with a particular part of the application, and put them under heavy testing.
Do not fix any failing test before having completed the test suite.
If you have access to other developers that know the codebase, review the tests with them, and fix failing tests together.
Do the same for the frontend / interface code, but do not go for unit testing imho, go for visual regression testing.
And the end of the process you will have a very large knowledge of the codebase, and you will have improved it at the same time.
You can create regression tests that prove that the code does what it does today - which helps you understand it, provides a readable description of what it does and protects against regression bugs in the future.
First of all you’re not going to be all by yourself. Ask previous developers or peers if there are any. Ask them to help you navigate the codebase and find good candidates.
On the opposite scenario, where you have to take ownership of a large codebase and there’s no one around that can guide you through it, if every time you learn something about it you turn it into a test case you will do your future self a favor and at the same time also tracing your progress.
1. Get confortable using “grep” [1], or better, “ripgrep” [2], which is quite faster than the former. They are both available in Linux, macOS, and Windows via WSL.
2. If the project uses a version control system (Git, Mercurial, Subversion, etc.) then take a look at the most recent additions, modifications, and/or deletions in the version control log (git-log, or whatever you want to call it). Sometimes, the most relevant files in a project are the ones people modify the most… obviously, ignore files associated to third-party dependencies (vendor, node_modules, that kind of stuff).
3. Install a Language Server Protocol (LSP) server [3] with support for the programming language(s) that you are going to use. Configure your favourite code editor to take advantage of as many LSP features as possible, with enphasis on “Jump To Definition” and “Find References” [4].
Tell us what programming language(s) is the project written in to give you more suggestions.
2 and 3 are great points, but honestly - and especially if you're working in multiple repos, or multiple subdirectories in a monorepo - VS code's folder search is better than grep. It'll find you the exact place in the file, and a click will take you there, nice and easy. I've more or less forgotten the quirks/flags of grep, because I haven't needed to use it in a while.
This opens a new tab with the grepped output. I simply navigate to the line so my cursor is on the filename, then
C-w gf
That's [Ctrl-W] followed by [g] and [f]
And just like that, the file is open in yet another tab. The best part? Since the grep output is just another vim buffer, I can search it as well, just like any other buffer.
And yes, ofc the above can be put into a custom command, just by putting this in the .vimrc:
command -nargs=1 Gr :tabnew | r ! shopt -s globstar && grep -sn <args> **
What if I told you VS code search is ripgrep? Pretty much every coding focused editor has built-in ability to view grep results, then click on the result to jump to the file location. It's 1980's UI technology.
Tangential wish - I'd love it if someone figured out a good/generally working way of jumping to definition/references through a URL.
E.g. from `axios.get(`/users/${currentUserId}`) in frontend code jump to `class UserSingleView: def get(self, user_id):` in the backend, and vice versa.
As someone who has worked for years on a UI consuming such information: it's bloody difficult. Each language has its own weirdnesses, you almost always have to build the code to figure out the cross references, and what's correct to the compiler is often non-intuitive to the user. Prime example: C++ code mixing macros and templates. There isn't even a good definition of what the definition is!
I probably do underestimate its difficulty, but certainly appreciate it's bloody difficult. Hence I'd love someone else to do it! I don't have the appetite for it, but it would be fantastic to use.
That is such an annoying thing in codebases. It's the same issue the other way around, I want to see all places in frontend that calls /v1/resource but then some people go and abstract the url into multiple variables.
If it was up to me the constant part of the relative url would never be abstracted away in the codebase. Just a simple grep would instantly get you everything you are looking for.
89 comments
[ 3.8 ms ] story [ 170 ms ] threadMake sure you're able to efficiently navigate through the code. If your IDE supports jumping to declarations and usages, make sure you get working.
Look at old commits to see how changes were added before, that will hint at how future changes should look.
Ask others for help if something doesn't make sense.
One of my favorites is “how did you know to do that?”
Find out the deployment process. Find out what to do if anything goes really, really wrong. Try to find patterns in the code. Just explore for a while and even try to create you own mental model of how things are structured.
Not sure when a bug was introduced? Write a test then bisect (if you can't automatically bisect frontend code you have other problems, but manual bisects work in a pinch).
Not sure why something was written the way it was? Hey sometimes this stuff just evolves naturally, try looking back through blame history to see how different commits effected the code.
Not sure what needs touching to add a new feature? Try seeing if you can find a similar feature that someone else wrote in the past.
To see how it uses the backend, open the network tab in Chrome developer tools (or equiv in other browsers), look at what requests it makes, and then find corresponding controllers within the code.
If there is no unicode setting "LC_ALL=C" can speed things up considerably too.
If it’s not already TypeScript, add a tsconfig.json, with allowJs: true, checkJs: true. You’ll probably need some tweaks from there. But merely adding the config is enough to kick VSCode or other language service providers into finding references and a lot more.
Apart from that:
- Keep notes as you uncover how things flow and interact.
- The best way to keep those notes is as part of your project’s documentation. Use JSDoc and TypeScript declaration files. Learn how these docs interact and aid navigation.
- @see is a particularly useful JSDOC tag!
- Speed up the build if it’s slow. No really, it’s going to help with discoverability in the codebase! Breaking flow to wait for a build is a sure way to lose track of your journey.
- Backfill tests before you touch implementation. Even if you suspect the tests might be redundant. Odds are they’re not, but even if they are, (1) that’s more documentation you can reference and (2) any other maintainers/contributors seeing redundancy can use that as a signal to shortcut your familiarization.
- Git blame is your friend. Some days it’s your best friend.
- Look in weird places! Sometimes that old version of a dependency is pinned for a reason. Sometimes it’s pinned to a fork, again for a reason.
Git grep (or ripgrep) to find usage - useful when refactoring and to see how data is used/accessed and where it's passed around. Also useful to note where certain data doesn't show up: you can infer some structure from this.
Looking at when something was last changed with git blame can be useful. Is something suddenly broken, but hasn't been changed in 5 years? Could give an indication of where not to look on a first pass.
Break some things (locally) on purpose. Get a feel for how errors bubble up through the application and how dependant code behaves when something is wrong.
Look over the last handful of PRs/merged patches. It can be helpful to see these smaller pieces of code, the changeset, and their associated context - whether it was a feature, a bugfix, and what the code was supposed to achieve.
Use existing code in the codebase as a styleguide. Most work on large codebases isn't groundbreaking or innovative, so you're likely to find existing code similar to what you're currently trying to achieve that you can use to guide you.
If possible, make use of code reviews with colleagues.
- Pull up the commit history for the file to see what other files were modified along with it the last few times. This will give you dependencies and linkages.
- Make your change and then ask your ide/command line to find all typing/lint errors in your project which will help you find other dependencies you may have missed.
- If you get stuck, reach out to the authors or reviewers of previous PRs. (Hint: you may want to include them as reviewers. They'll give good feedback and you'll engender good will by keeping them in the loop when you touch their corner of the codebase)
- Write a few solid unit tests. Maybe even clean up the testing code a little bit while you're there.
- Write a concise but informative description of your changes in your PR. If you made two or more logical changes, split your PR and stack them. Your teammates will appreciate the shorter PRs and you will get feedback more quickly.
- Land the PR in a timely manner and keep an eye on it until it hits prod.
- Once in prod, test it yourself and keep an eye on the logs for a day or three.
- Bonus: put all changes behind feature flags to do slow rollouts and so you can quickly revert without waiting for a deploy. Make a task to remind yourself to remove the dead code behind the flag in month or so when you're pretty sure it's stable.
I recently used it in a major re factor of a file (I didn't write) as it gave me background info (commit messages) on why the code ended up looking like that.
If it's a different language, you could try looking around in the docs if anyone's generated callgraphs, or you could look up ways to do so.
To trace the code you could use a debugger -- e.g. if it's gdb just issue the command 'start' and then step through from the main function to see how things go. Or (assuming it's something like a C program) you could get an strace (example usage: strace -vvvttf -o strace.log ./program) and maybe get a feeling for the config/etc. files read or written to, network services accessed, etc.
It would help if you could tell us what kind of program it is, or what kind of programming language it's written in.
This method has also worked for me when building features in a large codebase. Write a unit test first and then keep checking where the code breaks and fix those until your test succeeds. This is effectively TDD. Note that you might have to refactor the code for better design but it gets you started towards understanding the flow.
For example, let say I'm working on an ecommerce system and I try to understand what happens when a buyer adds an item to its cart. I'm going to put logs / debugger's prompts on the important steps of that operation. Next to those, add a comment that describes why this step is important. The important thing is to label the comment with incrementing numbers. Those numbers allow me to keep track of the order of execution.
Finally, commit this to a dummy branch and share it with your co-workers if both of you are new to this code base.
That's one of the first thing I do when I jump in a new codebase. Pick something that interests you, and log the whole operation.
Start by adding breakpoints for some key actions of the app. Then step through the flows with the debugger. First pass you can just step over functions to get the high-level idea. In subsequent passes, you can step into functions that seem important. Rinse and repeat until you understand those actions well. Then move on to other areas in the app.
This works because you can see the actual end-to-end execution flow (not always clear from reading code), inspect runtime data (impossible by reading code) and even change the runtime data (variables, DOM) to validate assumptions about how the code works.
Never used JB webstorm before, there will definitely be a learning curve with this new IDE. I am not sure the benefits weigh more than the time i have to get used to new IDE.
In Webstorm, you just press on Shift twice and a popup appears that allows you to search for everything you need. Since the IDE indexed all symbols in your project already, you'll be able to quickly find whatever you need.
Find large utilities that are not coupled with a particular part of the application, and put them under heavy testing. Do not fix any failing test before having completed the test suite.
If you have access to other developers that know the codebase, review the tests with them, and fix failing tests together.
Do the same for the frontend / interface code, but do not go for unit testing imho, go for visual regression testing.
And the end of the process you will have a very large knowledge of the codebase, and you will have improved it at the same time.
Here's a relevant trick I frequently use for this kind of work with Python: https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests...
On the opposite scenario, where you have to take ownership of a large codebase and there’s no one around that can guide you through it, if every time you learn something about it you turn it into a test case you will do your future self a favor and at the same time also tracing your progress.
2. If the project uses a version control system (Git, Mercurial, Subversion, etc.) then take a look at the most recent additions, modifications, and/or deletions in the version control log (git-log, or whatever you want to call it). Sometimes, the most relevant files in a project are the ones people modify the most… obviously, ignore files associated to third-party dependencies (vendor, node_modules, that kind of stuff).
3. Install a Language Server Protocol (LSP) server [3] with support for the programming language(s) that you are going to use. Configure your favourite code editor to take advantage of as many LSP features as possible, with enphasis on “Jump To Definition” and “Find References” [4].
Tell us what programming language(s) is the project written in to give you more suggestions.
[1] https://en.wikipedia.org/wiki/Grep
[2] https://github.com/BurntSushi/ripgrep
[3] https://microsoft.github.io/language-server-protocol/impleme...
[4] https://langserver.org/
[1] https://github.com/microsoft/vscode-ripgrep
And just like that, the file is open in yet another tab. The best part? Since the grep output is just another vim buffer, I can search it as well, just like any other buffer.
And yes, ofc the above can be put into a custom command, just by putting this in the .vimrc:
Then the whole thing can be used likeThe lineage is grep (C, 1973) -> ack (Perl, ~2006) -> ag (C, ~2011) -> rg (Rust, ~2016)
Ack inspired a wave of tool rewrites to accommodate better interfaces and usability.
git log --name-status --pretty=format: | sed 's!.*/!!' | grep java | cut --fields=2- | sort | uniq --count | sort --numeric-sort --reverse
E.g. from `axios.get(`/users/${currentUserId}`) in frontend code jump to `class UserSingleView: def get(self, user_id):` in the backend, and vice versa.
If it was up to me the constant part of the relative url would never be abstracted away in the codebase. Just a simple grep would instantly get you everything you are looking for.
And as part of getting comfortable, the -A and -C options are particularly useful.