Ask HN: How do you search large codebases before adding a feature or fixing bug?
Why do we need to search source-code?
1. Quickly learn the domain and context of the application
2. After adding a feature, we should aware if we broke anything (assume you work with code that doesn't have test-case), it helps even to search testcases
3. Find similar code and ensure you are improving quality of the overall similar code (not just fixing current bug)
4. Understand how application behaves when there are production issues.
You as a developer, what are all the ways you search source-code before working/fixing feature or bug? Do you use any CLI tools other than grep. I have used OpenGrok, But few times it is not maintained by me or other developers.
Below is my steps.
1. Read the relevant code, and know certain domain keyword, variable names (inclusive class/method/function)
2. Use the bitbucket/GitHub/git search
3. Use the grep
4. Use the git-grep
Seems like this is (Especially CLI based search) very valuable skill to have. Do you have any tips/tools for other developers?
86 comments
[ 5.4 ms ] story [ 154 ms ] threadYou don't always have to optimize for the right answer right away. Just asking someone who's worked with/on the codebase will often give you a jump start.
At any rate, I've got 32 GB of ram and I'm operating off an NVMe-drive, I really don't see what the problem is.
Honourable mentions to cscope and ctags. They work for me since most of my $dayjob involves me mucking around with C++.
All tools get invoked from within Vim. (Which _also_ works reasonably well in Windows Terminal).
Septum is neighborhood based (context-based) search, so you can find contiguous groups of lines which contain specific things, but exclude other things. It's also interactive so you can add/remove filters as needed. This makes it useful for those cases where terms change based on their context so you can exclude terms related to the contexts you don't want to keep. It reads .septum/config which contains its normal commands to load directories and settings, so you can have different configs per project you're working on.
Is there an easy way to install this on non-windows systems? I'm not familiar with Ada toolchains.
Wish to hear why did you choose to use Ada?
I wrote an article for these folks which includes some of the rationale here: https://blog.adacore.com/ada-crate-of-the-year-interactive-c...
It's surprising to me how much effort is not being put into whole-org code search. Most projects focus solely on single-repo search. If you need to make breaking changes or find examples and you don't even know where to look, single-repo search isn't so useful.
[0] https://github.com/hound-search/hound
How good are you at reading code, finding out how smaller parts work in a larger system and understanding the context&domain is a large part in how good you are.
Not so good developers, when they are not so good at this, often start blaming the system and people who have worked on it.
Not so good developers may however be able to handle smaller systems (in particular ones written in their favorite tech stack), and this experience leads them to erroneously believe they are good.
As OP says, one of the skills/talents that differentiates good vs not good developers is the ability to rapidly "grok" enough of a large codebase to figure out "where to tap". ( https://quoteinvestigator.com/2017/03/06/tap/ )
> Being able to navigate a large, complicated code base is a really important skill, and I think one of the important marks of a senior engineer. A big part of it is also just having the grit to dive in and understand a messy system without having things neatly laid out for you.
Says ~mostly the same thing, but without the condescension.
- Sourcetrail (GUI/Linux/Windows, closed-net-capable, archived) - https://github.com/CoatiSoftware/Sourcetrail
- SourceInsight (repo/web-server/closed-net) https://www.sourceinsight.com/
- OpenGrok (web server plug-in/Java/closed-net) https://oracle.github.io/opengrok/
- CLion (GUI-based IDE) - By IntelliJ/JetBrains - https://www.jetbrains.com/clion/
- SourceGraph (Web-based) https://about.sourcegraph.com (thanks, gravypod)
- Codesee.io (GitHub/web-based) - https://www.codesee.io/privacy-and-security
For free as in beer, I prefer OpenGrok so I can get more than JetBrains
https://about.sourcegraph.com/
At the root level I maintain two scripts:
clone.sh has one line per repo. When I run low on disk space I sometimes delete larger repos. The clone script allows me to easily re-clone everything in this case.update.sh is similar but pulls all repos.
For global search across all branches I do:
(when I forget the line I look it up in Stack Overflow [1])This is especially useful since I work a lot with Bitbucket and to the best of my knowledge you can only search the default branches there.
When I know the branch, but want to search across all history I use git pickaxe, aka
git -S ...
All of this is not very sophisticated and takes a lot of disk space but it works pretty well for me.
[1] https://stackoverflow.com/a/15293283
- grep
- ag - Same as grep, but faster!
- find - when looking for a file by name
- helm (an epic Emacs package which does interactive search)
Used to work at a Windows shop and we used Entrian in visual studio. That was pretty good, bust closed source and a pain to setup.
We use bitbucket at work which goes out of its way to stink regarding search. But I personally find it hard to find some code in some branch of some repository that uses x.
* Except at the windows shop where Azure Devops would provide ways to search multiple TFS branches of our product. Life is much better in companies where the master branch is the source of truth and you don't have to maintain different branches.
1. Try to find out which framework, architecture, design patterns used - get hold of that
2. Library dependancy 3. Database structure
4. Pick up your favourite editor (be it vim or emacs or vs code or any) in which you have mastery
5. Search for various entry points like routes, or start activity or main function etc & try step thru code (with possible debug tools open)
For example:
def foo(args):
(NOTE There is call tracing logic built into some languages but it doesn't always work for some complex code bases; try it before you write your own.)If the code creates html elements, my script adds attributes to the html element to link back the source code location so I can look at the html and figure out where the elements were created. If the html is built using templates, then I add html comments to the template so I can tell where they are used in the final page.
Then I test the app and look at the traces to figure how it works.
At first I trace everything but once I get to know the code I add the tracing to the areas that matter. I don't check in this code.
(1) breakpoint debugging, finding the connection between program start and various features
(2) Doxygen to generate a dependency graph
(3) create json performance profiles, manually instrumenting functions, and navigate traces using Google Chrome about://tracing or similar tools.
(4) trace and look at the data input and output, using a hex editor or over the network using wireshark
Then use Vim to read the concatenation and (regexp) search.
I use Vim on a concatenation. It's a simple technique that allows me to search, read, annotate, and understand how everything fits together in a medium sized C++ codebase full of templates (i.e., a pile of garbage).
I get that new devs need a mindset to understand the existing approaches, but tossing them right into the middle of the foray actually makes things worse.
And yeah, teams need to be able to respect and respond to the frustrations of new developers. If a new developer, especially a senior developer, needs hours or days and multiple approvals to do something as simple as change a string, for instance, something is seriously wrong and you need to actually re-evaluate what everyone is doing.
Yes, I've experienced that exact phenomenon, and it's amazing how little people are concerned by that kind of thing. I'm not the greatest programmer in the world by any means, but I know how to change a string and when a design is ridiculously inefficient. Yes, there was a project I was on-boarded to where one of the tasks was changing how a string appeared in the UI; it turned out there was a shared library carrying the target string, but it wasn't exactly obvious that was where the string was coming from, and multiple other projects were using that library and so the review process needed to apply to all projects involved. For some reason, although a role was delegated to deploying the app itself, it's up to individual devs to build the library and register it as an NPM package, and there was a tight protocol around. Because of the toolchain, the library was not easy to test with an app locally. So it ended up that changing a string could take several hours or even days, especially if someone wanted to squeeze in another string change into that same task.
As a developer who just joined a team, what are you really supposed to do? Just plow through believing that there's a good reason for the madness?
Maybe the existing approach is actually bad.
100%. That can be very, very hard, but yes. It's also got to go both ways. New devs need an MPP for the project, and the rest of the team and management needs an MPP in relation to the new dev.
> if the product is working well, the release cadence is acceptable, the defect rate is low, etc, I think the prior devs have earned a reasonable benefit-of-the-doubt MPP that a new dev should accommodate for learning the existing approach and rationales.
That's often true, though release cadence and defects in production can also be low if you hire a crapton of devs to work on the same project despite any present dysfunction. So that may be something to keep in mind, too. And maybe that means the new dev realizes that they need to change their expectations such that the underpinnings of a project are conceptually flawed and that it's really their job to deal with that given that the company has plenty of revenue that they can just hire however many devs it takes to keep things running.
> I've seen a senior dev completely change tooling and build process for a "fine" app at the end of their second week. All good ideas in the right time, but...
Haha, ohhh yes, I've been on both sides of that coin. This problem really highlights how the difficult part of coding isn't so much the code but the people (and in a good way!). I've made that mistake of coming in and fiddling with, kind of wrecking, the existing process. Some of that is overambition, some of it is the lack of formal training in our field, and some of it is the ambiguity of roles that is quite common. The word "senior" can illicit this self image that you can and should make sweeping decisions. I've appreciated it when teams have been able to communicate expectations upon joining rather than just saying "it's up to you" for everything; that is until you actually do change something like a part of the toolchain, resulting in technical and interpersonal issues.
In my next role as a senior, I plan on going in with the mindset of not changing the foundation of a project except requesting removing the spandrels and canards that no longer serve a purpose.
I had even made self-serve hosting for it but didn't put much effort into monetizing or otherwise promoting it.
[0] https://github.com/hound-search/hound
Because your time is so much more valuable than theirs!
In real life, I'm absolutely considerate of people's time. I just don't hesitate to ask questions early on and give others an "out" if they're strapped for time. It's just that I don't get it into my head that their time is so much more valuable than mine that I can't interrupt them to ask dumb questions.
Not all are that way though, which can be annoying, but i sincerely hope they can recognize a question only comes up because it is needed to understand the subject.
Tight deadlines or production issues should take precedence though, but other than that you really should try your best to help.