Show HN: Visualize the entropy of a codebase with a 3D force-directed graph (github.com)
I work at a startup where business evolves really fast, and requirements change frequently, so it's easy to end up with big piles of code stacked together without a clear structure, specially with tight deadlines. I made dep-tree [1] to help us maintain a clean code architecture and a logical separation of concerns between parts of the application, which is accomplished by: (1) Visualizing the source files and the dependencies between them using a 3D force-directed graph; and (2) Enforcing some dependency rules that allow/forbid dependencies between different parts of the application.
The 3D force-directed graph visualization works like this: - It takes an entrypoint to the codebase, usually the main executable file or a library's entrypoint (index.js, main.py, etc...) - It recursively crawls import statements gathering other source files that are being depended upon - It creates a directed graph out of that, where nodes are source files and edges are the dependencies between them - It renders this graph in the browser using a 3D force-directed layout, where attraction/repulsion forces will be applied to each node depending on which other nodes it is connected to.
With this, properly decoupled codebases will tend to form clusters of nodes, representing logical parts that live together and are clearly separated from other parts, and tightly coupled codebases will be rendered without clear clustering or without a clear structural pattern in the node placement.
Some examples of this visualization for well-known codebases are:
TypeScript: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F...
React: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F...
Svelte: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F...
Langchain: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F...
Numpy: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F...
Deno: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F...
The visualizations are cool, but it's just the first step. The dependency rules checking capabilities is what makes the tool actually useful in a daily basis and what keeps us using it every day in our CI pipelines for enforcing decoupling. More info about this feature is available in the repo: https://github.com/gabotechs/dep-tree?tab=readme-ov-file#che.... The code is fully open-source.
61 comments
[ 0.87 ms ] story [ 128 ms ] thread> I work at a startup where business evolves really fast, and requirements change frequently, so it's easy to end up with big piles of code stacked together without a clear structure, specially with tight deadlines
That smells.
It sounds like the team could benefit from better stack technologies and a bit more discipline in how it is applied to solutioning.
> Enforcing some dependency rules that allow/forbid dependencies between different parts of the application.
What is the alternative to this tool that lowers the cognitive barrier / builds the right muscles for the team to understand what they should / shouldnt depend on?
For our specific case it's actually pretty good, we've built a lot of discipline around maintainability, but in general this is a recurring problem in tech teams who might not be able to afford the time it takes to gain discipline.
> What is the alternative to this tool that lowers the cognitive barrier / builds the right muscles for the team to understand what they should / shouldnt depend on?
Some programming languages allow you to split the codebase into modular units (npm workspaces, cargo workspaces, etc..) which forces developers to modularize things, and dependencies between modules need to be explicitly declared.
This is good, but usually not enough, as nothing prevents you to mess things up within a module/workspace.
There's some other tooling with similar functionality to dep-tree, but language-specific and with visualizations not suitable for large codebases (.dot files, 2d svgs...)
These all help to lower the cognitive barrier to learning and maintaining the code base effectively. For developers new to the code base they help with learning and for those more experienced they help with ongoing design and maintenance.
Most long-lived code bases I've seen have adopted or built such tooling at some point, often with tools customized to the code base. For example in one large code base (c. 250 devs) we built tooling that simulated and helped optimize the changes to implement a major refactor of the overall module structure.
Maintaining a code base requires communication, PR reviews and discipline. That doesn't always happen.
Having lint check rules is brilliant. Never mind discipline, you just need a friendly error to say don't import services into an ORM model file. I'm going to adopt this right away.
This is why we see all these products targeted at companies with 24 microservices with 26 developers who have to run end to end testing on everything.
Architectural erosion is primarily a cultural issue and any tool that helps people discover and call out architectural violations is potentially useful.
Many companies can't just do the inverse Conway law, and if you look at the state of devops report, note how they call out CAB forums and controls being problematic for even high performing companies to become elite.
This product as an example, which just really means you want to keep k8s but have given up on loose coupling and high cohesion.
https://www.signadot.com/blog/how-uber-and-doordash-enable-d...
Throwing products at structure problems typically doesn't work.
To me the usage makes no sense whatsoever, there's no entropy to be found here, and if there is it's not what is being displayed. Perhaps it makes sense to people who don't know what entropy is, though most of them wouldn't know the word in the first place.
It's abuse of technical language in efforts to sound impressive, in my opinion, which I guess is a valid form of language evolution. There are other words in our lexicon that have become general purpose and fuzzy in spite of their precise technical origins.
a. avoid making certain subsets of people think you're using precise concepts that you aren't. b. make it easier for people that don't know what entropy even is to understand what this tool does. Disorder is a far more widely understood term in my view.
The visualization here is amazing in its own right as well, can I ask what part of the codebase renders it and handled the force-directed part?
Force-directed is an algorithm for displaying graphs in a 2d or 3d space, which simulates attraction/repulsion based on the dependencies between the nodes, the wikipedia page explains it really well https://en.wikipedia.org/wiki/Force-directed_graph_drawing
> Love it, I think dependency trees are super underused data for static analysis.
Definitely, specially for evaluating "the big picture" of a codebase
One piece of feedback, if I may. It's really difficult to read the blue labels against the black background. Is there any way to change the palette colors?
[1] https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F...
That's definitely is an improvement point, I have just calibrated things looking at my screen, which might have a high saturation/brightness setting.
Thanks for the feedback!
[1]https://erikbern.com/2016/12/05/the-half-life-of-code.html
Edit: oh, looking at the docs, apparently that's exactly what this tool does. Though it would be nice to have function level granularity. Maybe by annotating the code itself.
[1] Asciidoc/RsT (include directive for both), XML (DITA/S1000D/DocBook/etc, each with different transclude mechanisms), any markup that supports transclusion.
Lets say my project looks like this:
src/example/foo.py
src/example/bar.py
And If bar.py containse the statement "from example.foo import Foo" there is no link between the files foo and bar. Though, if the statement is "from .foo import Foo" it shows a link.
export PYTHONPATH=src
I thought this could be solved by changing the directory to src/ and then executing that command, but this didn't work.
This also seems to be an issue with the web app, e.g. the repository for the formatter black is only one white dot https://dep-tree-explorer.vercel.app/api?repo=https://github...
I've seen people using tricks like the `sys.path.extend(["src"])` in the main file for being able to place source code into an `src` folder, but unfortunately, dep-tree is not able to take that into account.
From the visualization perspective, it reminds me a lot of Gource. Gource is a cool visualization showing contributions to a repo. You see individual contributors buzzing around updating files on per-commit and per-merge.
https://github.com/acaudwell/Gource
Golang is very challenging to implement, because dependencies between files inside a package are not explicitly declared, you can just use any function from any file without importing it as long as they both belong into the same package, so supporting Golang would probably require spawning an LSP and resolving symbols.
The reason for implementing dep-tree in Go was because things were going to get algorithmic af, and better to choose a language as simple as possible, knowing that it also needed to be performant.
https://primitive.io
___
0. https://nodejs.org/api/packages.html#subpath-patterns
React's graph looks like a mess. Why am I not surprised...
Graphs are wonderful abstractions for the structures that arise in many kinds of engineering, but you need to focus on understanding those abstractions, not just pictures rendered by heuristics. Visualization can be wonderful, but has its limitations, especially when used out of the box.
For some projects, you need to think really hard to design it correctly. The most extreme experience that I've had of this was when I was working in the blockchain sector.
Initially, when I joined, the project was a tangled mess. Every module was connected to many other modules without clear separation of concerns and with tight coupling.
For the refactoring, we extracted the core crypographic logic and separated it from the network/P2P logic which we re-wrote from scratch. I designed it so that the P2P module would be fully data-agnostic; meaning that it would have no concept of what kind of data it would have to propagate through the network. It was a significant challenge to come up with such design while also supporting features like peer banning, peer selection, peer shuffling, preventing messages from rebounding back to the sender, preventing spam and duplicates... During the design phase, I was tempted many times to add some kind of business domain awareness to the P2P module but managed to resist until the project's completion.
The result was that the P2P module ended up with a very simple interface and was very versatile. Because it wasn't tied to any specific business domain, it could be used for a broad range of different blockchain consensus mechanisms and didn't require any code updates when business requirements changed. This was useful to us at the time since we had not settled on most details of our consensus mechanism. Also, it could be used for a wide range of other P2P use cases beyond blockchains; later, I was able to use that exact same module (without any changes) to build a DEX (decentralized exchange) with only about 4000 lines of additional custom code.
More interestingly, there was another blockchain project which was similar to the one I was working on and they also decided to have a P2P module but their module had awareness of business domain concepts such as 'transactions' and 'blocks' and their code was much messier, much longer and not reusable at all. They had to update it almost every time their business domain requirements changed and it wasn't as reliable.
It was the exact same problem, but the second project gave in to the temptation of sharing business domain responsibilities across multiple modules (low cohesion) and this led to tight coupling.
Always interesting to see different ways of visualising the same thing. A while ago my friend and I also made a codebase visualisation tool ([https://www.codeatlas.dev/gallery](https://www.codeatlas.dev...), but instead of taking the graph route, we opted for Voronoi treemaps in 2D! It's a tradeoff between form and function for sure, modelling code as a DAG is definitely more powerful for static analysis. However, in most graph-based visualizations (this, gource) I just find myself getting lost super quickly, because the shapes are just not very recognisable.
Really impressed by how polished this already is, nice docs, on-the-fly rendering, congrats!
If I ever find time to work on codebase visualisation again, I might have to steal the idea of codebase entropy to better layout which files to place close to which others!
Would love to see a tool that could automatically break these dependencies into optional features within their crate. It felt like a poor use of my time to track everything down manually.
https://youtu.be/oyLBGkS5ICk?si=cawjnPnR9riEyvf2
Out of interest, I'm thinking how this sort of method works if you ignore the semi-arbitrary distinction between your own code and other libraries. If, say, an array class is used everywhere, wouldn't that look like a bad pattern on the dependency graph? Or is there a way to read the graph that tells you that your pervasive use of np.array is still appropriately decoupled?
If a node is depended upon a lot, all the resulting edges induce weaker forces to adjacent nodes, so this accounts for the fact that some files will be depended upon a lot, and that's fine.
There's also the option to just exclude that kind of files from the analysis with the --exclude flag. I've found that to be useful for massive auto-generated files.