Launch HN: Buildt (YC W23) – Conversational semantic code search

Some demos: https://twitter.com/AlistairPullen/status/162848600700289433... and https://twitter.com/AlistairPullen/status/162848600806408601...

We’ve been devs on projects ranging from mobile apps, arbitrage trading systems, VR platforms to on-demand startups. Without fail, whenever a codebase gets over a certain size or we inherit legacy code, we get slowed down from not knowing where a certain snippet lives, or how it works. I’m sure we’ve also bothered our colleagues when we first get onboarded for longer than they would like.

Current code search products aren’t too different from CMD + F. We’ve often wanted results that aren’t captured by string matches or require some nuanced understanding of our codebase—questions such as “How does authentication work on the backend?”, "Find where we initialize Stripe in React”, or “Where do we handle hardware failures?”

To build a tool to help developers quickly search and understand large codebases requires contextual understanding of every line of code, and then how to surface that understanding in a useful format.

First we need to parse your codebase; this isn’t a walk in the park as we can’t simply embed your code files because in that instance if you were to surface a result for a specific search you’d only be brought to the file that the result was in, and no deeper. To be able to find specific snippets of code you’re looking for, we need to be much more granular in how we split up your codebase. We’ve used a universal parser (TreeSitter), so we can traverse the Abstract Syntax Tree (AST) of your code files to pick out individual functions, classes, and snippets to be embedded; not the entire file. This allows us to work on your codebase on a more semantic level than the raw source code.

Once we have extracted all of the relevant code from the AST, we have to embed them. (We use a number of other search heuristics too, such as edit distance and exact matches, but embeddings are the highest weighted and core heuristic.) We’ve learned a great deal about the best implementations of embeddings for this use case, particularly in this case when using embeddings to search between modalities (natural language and code) we found that hypothetical search queries were the optimal way to surface relevant code, as well as creating a custom bias matrix for our embeddings to better optimize them at finding code from short user queries. Simply embedding the user’s search query and searching the answer space with it was a poor solution.

One embeddings heuristic we use is a HyDE comparison, which involves using an LLM to take the user’s search query, and then generate code that it thinks will be similar to the actual code the user’s trying to find. This process is well documented and has given us a huge increase in performance (https://www.buildt.ai/blog/3llmtricks). Another heuristic allows us to achieve “search for what your code does, not what it is” functionality—this involves the embeddings gaining some form of understanding of what the code actually does. For this we used embeddin...