I believe HN now has some built-in detection for AI generated comments, which is probably why it was dead (pangram rates it as 100% AI generated, btw).
I also built something similar to solve this, but more aimed at inspecting the impact of changes to a concept - think a vertical slice or domain, or even a feature. Often the same thing might be referred to by similar, but slightly different names, or separate pieces of the codebase might not immediately look related when they do share dependencies. So symbols, phrases, and words are attached to concepts for either LLM-based or user-based querying via MCP.
An indexer crawls the codebase and maps slices of code to concepts, then groups related concepts. You can inspect a concept to see things related to it, or ask about the impact of a change and see second-order changes required - things not so easy to see that are related by basic symbol linking (although it does use symbol linking - it heavily relies on Roslyn, since it's all C# focused).
It also returns confidence scores with everything, which adjust based on a dozen or so factors like whether the user defined the concept or the concept was auto-generated during the code crawl. Low confidence score === fall back to grep.
It was a super fun project and ended up being very useful, as much for people as the LLM. But, I'm not convinced it is materially better than grep in some ways and just haven't had the time to really focus on improving it.
> So I built Pensieve. It's a local index of your code, documents, and past agent sessions. Any agent can query it over MCP and get the few files that matter on the first try
What are the various ways Pensieve is superior to iterative grep? Cache invalidation is hard - how does Pensieve address it?
Could you take a few minutes to explain the what and the how? Links to writeup to blog posts or files in your repo would be very helpful.
Finally, can I use Pensieve with Paseo? What if I wanted to expose Pensieve's capability to my Paseo setup?
this has made my M4 Macbook Pro with 25 GB RAM unusable. Pensieve is just spamming it with new python processes. When it's not eating up my CPU cycles, it keeps crashing my machine.
Hah yes. Ill throw my own into the ring too https://github.com/boyter/cs although its a lot older than most other tools having been started 7 years ago.
Cursor was an interesting case where they were maintaining an index. I think everyone has converged on not doing that, which is where I started, with BM25 with some tweaks on top.
Notable link related to this https://www.youtube.com/watch?v=ZnmtJKFonLk where Jo Kristian Bergum (notable in the search space) agrees with BM25 being a good approach for agentic workflows.
> Cursor had something like this at some point. They've removed it.
No, they say that they have removed it, but they haven't. Cursor claims they retired local indexing in favor of something they call "instant grep," which, according to their own support, doesn't use a local index, and they removed the tool in Cursor settings to rebuild the index.
But Cursor still uses a local index. They have just hidden it as a subfolder in your ".git" folder. "~/.git/cursor".
Their hidden index often goes stale, and I had to nuke that folder every time I changed many files. This isn't new; when they officially had a local index, it was often stale too. I developed a script that ran on a daily schedule, every few hours, to alert me when Cursor's index went stale.
I constantly had to watch for Cursor suddenly not finding documents that clearly existed, and it often worked on outdated content. Disabling instant grep removes the hidden index, btw and problem gone.
I have a skill named read-opencode-db that's literally a 5 line md file as a simple documentation for a python script that allows agent to read other sessions
It's meant for me to reference session ID's directly to look at another agent review results for example
But I don't think that my workflow is well suited for allowing agents to freely grep the database for facts or decisions, I prefer Matt Pocock's domain modelling skill that bakes decisions into versioned and commitable markdown files
https://github.com/huggingface/funes/ has been my go-to here (specifically for conversation search), but I do appreciate that there's a frontend on this one that can recall a session and open the correct agent for it (rather than my opening one agent, then opening another when I've gotten it wrong). Looks nice. Well done.
The Borges reference is neat. I’m a little lost in the sloppy documentation (“funes makes four deliberate, load-bearing choices”). I want to trust this one a little more than the average vibe code because it’s from HF.
My scepticism is that agent memory must be a non trivial problem to solve, or the labs would ship a solution in the harness. The models are happy to check against previous conversations if prompted, but they rarely do it on their own by default.
I feel a lot of these tools are barking up the wrong tree. Solving problems that are just symptoms of a deeper root cause. Index because my codebase is too big to grep. Index to paraphrase hard_decline via card retries. Reduce tool calling to save my agent YOLO search loops. etc. etc.
It feels like trying to overcome bad codebase hygiene by throwing more tools at it, and wallpapering the mold over. I've found that codebases left entirely to LLMs suffer from this gravely, but with even a bit of effort - you'll get outsized returns. Don't let the agent smosh up everything into a giant ball of code. Separate concerns, build services, do the architecture step of the software design with a bit more care than vibe coding.
A bit of a rant, yeah, but I don't see why tools like this exist (and ask you to curl a script into bash, no less). You're doing stuff a bit wrong if grepping is the bottleneck.
I mostly feel the same, but it'd be really interesting to measure it, I think. Like, how many tokens did it cost in a codebase of quality Q1 vs one of quality Q2 to `grep` vs some semantic search. And did you find the correct `DuplicateSymbolName` in the correct namespace?
I don't think most people have control of the code base they're required to work in.
I too have converted a code base I made to typescript and peppered it with AI docs to make visibility easier.
I've converted i18n files to a k:v[] arrays per language because the AI is shit at working with objects. The ability now for it to just find/insert additional languages is well improved.
There definitely are shapes in code that the AI will have trouble with.
But again, I don't think most people have control over the code they maintain.
>I feel a lot of these tools are barking up the wrong tree.
Its the right tree, but not the right branch. RAG systems are the next big thing. You can take a small model running locally, feed it a bunch of context and it will be able to generate the correct answer on par with a larger model with no context.
Context doesn't have to be documents/code either, it can be LLM generated context that guides next iterations of inference on how to do things.
OK, so, I too have built a local index tool called slopdex [1].
It's very simple, parses the repo code and docs with treesitter, indexes it, vectorizes everything, then allows you do things like create a map of the codebase or do a semantic search.
One cool trick that does work well is the cross-search command, which allows you to do a duplicate code search using the vector index. This gets rid of a blind spot that lots of LLMs have that make it create lots of dupliclate helper functions. This works quite well in practice.
It has a few evals that I'm still working on, showing only modest improvements in token usage when using the map command, maybe around ~10% less. I think this can depend a lot on the type of task though.
I am interested to know your thoughts about publishing with the repository a natural-language implementation-agnostic specification that would allow other coding agents to create and mutate their own parallel implementations.
That's an interesting idea, though I believe that this should work just as well by pointing an agent at the code directly. At least, I don't see why a natural language sepecification would be better for this purpose.
Curious about this implementation. How does the vector search work with a versioned code base?
If I'm on commit A, then save the work and now I'm on commit B, how does cross-search ensure it doesn't return results for code JUST in commit A (i.e. code that is no longer in the codebase).
The index state corresponds to a commit hash and changing it (moving to commit B) will use git to compare the changed files and remove deleted files/functions from the index (and parse new ones).
Vector embeddings are cached, but added/removed from the index to correspond to the commit.
Are you modeling the actual API costs? If you can cut round trips at the end of a long context window you can end up reducing usage even if the total token count is the same.
Nice tool. Does it support local embedding providers such as qdrant https://qdrant.tech/ ? Why only support paid, cloud-based embeddings provider ?
ChatGPT says it could work without updating the code, but since README doesn't advertise it, I was curious as to the reason behind this choice.
It says: "Can Slopdex run without paid embedding APIs?
Potentially yes, without modifying its code.
[...] Slopdex supports a custom embeddingBaseUrl, allowing it to call a locally hosted, OpenAI-compatible embedding server.
For example, with a local model producing 768-dimensional vectors:
{
"provider": "openai",
"model": "your-local-embedding-model",
"dimensions": 768,
"symbolDimensions": 768,
"embeddingBaseUrl": "http://localhost:1234/v1"
}
Its code still requires an OPENAI_API_KEY value, but a dummy key should work if your local server doesn't enforce authentication.
The server must implement /v1/embeddings and accept Slopdex's request format, including the dimensions parameter.
The relevant code is in vector_openai.rs."
LLM generated websites are so instantly recognisable
Vibe coded tools are great. I have loads of them to work for my workflows. AI makes it almost free to make your own tools. Almost none of them are worth sharing or distributing
I think there's a time-spent curve. If you put enough time into iterating and refining with attention to arch and design (weeks or months) then it might be worth sharing. If you've done it in days or a week, probably not.
At some point if these types of front pages that are promoting a tool like this will be optimized for the agent itself to make the decision to install and use it. The human just says "please optimize for cost" and then the agent finds this thing and starts using it without the human even knowing or caring.
"Search your files, repos and agent sessions, locally, and give your agents the best context for the job. One search points them to the few passages that matter, instead of grepping the whole disk."
Grep is already searching for relevant passages. This is clearly LLM written copy. It makes literally no sense.
This reminds me of the dating app advert I keep seeing on tiktok, which says that other dating sites "force you to be the loudest voice on your profile". There's no interpretation that makes sense, it's just the typical LLM gibberish.
Indexing local files makes search faster, but not more flexible or requiring less tool calls. Grep can take general regex, which more or less can't be indexed, so there's very few things that are more flexible or produce less tool calls.
Although I agree with all your points, I think there is a potential caveat with grep: it is not very token-efficient as it can return lots of matches, filling up the agent's context.
A potential solution for these indexing systems is to maintain several indexes (one per branch for example). Idk how they handle code changes though.
But I think the bigger negative signal is that these tools have been around for roughly as long as LLMs have, yet they are still barely used. Maybe because they are not that useful ?
tl,dr: Semantic search based on "Qwen3-Embedding-0.6B and stored in SQLite (with a trigram full-text index for exact matches) plus turbovec vector indexes".
65 comments
[ 0.23 ms ] story [ 188 ms ] threadAn indexer crawls the codebase and maps slices of code to concepts, then groups related concepts. You can inspect a concept to see things related to it, or ask about the impact of a change and see second-order changes required - things not so easy to see that are related by basic symbol linking (although it does use symbol linking - it heavily relies on Roslyn, since it's all C# focused).
It also returns confidence scores with everything, which adjust based on a dozen or so factors like whether the user defined the concept or the concept was auto-generated during the code crawl. Low confidence score === fall back to grep.
It was a super fun project and ended up being very useful, as much for people as the LLM. But, I'm not convinced it is materially better than grep in some ways and just haven't had the time to really focus on improving it.
What are the various ways Pensieve is superior to iterative grep? Cache invalidation is hard - how does Pensieve address it?
Could you take a few minutes to explain the what and the how? Links to writeup to blog posts or files in your repo would be very helpful.
Finally, can I use Pensieve with Paseo? What if I wanted to expose Pensieve's capability to my Paseo setup?
Pensieve isn't ready for prime time yet.
Referencing other sessions is a cool idea tho
Any insight into why would be much appreciated because those insights might hold the key to whether this is a usable, sustainable idea to begin with.
Cursor was an interesting case where they were maintaining an index. I think everyone has converged on not doing that, which is where I started, with BM25 with some tweaks on top.
Notable link related to this https://www.youtube.com/watch?v=ZnmtJKFonLk where Jo Kristian Bergum (notable in the search space) agrees with BM25 being a good approach for agentic workflows.
No, they say that they have removed it, but they haven't. Cursor claims they retired local indexing in favor of something they call "instant grep," which, according to their own support, doesn't use a local index, and they removed the tool in Cursor settings to rebuild the index.
But Cursor still uses a local index. They have just hidden it as a subfolder in your ".git" folder. "~/.git/cursor".
Their hidden index often goes stale, and I had to nuke that folder every time I changed many files. This isn't new; when they officially had a local index, it was often stale too. I developed a script that ran on a daily schedule, every few hours, to alert me when Cursor's index went stale.
I constantly had to watch for Cursor suddenly not finding documents that clearly existed, and it often worked on outdated content. Disabling instant grep removes the hidden index, btw and problem gone.
Something something if you're not already doing this you're behind.
It's meant for me to reference session ID's directly to look at another agent review results for example
But I don't think that my workflow is well suited for allowing agents to freely grep the database for facts or decisions, I prefer Matt Pocock's domain modelling skill that bakes decisions into versioned and commitable markdown files
My scepticism is that agent memory must be a non trivial problem to solve, or the labs would ship a solution in the harness. The models are happy to check against previous conversations if prompted, but they rarely do it on their own by default.
If not for the rest of the page I would have assumed this was a joke.
It feels like trying to overcome bad codebase hygiene by throwing more tools at it, and wallpapering the mold over. I've found that codebases left entirely to LLMs suffer from this gravely, but with even a bit of effort - you'll get outsized returns. Don't let the agent smosh up everything into a giant ball of code. Separate concerns, build services, do the architecture step of the software design with a bit more care than vibe coding.
A bit of a rant, yeah, but I don't see why tools like this exist (and ask you to curl a script into bash, no less). You're doing stuff a bit wrong if grepping is the bottleneck.
I too have converted a code base I made to typescript and peppered it with AI docs to make visibility easier.
I've converted i18n files to a k:v[] arrays per language because the AI is shit at working with objects. The ability now for it to just find/insert additional languages is well improved.
There definitely are shapes in code that the AI will have trouble with.
But again, I don't think most people have control over the code they maintain.
A library card catalog system is an index. It exists to make it easy to find the right book quickly.
Your grocery store aisles are indexes to make it easier for you to find baking supplies from laundry and cleaning.
Well-organized codebases are already indexes by project, by folder structure, by naming convention, etc.
We use indexes everywhere, big or small.
The problem is really that the harnesses/SDKs/APIs themselves appear as if they weren't designed with indexes in mind from the outset.
Its the right tree, but not the right branch. RAG systems are the next big thing. You can take a small model running locally, feed it a bunch of context and it will be able to generate the correct answer on par with a larger model with no context.
Context doesn't have to be documents/code either, it can be LLM generated context that guides next iterations of inference on how to do things.
It's very simple, parses the repo code and docs with treesitter, indexes it, vectorizes everything, then allows you do things like create a map of the codebase or do a semantic search.
One cool trick that does work well is the cross-search command, which allows you to do a duplicate code search using the vector index. This gets rid of a blind spot that lots of LLMs have that make it create lots of dupliclate helper functions. This works quite well in practice.
It has a few evals that I'm still working on, showing only modest improvements in token usage when using the map command, maybe around ~10% less. I think this can depend a lot on the type of task though.
Would be happy to hear any feedback.
[1]: https://github.com/ninjaxtools/slopdex
If I'm on commit A, then save the work and now I'm on commit B, how does cross-search ensure it doesn't return results for code JUST in commit A (i.e. code that is no longer in the codebase).
Vector embeddings are cached, but added/removed from the index to correspond to the commit.
ChatGPT says it could work without updating the code, but since README doesn't advertise it, I was curious as to the reason behind this choice.
It says: "Can Slopdex run without paid embedding APIs?
Potentially yes, without modifying its code. [...] Slopdex supports a custom embeddingBaseUrl, allowing it to call a locally hosted, OpenAI-compatible embedding server. For example, with a local model producing 768-dimensional vectors: { "provider": "openai", "model": "your-local-embedding-model", "dimensions": 768, "symbolDimensions": 768, "embeddingBaseUrl": "http://localhost:1234/v1" }
Its code still requires an OPENAI_API_KEY value, but a dummy key should work if your local server doesn't enforce authentication.
The server must implement /v1/embeddings and accept Slopdex's request format, including the dimensions parameter. The relevant code is in vector_openai.rs."
I've not even considered adding this because I expect my underpowered laptop won't do well with local models, but it makes sense to add it.
Vibe coded tools are great. I have loads of them to work for my workflows. AI makes it almost free to make your own tools. Almost none of them are worth sharing or distributing
Grep is already searching for relevant passages. This is clearly LLM written copy. It makes literally no sense.
This reminds me of the dating app advert I keep seeing on tiktok, which says that other dating sites "force you to be the loudest voice on your profile". There's no interpretation that makes sense, it's just the typical LLM gibberish.
Indexing local files makes search faster, but not more flexible or requiring less tool calls. Grep can take general regex, which more or less can't be indexed, so there's very few things that are more flexible or produce less tool calls.
A potential solution for these indexing systems is to maintain several indexes (one per branch for example). Idk how they handle code changes though.
But I think the bigger negative signal is that these tools have been around for roughly as long as LLMs have, yet they are still barely used. Maybe because they are not that useful ?
But someone will come in crying saying “Nooo!! It does so much more like indexing and bla bla etc”