If coding agents are the new entry point to your library, how sure are you that they’re using it well?
I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know.
Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation.
So we built a simple app to test how well coding agents interact with libraries:
• Takes your library’s docs
• Automatically extracts usage examples
• Tasks AI agents (like Claude Code) with generating those examples from scratch
• Logs mistakes and analyzes performance
We’re testing libraries now, but it’s early days. If you're interested: Input your library, see what breaks, spot patterns, and share the results below.
We plan to expand to more coding agents, more library-specific tasks, and new metrics. Let us know what we should prioritize next.
The skip-to-the-end answer: Context7 MCP is so good it seems like magic, even to many well-informed, highly capable hackers. Simply wildly good for libraries and SDKs. All it takes to start using it is to add the MCP provider to your agent config and save your arms, "Use Context7 for this".
I've done a lot of work recently to make my library more "LLM Friendly", but I'm not willing at this time to sign up to a service which I don't know I'd ever use again just to run a test on your behalf. If you want to run the test on my library then its GitHub can be found here: https://github.com/KaliedaRik/Scrawl-canvas
We’ve been working on this problem off and on for over a year now. Many models bake knowledge of particular tools/libraries/patterns into their weights very well and others quite poorly. In my experience Claude is quite good at integrating the dog.ceo API and noticeably ignorant when it comes to Postgres features, and it knows gcloud commands enough to very confidently and consistently hallucinate arguments.
We’ve baked a solution to this into our product, so if anybody is working on an API/SDK/etc feel free to contact me if your users are running into problems using LLMs to integrate them.
One thing we’ve noticed is that subtle changes to library/api integration prompts’ context can be surprisingly impactful. LLMs do very well with example commands and explicit instructions to consider X, Y, and Z. If you just dump an API reference and information that implicitly suggests that X, Y, and Z might be beneficial, they won’t reliably make the logical leaps you want them to unless you let them iterate or “think” (spend more tokens) more. But you can’t as easily provide an example for everything, and the ones you do will bias the models towards them, so you may need a bit of both.
Why did my engineering team handle payments through Stripe instead of building a custom payment processor? Aren’t they supposed to be engineering things?
It's a neat idea. But if, as we're told, LLMs will get better and better, something like this, in theory, will be increasingly unnecessary.
I feel like most of the problems with AI using a library is how we mix code and implementation. C, C++ got it right (even if by accident) with separating specification from implementation.
Instead of lamenting the design trend of not maintaining this split, for my own code I wrote a utility to extract specifications from my existing code.
> It's a neat idea. But if, as we're told, LLMs will get better and better, something like this, in theory, will be increasingly unnecessary.
I don't think so. I think understanding the context of a project will always produce superior results. I think instead we'll just make it a lot easier to add to the training corpus the LLM pulls from.
In Elixir land, the Ash Framework created a package called usage_rules[0] as an experimental attempt to solve this problem a few months ago. The latest version of the Phoenix Framework (1.8) includes it in their `mix phx.new` generator and in their own hex packages[1]. Library owners would need to add their own usage rules, but it seems to help even for just the core packages Phoenix includes.
Why would I care? I write libraries to help humans write code. If an LLM was actually good at writing code, it would engage with the library in the same way as a human. If it can't, that's not a problem with the library. I don't lose anything if an LLM doesn't use a library I've written.
What I often do, and completely works, is telling Claude Code to git-clone the library in question. Then I /add-dir it, so I can tell it to understand the code factually and as of the exact version I'm using.
You often don't even need cloning, for instance in the Ruby ecosystem, for the Pundit gem `bundle show pundit` tells you `/Users/vemv/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/pundit-2.3.2` which is a vanilla dir (non-zipped), ready to be added to the Claude context.
I can foresee it being useful when I write documentation, e.g. given an existing repo and my new guide, can the AI agent implement said feature? Does it fumble?
I only got one run it looks like for free and it was successful so I can't see what would happen if it failed. Curious what sort of feedback it provides on failure.
22 comments
[ 6.1 ms ] story [ 46.6 ms ] threadI asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know.
Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation.
So we built a simple app to test how well coding agents interact with libraries: • Takes your library’s docs • Automatically extracts usage examples • Tasks AI agents (like Claude Code) with generating those examples from scratch • Logs mistakes and analyzes performance
We’re testing libraries now, but it’s early days. If you're interested: Input your library, see what breaks, spot patterns, and share the results below.
We plan to expand to more coding agents, more library-specific tasks, and new metrics. Let us know what we should prioritize next.
https://context7.com/
We’ve baked a solution to this into our product, so if anybody is working on an API/SDK/etc feel free to contact me if your users are running into problems using LLMs to integrate them.
One thing we’ve noticed is that subtle changes to library/api integration prompts’ context can be surprisingly impactful. LLMs do very well with example commands and explicit instructions to consider X, Y, and Z. If you just dump an API reference and information that implicitly suggests that X, Y, and Z might be beneficial, they won’t reliably make the logical leaps you want them to unless you let them iterate or “think” (spend more tokens) more. But you can’t as easily provide an example for everything, and the ones you do will bias the models towards them, so you may need a bit of both.
Don't they know how to write their own code? Isn't that a coding agent's entire purpose in life?
There must be conflicting definitions out there. What does "coding agent" mean in this context?
or would reusability/modularity across projects and teams be beneficial?
I feel like most of the problems with AI using a library is how we mix code and implementation. C, C++ got it right (even if by accident) with separating specification from implementation.
Instead of lamenting the design trend of not maintaining this split, for my own code I wrote a utility to extract specifications from my existing code.
I don't think so. I think understanding the context of a project will always produce superior results. I think instead we'll just make it a lot easier to add to the training corpus the LLM pulls from.
[0] https://hexdocs.pm/usage_rules/readme.html
[1] https://github.com/phoenixframework/phoenix/tree/main/usage-...
Please write the Apis for people so we can go see when the LLM fails.
If you write it for the llm, the moment is starts lying the whole thing is broken, if its not for me I dont go digging in, I rather use something else
Dev Keys One or more of your connections are currently using Auth0 development keys and should not be used in production. Learn More
You often don't even need cloning, for instance in the Ruby ecosystem, for the Pundit gem `bundle show pundit` tells you `/Users/vemv/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/pundit-2.3.2` which is a vanilla dir (non-zipped), ready to be added to the Claude context.
I can foresee it being useful when I write documentation, e.g. given an existing repo and my new guide, can the AI agent implement said feature? Does it fumble?
I only got one run it looks like for free and it was successful so I can't see what would happen if it failed. Curious what sort of feedback it provides on failure.