I've been building agent type stuff for a couple years now and the best thing I did was build my own framework and abstractions that I know like the back of my hand.
I'd stay clear of any llm abstraction. There are so many companies with open source abstractions offering the panacea of a single interface that are crumbling under their own weight due to the sheer futility of supporting every permutation of every SDK evolution, all while the same companies try to build revenue generating businesses on top of them.
1. If your agent needs to write a lot of code, it's really hard to beat Claude Code (cc) / Agent SDK. We've tried many approaches and frameworks over the past 2 years (e.g. PydanticAI), but using cc is the first that has felt magic.
2. Vendor lock-in is a risk, but the bigger risk is having an agent that is less capable then what a user gets out of chatgpt because you're hand rolling every aspect of your agent.
3. cc is incredibly self aware. When you ask cc how to do something in cc, it instantly nails it. If you ask cc how to do something in framework xyz, it will take much more effort.
4. Give your agent a computer to use. We use e2b.dev, but Modal is great too. When the agent has a computer, it makes many complex features feel simple.
0 - For context, Definite (https://www.definite.app/) is a data platform with agents to operate it. It's like Heroku for data with a staff of AI data engineers and analysts.
I liked reading this but got a silly question as I am a noob in these things. If explicit caching is better, does that mean the agent is just forgetting stuff unless we manually save its notes? Are these things really that forgetful? Also why is there a virtual file system? So the agent is basically just running around a tiny digital desktop looking for its files? Why can't the agent just know where the data is? I'm sorry if these are juvenile questions.
The 'Reinforcement in the Agent Loop' section is a big deal, I use this pattern to enable async/event steered agents, it's super powerful. In long context you can use it to re-inject key points ("reminders"), etc.
I still feel there is no sure shot way to build an abstraction yet. Probably that is why Loveable decided to build on Gemini AI rather than giving options of choosing model. On the other hand I like Pydantic AI framework and got myself a decent working solution where my preference is to stick with cheaper models by default and only use expensive only in cases where failure rate is too high.
I've started a company in this space about 2 years ago. We are doing fine. What we've learned so far is that a lot of these techniques are simply optimisations to tackle some deficiency in LLMs that is a problem "today". These are not going to be problems tomorrow because the technology will shift. As it happened many time in the span of the last 2 years.
So yah, cool, caching all of that... but give it a couple of months and a better technique will come out - or more capable models.
Many years ago when disc encryption on AWS was not an option, my team and I had to spend 3 months to come up with a way to encrypt the discs and do so well because at the time there was no standard way. It was very difficult as that required pushing encrypted images (as far as I remember). Soon after we started, AWS introduced standard disc encryption that you can turn on by clicking a button. We wasted 3 months for nothing. We should have waited!
What I've learned from this is that often times it is better to do absolutely nothing.
This is the "Wait Calculation" and it's fiendish because there exists only some small, finite window in which it is indeed better to start before the tech is "better" in order to "win" (i.e. get "there" first, wherever "there" is in your scenario).
Going to hijack this to ask a question that’s been on my mind: does anybody know why there’s no agentic tools that use tree-sitter to navigate code? It seems like it would be much more powerful than having the LLM grepping for strings or rewriting entire files to change one line.
My tip is: don’t use SDK:s for agents. Use a while loop and craft your own JSON, handle context size and handle faults yourself. You will in practice need this level of control if you are not doing something trivial.
Amen. Been seeing these agent SDKs coming out left and right for a couple of years and thought it'd be a breeze to build an agent. Now I'm trying to build one for ~3 weeks, and I've tried three different SDKs and a couple of architectures.
Here's what I found:
- Claude Code SDK (now called Agent SDK) is amazing, but I think they are still in the process of decoupling it from the Claude Code, and that's why a few things are weird. e.g, You can define a subagent programmatically, but not skills. Skills have to be placed in the filesystem and then referenced in the plugin. Also, only Anthoripic models are supported :(
- OpenAI's SDK's tight coupling with their platform is a plus point. i.e, you get agents and tool-use traces by default in your dashboard. Which you can later use for evaluation, distillation, or fine-tuning.
But:
2. They have agent handoffs (which works in some cases), but not subagents. You can use tools as subagents, though.
1. Not easy to use a third-party model provider. Their docs provide sample codes, but it's not as easy as that.
- Google Agent Kit doesn't provide any Typescript SDK yet. So didn't try.
- Mastra, even though it looks pretty sweet, spins up a server for your agent, which you can then use via REST API. umm.. why?
- SmythOS SDK is the one I'm currently testing because it provides flexibility in terms of choosing the model provider and defining your own architecture (handoffs or subagents, etc.). It has its quirks, but I think it'll work for now.
Question: If you don't mind sharing, what is your current architecture? Agent -> SubAgents -> SubSubAgents? Linear? or a Planner-Executor?
I'll write a detailed post about my learnings from architectures (fingers crossed) soon.
This is why I use the agent I use. I won't name the company, because I don't want people to think I'm a shill for them (I've already been accused of it before, but I'm just a happy, excited customer). But it's an agentic coding company that isn't associated with any of the big model providers.
I don't want to keep up with all the new model releases. I don't want to read every model card. I don't want to feel pressured to update immediately (if it's better). I don't want to run evals. I don't want to think about when different models are better for different scenarios. I don't want to build obvious/common subagents. I don't want to manage N > 1 billing entities.
I just want to work.
Paying an agentic coding company to do this makes perfect sense for me.
Design is hard because models change almost on a weekly basis. Quality abruptly falls off or drastically changes without announcements. It’s like building a house without proper foundation. I don’t know how many tokens I wasted because of this. I want to say 30% of the cost and resources this year for me.
Great post. Re: frameworks, I tried a number of them and then found Pocketflow and haven’t found a reason to try anything else since. It’s now been ported to half a dozen or more languages (including my port to Ruby). The simple api and mental model makes it easy for everyone on my team to jump into, extend, and compose. Highly recommend for anyone frustrated with the larger SDKs.
We're repeating the same overengineering cycle we saw with early LangChain/RAG stacks. Just a couple of months ago the term agent was hard to define, but I've realized the best mental model is just a standard REPL:
Read: Gather context (user input + tool outputs).
Eval: LLM inference (decides: do I need a tool, or am I done?).
Print: Execute the tool (the side effect) or return the answer.
Loop: Feed the result back into the context window.
Rolling a lightweight implementation around this concept has been significantly more robust for me than fighting with the abstractions in the heavy-weight SDKs.
> We find testing and evals to be the hardest problem here. This is not entirely surprising, but the agentic nature makes it even harder. Unlike prompts, you cannot just do the evals in some external system because there’s too much you need to feed into it. This means you want to do evals based on observability data or instrumenting your actual test runs. So far none of the solutions we have tried have convinced us that they found the right approach here.
I'm curious about the solutions the op has tried so far here.
What are the main shortcomings of the solutions you tried out?
We believe you need to both automatically create the evaluation policies from OTEL data (data-first) and to bring in rigorous LLM judge automation from the other end (intent-first) for the truly open-ended aspects.
"Because there’s too much you need to feed into it" - what does the author mean by this? If it is the amount of data, then I would say sampling needs to be implemented. If that's the extent of the information required from the agent builder, I agree that an LLM-as-a-judge e2e eval setup is necessary.
In general, a more generic eval setup is needed, with minimal requirements from AI engineers, if we want to move forward from Vibe's reliability engineering practices as a sector.
If you are building an agent, start from scratch and build your own framework. This will save you more headache and wasted time down the line. one of the issues when you use someone else framework is that you miss out on learning and understanding important fundamentals about LLM's, how they work, context, etc... Also many developers don't learn the fundamentals of running LLM's locally and thus miss crucial context (giggidy) that would have helped them better understand the whole system. It seems to me that the author here came to a similar conclusion like many of us. I do want to add my own insights though that might be of use to some.
One of the things he talked about was issues with reliable tool calling by the model. I recommend he try the following approach. Have the agent perform a self calibration exercise that makes the agent use his tools at the beginning of the context. Make him perform some complex stuff. Do it many times to test for coherence and accuracy while adjusting the system prompt towards more accurate tool calls. Once the agent had performed that calibration process successfully, you "freeze" that calibration context/history by broadening the --keep n to include not just the system prompt in the rolling window but also up to the end of this calibration session. then no matter how far the context window drifts the conditioned tokens generated by that calibration session steer the agent towards proper tool use. From then on your "system prompt" includes those turns. Note that this is probably not possible on cloud based models as you don't have access to the inference engine directly. A hacky way around that is emulate the conversation turns inside the system prompt.
On the note of benchmark's. The calibration test is your benchmark from then on. When introducing new tools to the system prompt or adjusting any important variable, you must always rerun the same test to make sure the new adjustments you made don't negatively affect the system stability.
On context engineering. That is a must as a bloated history will decohere the whole system. So its important to device an automated system that compresses the context down but retains overall essence of the history. there are about a billion ways you could do this and you will need to experiment a lot. LLM's are conditioned quite heavily from their own outputs, so having the ability to remove error tool calls from the context is a big boon as now the model is less likely to repeat its same mistakes. There are trade offs though, like he said caching is a no go when going this route but you gain a lot more control and stability within the whole system if you do this right. its basically reliability vs cost here. I tend to lean towards reliability. Also i don't recommend using the whole context size of the model. Most llms perform very poorly past a specific amount and I find that using maximum of 50% of the whole context window is recommended for cohesion. Meaning that if lets say max context window is 100k tokens, treat 50k as the max limit and start compressing the history around 35k tokens. Granular and step wise system can be set up. Where the most recent context is most detailed and uncompressed but as it goes further from the current time it gets less and less detailed. Obviously you want to store the full uncompressed history for a subagent that uses rag. This allows the agent to see in detail the whole thing if it finds the need to.
ahh also on the matter of output. I found great success with making input and output channels for my agent. there are many channels that the model is conditioned in using for specific interactions. <think> channel for cot and reasoning. <message_to_user> channel for explicit messages to user. <call_agent> channel for calling agents and interacting with them. <call_tool> for tool use. and then a few other environment and system channels that are input channels from error scripts and environment towards the agent. This ch...
I use CLI Terminals as agent frameworks. You have big tech and open source behind them. All the problems are solved with zero work. They take care of new problems. You don't need to remove all the stuff that becomes outdated because the latest model doesn't make the same mistakes. Free via free tiers, cheap via openai compatible open source models like z.ai. Insanely smart and can easily talk to MCP servers.
I had built an agent with LangGraph a 9 months ago--now seems React agents are in LangChain. Over all pretty happy with that, I just don't use any of the dumb stuff like embedding/search layer: just tools&state
But I was actually playing with a few frameworks yesterday and struggling--I want what I want without having to write it. ;). Ended up using pydantic_ai package, literally just want tools w/ pydantic validation--but out of the box it doesn't have good observability, you would have to use their proprietary SaaS; and it comes bundled with Temporal.io (yo odio eso proyecto). I had to write my own observability which was annoying, and it sucks.
If anyone has any things they've built, I would love to know, and TypeScript is an option. I want:
- ReAct agent with tools that have schema validation
- built in REALTIME observability w/ WebUI
- customizable playground ChatUI (This is where TypeScript would shine)
- no corporate takeover tentacles
p.s.s: I know... I purposely try to avoid hard recommendations on HN, to avoid enshittification. "reddit best X" has been gamed. And generally skip these subtle promotional posts..
My experience with BAML was really good. For structured outputs, JSON schema is slow and sluggish while BAML performs well. Interestingly don't see much attention on HN, wondering is it hype of other SDKs or BAML doesn't deliver enough.
I think output functions aren't necessary, you should just use the text output when the agentic loop ends, and prompt to what kind of output you want (markdown, summary of changes, etc.)
41 comments
[ 3.6 ms ] story [ 59.7 ms ] threadI'd stay clear of any llm abstraction. There are so many companies with open source abstractions offering the panacea of a single interface that are crumbling under their own weight due to the sheer futility of supporting every permutation of every SDK evolution, all while the same companies try to build revenue generating businesses on top of them.
1. If your agent needs to write a lot of code, it's really hard to beat Claude Code (cc) / Agent SDK. We've tried many approaches and frameworks over the past 2 years (e.g. PydanticAI), but using cc is the first that has felt magic.
2. Vendor lock-in is a risk, but the bigger risk is having an agent that is less capable then what a user gets out of chatgpt because you're hand rolling every aspect of your agent.
3. cc is incredibly self aware. When you ask cc how to do something in cc, it instantly nails it. If you ask cc how to do something in framework xyz, it will take much more effort.
4. Give your agent a computer to use. We use e2b.dev, but Modal is great too. When the agent has a computer, it makes many complex features feel simple.
0 - For context, Definite (https://www.definite.app/) is a data platform with agents to operate it. It's like Heroku for data with a staff of AI data engineers and analysts.
So yah, cool, caching all of that... but give it a couple of months and a better technique will come out - or more capable models.
Many years ago when disc encryption on AWS was not an option, my team and I had to spend 3 months to come up with a way to encrypt the discs and do so well because at the time there was no standard way. It was very difficult as that required pushing encrypted images (as far as I remember). Soon after we started, AWS introduced standard disc encryption that you can turn on by clicking a button. We wasted 3 months for nothing. We should have waited!
What I've learned from this is that often times it is better to do absolutely nothing.
Here's a nice article about it: https://www.oneusefulthing.org/p/the-lazy-tyranny-of-the-wai...
Any framework you build around the model is just behaviour that can be trained into the model itself
Here's what I found:
- Claude Code SDK (now called Agent SDK) is amazing, but I think they are still in the process of decoupling it from the Claude Code, and that's why a few things are weird. e.g, You can define a subagent programmatically, but not skills. Skills have to be placed in the filesystem and then referenced in the plugin. Also, only Anthoripic models are supported :(
- OpenAI's SDK's tight coupling with their platform is a plus point. i.e, you get agents and tool-use traces by default in your dashboard. Which you can later use for evaluation, distillation, or fine-tuning. But: 2. They have agent handoffs (which works in some cases), but not subagents. You can use tools as subagents, though. 1. Not easy to use a third-party model provider. Their docs provide sample codes, but it's not as easy as that.
- Google Agent Kit doesn't provide any Typescript SDK yet. So didn't try.
- Mastra, even though it looks pretty sweet, spins up a server for your agent, which you can then use via REST API. umm.. why?
- SmythOS SDK is the one I'm currently testing because it provides flexibility in terms of choosing the model provider and defining your own architecture (handoffs or subagents, etc.). It has its quirks, but I think it'll work for now.
Question: If you don't mind sharing, what is your current architecture? Agent -> SubAgents -> SubSubAgents? Linear? or a Planner-Executor?
I'll write a detailed post about my learnings from architectures (fingers crossed) soon.
I don't want to keep up with all the new model releases. I don't want to read every model card. I don't want to feel pressured to update immediately (if it's better). I don't want to run evals. I don't want to think about when different models are better for different scenarios. I don't want to build obvious/common subagents. I don't want to manage N > 1 billing entities.
I just want to work.
Paying an agentic coding company to do this makes perfect sense for me.
Read: Gather context (user input + tool outputs). Eval: LLM inference (decides: do I need a tool, or am I done?). Print: Execute the tool (the side effect) or return the answer. Loop: Feed the result back into the context window.
Rolling a lightweight implementation around this concept has been significantly more robust for me than fighting with the abstractions in the heavy-weight SDKs.
I'm curious about the solutions the op has tried so far here.
We believe you need to both automatically create the evaluation policies from OTEL data (data-first) and to bring in rigorous LLM judge automation from the other end (intent-first) for the truly open-ended aspects.
In general, a more generic eval setup is needed, with minimal requirements from AI engineers, if we want to move forward from Vibe's reliability engineering practices as a sector.
I wonder what this means for the agents that people are deploying into production? Are they tested at all? Or just manual ad-hoc testing?
Sounds risky!
One of the things he talked about was issues with reliable tool calling by the model. I recommend he try the following approach. Have the agent perform a self calibration exercise that makes the agent use his tools at the beginning of the context. Make him perform some complex stuff. Do it many times to test for coherence and accuracy while adjusting the system prompt towards more accurate tool calls. Once the agent had performed that calibration process successfully, you "freeze" that calibration context/history by broadening the --keep n to include not just the system prompt in the rolling window but also up to the end of this calibration session. then no matter how far the context window drifts the conditioned tokens generated by that calibration session steer the agent towards proper tool use. From then on your "system prompt" includes those turns. Note that this is probably not possible on cloud based models as you don't have access to the inference engine directly. A hacky way around that is emulate the conversation turns inside the system prompt.
On the note of benchmark's. The calibration test is your benchmark from then on. When introducing new tools to the system prompt or adjusting any important variable, you must always rerun the same test to make sure the new adjustments you made don't negatively affect the system stability.
On context engineering. That is a must as a bloated history will decohere the whole system. So its important to device an automated system that compresses the context down but retains overall essence of the history. there are about a billion ways you could do this and you will need to experiment a lot. LLM's are conditioned quite heavily from their own outputs, so having the ability to remove error tool calls from the context is a big boon as now the model is less likely to repeat its same mistakes. There are trade offs though, like he said caching is a no go when going this route but you gain a lot more control and stability within the whole system if you do this right. its basically reliability vs cost here. I tend to lean towards reliability. Also i don't recommend using the whole context size of the model. Most llms perform very poorly past a specific amount and I find that using maximum of 50% of the whole context window is recommended for cohesion. Meaning that if lets say max context window is 100k tokens, treat 50k as the max limit and start compressing the history around 35k tokens. Granular and step wise system can be set up. Where the most recent context is most detailed and uncompressed but as it goes further from the current time it gets less and less detailed. Obviously you want to store the full uncompressed history for a subagent that uses rag. This allows the agent to see in detail the whole thing if it finds the need to.
ahh also on the matter of output. I found great success with making input and output channels for my agent. there are many channels that the model is conditioned in using for specific interactions. <think> channel for cot and reasoning. <message_to_user> channel for explicit messages to user. <call_agent> channel for calling agents and interacting with them. <call_tool> for tool use. and then a few other environment and system channels that are input channels from error scripts and environment towards the agent. This ch...
But I was actually playing with a few frameworks yesterday and struggling--I want what I want without having to write it. ;). Ended up using pydantic_ai package, literally just want tools w/ pydantic validation--but out of the box it doesn't have good observability, you would have to use their proprietary SaaS; and it comes bundled with Temporal.io (yo odio eso proyecto). I had to write my own observability which was annoying, and it sucks.
If anyone has any things they've built, I would love to know, and TypeScript is an option. I want: - ReAct agent with tools that have schema validation - built in REALTIME observability w/ WebUI - customizable playground ChatUI (This is where TypeScript would shine) - no corporate takeover tentacles
p.s.s: I know... I purposely try to avoid hard recommendations on HN, to avoid enshittification. "reddit best X" has been gamed. And generally skip these subtle promotional posts..