HN Explorer

Should I Use a Framework to Build an Agent? Code vs. LangGraph vs. Workflows

21 points by jgilhuly ↗ HN
LangGraph and LlamaIndex Workflows are generating a lot of buzz right now, and we wanted to see how they measure up in practice versus just writing the code. To do that, we took a straightforward agent architecture—one we've built and deployed in code without a framework—and implemented it using LangGraph and Workflows. Our main goal was to explore how these frameworks translate a simple agent design into their abstractions and assess the impact on the development and debugging process.

We want to share our findings with the community, providing practical examples and honest observations about these frameworks where they introduce friction and where they shine. There’s a lot of hype out there, and we hope to offer some clarity with real code examples and unbiased perspectives.

For context, we’ve been running our own Co-pilot agent/assistant in production for about eight months. We’ve also helped clients troubleshoot their assistants at scale, so we’ve seen a wide range of use cases and challenges.

The architecture we tested is a single-tier LLM router—a pattern we often see in various client implementations. It involves a single LLM router that uses function calling to route tasks or skills, which might include another LLM call before returning control to the router. It’s a simple but versatile pattern.

Here’s a Towards Data Science write up we did on the project: https://towardsdatascience.com/choosing-between-llm-agent-frameworks-69019493b259

Full code: https://github.com/Arize-ai/phoenix/tree/main/examples/agent_framework_comparison

Hot take #1: For experienced developers, framework abstractions can add unnecessary complexity. Hot take #2: Built-in parallelism, while promising, can complicate debugging a lot. Hot Take #3: In environments with less experienced development teams that have no scaffolding, these frameworks could offer some useful structure. At least in the POC phase.

We’re repeating this process now with CrewAI and Autogen - learnings to follow soon.

And if you want to deep dive into the logs of any of these, we’ve published the traces captured with Arize Phoenix here. Pure code: https://phoenix-demo.arize.com/projects/UHJvamVjdDo2 LangGraph: https://phoenix-demo.arize.com/projects/UHJvamVjdDoy LlamaIndex Workflows: https://phoenix-demo.arize.com/projects/UHJvamVjdDo1

We’re curious to hear what others think. What’s been your experience with these frameworks, and how do they compare to rolling your own agent solutions?

6 comments

[ 1.5 ms ] story [ 29.9 ms ] thread
nfcampos
Hi, you don't have to use langchain tools or ToolNode with langgraph, you can absolutely write your custom tool handling logic. In langgraph nodes are just functions that can do whatever you want, seems to be some misconception somewhere?
jgilhuly
That's true, you could not use the ToolNode object and define your own logic to handle tools. You do need to use at least the @tool decorator in order to attach those functions to a model using LangGraph's bind_tools function.

Defining some of the functions as tools was where some of the "self" parameter issues mentioned in the TDS write up came up - since those parameters aren't added by the LLM, but LangGraph/Pydantic errors if there missing.

Technically you could forgo using the ToolNode object, not define your functions as tools, not use the bind_tools method, and potentially not use Langchain's model abstractions - but at that point, you're not using much of the framework at all.

Let me know if I'm missing something there!

wfhpw
Almost none of those things are part of the LangGraph framework? LangGraph does the scheduling, checkpointing, state management, etc.

All of those things are LangChain abstractions that you can use when building a graph.

jackmpcollins
I'm building magentic, which aims to be a middle ground between the current "no framework" and "too much framework" options. It handles retries, logging/tracing, concurrency with asyncio, and other boilerplate, without mandating any prompts or particular patterns. Please check it out for the next comparison!

https://github.com/jackmpcollins/magentic

jgilhuly
We'll check it out! Definitely interested in finding the right level of abstraction in these frameworks
qwertyquirky
would be interesting to see crew and autogen takes. i feel like the consistency of the higher abstracted frameworks would be interesting to see