Launch HN: Laminar (YC S24) – We trained a model to catch silent agent failures
Here’s a one-minute walkthrough: https://youtu.be/KJa3h6ZJNLQ
We also put together a post with example traces and analyses from flow-1 and other models: https://laminar.sh/blog/flow-1-in-action
Agent observability is crowded, so instead of a bullet list of how Laminar differs from X, we’ll explain how we’re solving agent observability from first principles.
Agents now run for many steps and fail implicitly. An agent can click the wrong button or cite the wrong document while every call completes successfully. These errors are the most useful ones to catch and understand. To find them, you need to understand what the agent was trying to do and inspect its LLM calls and tool results. Simply looking at the final agent output is not enough.
Collecting traces alone doesn’t solve this. A lot of agent observability platforms are still data collectors at their core. You can end up paying to store traces that nobody has time to read or extract value from.
Using another LLM to inspect traces is THE solution, but capable models are expensive at scale and cheaper ones often miss errors or flag unrelated things.
So we built our model to solve exactly this problem. We finetuned a small OSS model and then post trained it with RL on thousands of synthetic investigations of realistic agent workflows in software, legal, customer support, etc. We got a model that is both good at agent trace analysis and very cheap to run.
flow-1 runs inside a harness we call Signals. It acts as a coding agent, treating an agent trace like a repo, with each span (an LLM call, tool call, etc.) as a file. It starts from a compressed string representation of a full agent trace and a user prompt, then greps across spans and reads the relevant ones in full.
The beauty of Signals is that it can respond to a general prompt. You can ask it to identify user frustration, check whether your browser agent completed a checkout, and so on. It’s not limited to raw error detection. It returns findings as JSON in a schema users can freely define. We call these signal events.
Once you’re analyzing every trace, you also have too many individual events to read. That’s why we automatically group similar signal events into clusters. You can see recurring problems and how often they happen, then drill into individual events and the traces behind them.
You can use traces from a cluster to build an eval dataset for that type of failure. You can then use your favourite coding agent to hillclimb on those evals and check that the fix works across these cases before shipping it to prod.
Trace → analyze → catch errors and form clusters → fix → release. Making this loop easy is why we built Laminar.
We’ve put the same thought into the rest of Laminar. Our blog (https://laminar.sh/blog) explains how looking closely at how agents actually work led us to build 20x trace compression, a custom SQL query engine and automatic task extraction.
Laminar is Apache 2.0 and self-hostable (https://github.com/lmnr-ai/lmnr). We make money from the managed cloud.
We’d love to hear how you track your agents’ performance today. How much do you lea...
33 comments
[ 50.7 ms ] story [ 157 ms ] threadI wonder if you have thought of using this for the voice AI space for creating traces from many test calls to speed up testing of voice bots? It's true there are a lot of challenges _before_ one gets to the stage of running actual calls like creating good test data etc, but I wonder if you see a definite role for this framework in that space?
Workflows are constantly changing but our design process happens across Figma, old-fashioned paper, and code.
> Agent observability is crowded, so instead of a bullet list of how Laminar differs from X, we’ll explain how we’re solving agent observability from first principles
my brain shut down and bounced minus the comment because it's a launch and maybe you'd want feedback.
I'd be using this for post-training in a custom harness: so what you focused on during post-training and what it excels at are the most interesting things
Also, a question for everyone, how do you reuse traces as benchmarks (eval datasets)? Is it about being able to synthesise an agent session from the trace, and having another agent watch the first agent's behavior? Having trouble wrapping my mind around all the little details.
Agent observability is based on similar primitives as regular observability (OTEL), but is wildly different. Trace payloads are huge, and errors are much more subtle, they are not increased latency or elevated 500 response rate. We recommend tracing everything in detail, and reading a few traces manually at the beginning while you iterate. At scale, the answer is almost always use another LLM look at your traces, which is what this post is all about.
Evals are critical. I really recommend adopting the Anthropic's capability / regression eval model [1]. We also deeply believe that agent evals should be end-to-end, and as high-level as possible. This again often involves a quality LLM-as-a-judge, tuned on a set of manually labelled data.
With this idea in mind (to answer your question), we think that, for eval datasets, the most crucial bit is to be able to store input of what your agent was asked about in production. Collect a certain distribution of inputs for happy and unhappy cases, and then run evals on those inputs. That's why Laminar went great lengths to help our users automatically get actual inputs their users send to their agents [2].
[1] https://www.anthropic.com/engineering/demystifying-evals-for...
[2] https://laminar.sh/blog/extracting-tasks-from-agent-runs
flow-1 actually predicts how critical each finding is
Can flow-1 flag that kind of thing? And how does it tell the difference between the tool giving back something wrong and the agent misreading a result that was actually fine? From the trace alone those look pretty similar to me.
So in this case it's not that it was a different failure, it was that a signal that seemed to indicate failure was actually a signal of success.
We’ve also heard from other companies that as the first line paper cut questions get answered, customer adoption increases - which further increases the “hard” support questions.
Lesson learned for me on that one that was rolling out a support agent led to both a better experience for our customers and an increased need for talented human engineers.
Does Laminar catch that case, where the trace looks fine but the outcome is wrong, or does it need a test result to compare against?
1. What exactly counts as an error in your benchmark? Is it binary error vs. no error, or do you pinpoint root causes like tool failures or policy violations? How do you get ground truth—human labels, heuristics, or replayed failures?
2. What metrics beyond one detection score do you use? Precision, recall, false positive rate, and model calibration all impact real-world debugging value. Any measure of whether this actually speeds up fixes would be helpful too.
3. The “23x cheaper” claim needs context. What are typical trace lengths, token counts, decoding settings, latency, and throughput? Were the models tested under comparable conditions?
4. What failure modes have you observed? Do false positives occur when the model misattributes causes? Does your approach generalize well beyond your dataset, or does it rely on brittle triggers?
5. Finally, can you share datasets, trace formats, and evaluation scripts? Without these, it’s hard for others to verify or build on your results.
Addressing these points would make this more than a proof of concept—turning it into a practical, reproducible foundation for agent observability rather than just another hype cycle.
Does flow-1 catch that class or mostly the errors that already show in the trace?