HN Explorer

Show HN: Selfship.ai – Surface and fix isues with your agentic applications 24x7 (selfship.ai)

4 points by pranny ↗ HN
We've been building an AI chat based trading system for last 3 years. The biggest issue was that when our Agent would mess up, we wouldn't know until a user reported. Agent traces helped us uncover what's wrong. But surfacing issues was almost always a manual trigger. So from those learnings, we built Selfship.ai. It's an autonomous system that observes every trace/turn/multi turn convo to find out issues. If a user got what they wanted, if a tool call is failing repeatedly, if users have to always reframe their questions, if the agent is taking optimal paths and many more. It's a loop - group failures by user intent, evaluate them, and ship fixes as PRs. After a fix is deployed, it evaluates if it worked or not. We recently opened it up as a SaaS. If you have an agentic product in production, we would love for you to try it out.

11 comments

[ 0.30 ms ] story [ 41.0 ms ] thread
s4h1lk
Very cool. Curious if the results are better for certain domains vs others and if there's a sweet spot in terms of number of turns in the convos. What have you seen deploying this in the wild so far?
pranny
So far, our early design partners have been running this on (a) AI Based Ads creation and optimisation product [Multi Modal] (b) Customer Support for eCommerce [Text Based] and (c) our own AI trading system.

We’re actively looking for more use cases because we want to understand where the approach breaks down.

We haven't seen a clear domain specific sweet spot yet. What surprises us is that it becomes seems to get better as the complexity / turn count increases.

It adopts very well to multi turn sessions, scheduled runs and even trigger based Agentic systems that don't involve human in the loop.

We don’t yet have enough data to say something like “10–20 turns is optimal,” though. That’s one of the things we’re hoping to learn as we get it into more production systems.

0xAron
Can I run this on-prem?
Jayesh_19
What happens when Selfship proposes the wrong fix?
pranny
Great question. It usually never happens. Our system is based on an adoption of Domino (systematic error/slice discovery), AgentBoard (trajectory/progress evaluation), τ-bench (goal/outcome correctness), MAST (failure taxonomies) and process mining (recurring session paths).

It surfaces issues at trace, session, and systemic levels by tracking user goals/constraints, progress and failure sequences, then grouping recurring high-impact failure modes rather than just clustering similar conversations.

The chances of a wrong fix is very low. However, even if it happens, the devs have the final say at "Merge the PR" stage. If it seems wrong, you can reply in the Github PR and selfship will improve upon it. If it's still going nowhere, the PR can be closed.

A bad fix never lands.

xsaurabhsuthar
How do you know the PR fixed the actual problem rather than just overfitting to the conversations that triggered it?
sanjitagion
How does selfship understand the goals of the users and what our agent's ideal path is going to be? How does it come to the conclusion of Agent doing a mistake versus a regular user drop off of some sorts?
arjunkrishnavas
How often do engineers actually merge the PRs it creates?
VarunGupta
How does it detect the failure mode of an agent, given that the agent is meant to continuously loop to work around any wall it hits?