HN Explorer

Ask HN: Has anyone measured how often agents use the skills you ship?

1 points by sohaibtariq ↗ HN
A lot of SaaS and API platforms now publish agent skills. We did too, but I cannot tell how useful they are for real users.

We have evals for testing that the skills work under ideal conditions, but I'm interested in runtime behaviour in the wild. For instance:

How often did an agent fall back to web search instead of following what the skill said? How often does an installed skill get triggered for an average user?

Is anyone tracking real use, beyond the install counts that platforms like skills.sh report? Does it even make sense to try and get these insights?

5 comments

[ 0.16 ms ] story [ 23.7 ms ] thread
verdverm
In my experience, I have not found any platform's provided skills worthy of inclusion. The number one reason is that they are often huge and cover the entire surface, or there are a bajillion of them. That these are presented as such shows they lack understanding about how to context engineer and have no respect for our token bills. Most seem Ai generated and come with the hallucinations and all that.

Have you vetted your skills for accuracy and conciseness?

Regarding harness level skill usage, that should be relatively easy for the user, as their harness should have saved the complete history of all sessions.

sohaibtariq
> Have you vetted your skills for accuracy and conciseness? Accuracy, yes. Conciseness could be better.

> Regarding harness-level skill usage, that should be relatively easy...

Users can analyse their own sessions, but I'm wondering if there's a feedback channel from the harness back to the skill author. Something that respects the users' privacy while still surfacing useful feedback.

verdverm
While I have take snippets or ideas from others skills, I don't think serious people are using other people's skills as is. Like everything in development, everyone has their own way of doing things and our agent instructions need to reflect that uniqueness.
flessio
Absolutely makes sense.

Alongside our MCP, skill, public API, .md files for all pages, llms.txt and https://fless.io/ai-agents info page we also wired up detailed analytics at every stage and for every request isolated by channel, both human and AI agent.

We track all agent calls at a pretty granular level to understand the agent funnel, by channel and provider (ex. Perplexity User, OpenAI User etc..).

We can't see what the agent did after calling us but we can see a funnel clearly which is useful to see where engagement and drop-off are.

Our first goal is for users to start a new apartment search - moving from 'landing page' - one of the surfaces mentioned above - to search created (ie. we're hunting for apartments on their behalf).

This has been helpful to understand agent behavior and where we need to improve skill / MCP etc..

sohaibtariq
Thanks for sharing! In your case, since the skill terminates in an MCP/API call, you have a closed feedback loop.

I was wondering about skills that don't have that feedback loop, where the output the agent produced doesn't touch an external server. For instance, the frontend-design skill from Anthropic.

How would someone get insights into the user's experience of using that kind of skill?