Show HN: A MitM proxy to see what your LLM tools are sending (github.com)

218 points by jmuncor ↗ HN
I built this out of curiosity about what Claude Code was actually sending to the API. Turns out, watching your tokens tick up in real-time is oddly satisfying.

Sherlock sits between your LLM tools and the API, showing you every request with a live dashboard, and auto-saved copies of every prompt as markdown and json.

39 comments

[ 6.3 ms ] story [ 84.2 ms ] thread
(comment deleted)
Nice work! I'm sure the data gleaned here is illuminating for many users.

I'm surprised that there isn't a stronger demand for enterprise-wide tools like this. Yes, there are a few solutions, but when you contrast the new standard of "give everyone at the company agentic AI capabilities" with the prior paradigm of strong data governance (at least at larger orgs), it's a stark difference.

I think we're not far from the pendulum swinging back a bit. Not just because AI can't be used for everything, but because the governance on widespread AI use (without severely limiting what tools can actually do) is a difficult and ongoing problem.

This is great.

When I work with AI on large, tricky code bases I try to do a collaboration where it hands off things to me that may result in large number of tokens (excess tool calls, unprecise searches, verbose output, reading large files without a range specified, etc.).

This will help narrow down exactly which to still handle manually to best keep within token budgets.

Note: "yourusername" in install git clone instructions should be replaced.

So is it just a wrapper around MitM Proxy?
(comment deleted)
Nice work! Do i need to update Claude Code config after start this proxy service?
You don't need to mess with certificates - you can point CC at a HTTP endpoint and it'll happily play along.

If you build a DIY proxy you can also mess with the prompt on the wire. Cut out portions of the system prompt etc. Or redirect it to a different endpoint based on specific conditions etc.

This is fantastic. Claude doesn't make it easy to inspect what it's sending - which would actually be really useful for refining the project-specific prompts.
As someone who just set up mitmproxy to do something very similar, I wish this would've been a plugin/add-on instead of a standalone thing.

I know and trust mitmproxy. I'm warier and less likely to use a new, unknown tool that has such broad security/privacy implications. Especially these days with so many vibe-coded projects being released (no idea if that's the case here, but it's a concern I have nonetheless).

I understand this helps if we have our own LLM run time. What if we use external services like ChatGPT / Gemini (LLM Providers)? Shouldn't they provide this feature to all their clients out of the box?
Pretty slick. I've been wanting something like this that gets stored with a hash that is stored in the corresponding code change commit message. It'd be good for postmortems of unnoticed hallucinations, and might even be useful to "revive" the agent and see if it can help debug the problem it created.
lmao WTAF is this?

build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/build/lib/sherlock

This tool looks like it unconditionally disables tls verification for upstream requests.

It shells out to mitmproxy with "--set", "ssl_insecure=true"

This took all of 5 minutes to find reading through main.py on my phone.

https://github.com/jmuncor/sherlock/blob/fb76605fabbda351828...

Edit: In case it’s not clear, you should not use this.

The thing you want has a kind of academic jargon name (coeffects algebra with graded/indexed monads for discharge) but is very intuitive, and it can do useful and complete attestation without compromising anyone credentials (in the limit case because everyone chooses what proxy to run).

https://imgur.com/a/Ztyw5x5

Could you use an approach like this much like a traditional network proxy, to block or sanitise some requests?

E.g. if a request contains confidential information (whatever you define that to be), then block it?

Or we could just demand agents that offer this level of introspection?
LiteLLM does this, and can do a lot more beyond that.
It’s actually really easy to use mitmproxy as a…proxy. You set it up as a SOCKS proxy (or whatever) and point your network or browser to the proxy. I did this recently when a python tool was too aggressive on crawling the web and the server would reject me. Forced my session to limit 5 requests per second and it worked rather than finding the exact file to change in the library. Just do the same to your browser and then turn on the capture mode and you’ll see the requests
Amusingly, I had the same question and asked Claude Code to vibe code me something similar. :)