One Wikipedia page costs your AI agent 68,000 tokens

4 points by arhamislam5766 ↗ HN
i use claude code daily and measured what pages cost it while doing research. an average wikipedia article, for instance, is 68,240 tokens of raw html (tiktoken); nike's homepage is 353,000.

claude code's built-in webfetch handles the easy case well. it summarizes wikipedia to about 950 tokens and clears cloudflare on some sites like indeed and ticketmaster. but, and there's always a but, on js-rendered and some anti-bot pages it returns nothing.

quotes.toscrape.com/js gives "no quotes found"; nike.com gives a 403. your agent then dumps the raw html back into context and still fails. (note: i have also had cases where i read through the chat at the end and saw that it failed and just pulled from either training data or stale caches from other sources)

so i worked on building an open-source stealth browser (recompiled chromium) that runs as an mcp for claude code, cursor, and claude desktop. essentially all i have to change form my end is add the mcp, and it returns the cleaned up tokens while also beating detection: the js quotes come back in 285 tokens, nike in about 700 instead of a 403.

there is still stuff i am actively working on: there's no residential egress yet, and it won't beat kasada-style walls. it's for agents, qa, and research.

repo and the reproducible benchmark: https://github.com/tiliondev/fortress/tree/main/mcp i'm the author and here for feedback.

14 comments

[ 2.8 ms ] story [ 39.4 ms ] thread
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations
> on js-rendered and some anti-bot pages it returns nothing

yeah, and my (human piloted) browser gets blocked by so many websites now. I routinely get "Sorry." when trying to log into hn.

sigh.

Hacker News is the one page that hasn't assumed I'm a bot, yet.

At this rate, we'll need to use bots to browse the web for us, because they're the only way to get through the anti-bot filters.

stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.
+1. i think the the additional value prop is the bypassing blockers. ideally op should just use jina on top of whatever they are building.
And how much water?
1/1000th to 1/10000th as much as typical meat consumption.
Token reduction is useful, but knowing whether the agent actually accessed the real page is even more important.
gonna try this out this week, one ask: when a wall does beat it, return "blocked, here's why" instead of empty, avoiding silent fetch failures.