HN Explorer

Show HN: 500 TB internet index in ClickHouse, with congestion pricing (scry.io)

1 points by Xyra ↗ HN
Hello. It's 2026, we're training simulated fruit fly brains to play Beat Saber, do we still have to be stuck with internet (re)search as fn: natural language -> black box we can't do anything about -> ranked_list/summary?

There is a long history of people trying to do very fancy things that end up being done in relational databases and a little SQL. There is a gravity to them, a bitter lesson, just like scaling of generalized ml training methods. I mean many, many information products can be built off essentially giant real-time OLAP databases and frontier LLMs writing brilliant SQL+Datalog+vector+Jev etc. queries.

Google Search, Tavily, Exa essentially have the problem of mapping your agents' context you are willing to provide, to a tiny subset of their index. You pay a fixed cost to an extremely hard problem that has a distribution of hardness, which means YOU eat the downsides when they are running out of budgeted compute to help you out.

Their algorithms are opaque to the caller, there's really not much user control, and there's not a serious opportunity to communally improve search recipes, like the lexical+Jev recipes you trust to select bleeding edge AI builders.

Furthermore, search companies aren't even pursuing text-to-SQL anymore (several have talked to me)... they made up their minds during the traumatic 2024 text-to-sql days. They were just too early.

Meet Scry, where I have a 500 TB NVMe internet index (I'm doing my best indexing and normalizing all the intelligence explosion alpha) that you can run ~arbitrary readonly SQL and some of Datalog over, and I handle the problem of resource-contention with congestion-based micro-auction pricing. When there's capacity, the service is free for non-commercial use.

I hope you enjoy. I'm intent on scaling this paradigm on differentiated hardware over much more data, so any compelling use cases or queries I could show off, would be much appreciated!

29 comments

[ 0.21 ms ] story [ 44.5 ms ] thread
MrDrMcCoy
This is a very good thing, thanks! Have you talked with any of the smaller search engines like Kagi, Qwant, Brave, Mwmbl, DDG, etc to have this supplement the quality of their results? This seems like a big step towards breaking Google and Bing's dominance in search.
Xyra
that's a good idea. it's very doable
neilellis
Please have a 'readable version' option so I don't have to exhaust myself parsing the sites layout. I get that it's unique but most of us just want to work out what you're offering in 5-10 seconds of our time.
JLO64
I strongly second this, although I must admit it loaded surprisingly fast for me as I'm on a mobile hotspot in the back of a car.
hmartin
HN: This site looks like all the other slop, awful to read.

Also HN: This site is doesn't look like other sites, awful to read.

ai-inquisitor
I'd bet that no one, not even the site's [human] creators, has ever read that homepage end to end. At best, it might have been handed over to a swarm of reviewer agents.
DylanMerigaud
Congestion pricing for queries sounds innovative.
vova_hn2
Pricing model is hard to understand at a glance. It uses a term "second of query time" which is not a conventional term and not defined anywhere. Also, all pricing related pages seem to be LLM-generated and are hard to read for a human.

Please, just explain in your own words, how the pricing works, without using made up terms invented by an LLM.

Xyra
Hi, yes. New users right now get free credit. When the server isn't burdened, queries are currently ~free (under $.01 per second, when most are well under a second). We're just trying to learn here how to use this thing productively together.

As load increases, the price goes up. There's also exponential egress pricing because Scry is a place where computation happens, not a metaphorical torrent. I'm still tweaking things, balancing between multi-user server load, giving normal active human users lots of priority and no fear in using it hard and deliberately, and fair pricing for agents and automated heavy workflows people set up in the background.

ashkankiani
Cool idea. The pricing model reminds me of my time working in algorithmic trading, haha. I'll try this out for some queries I wanted to run.

I suppose the scraping you're doing is a huge part of your value proposition, but I would like to gently nudge you in the direction of making the datasets available via p2p (e.g. a torrent) like how Wikipedia distributes its snapshots in the spirit of democratizing access to data that is becoming increasingly walled off. Also, I think another potential benefit that kind of bulk sharing would have is relieving the congestion from those doing the equivalent operation to extract data via the querying interface.

Xyra
thanks, let me know if its faster or slower than you expected
Xyra
let me know if it was faster and more compositionally expressive than you expected
ares623
> Furthermore, search companies aren't even pursuing text-to-SQL anymore (several have talked to me)... they made up their minds during the traumatic 2024 text-to-sql days. They were just too early.

Can you expand on this? Is text-to-sql a deadend? (I personally think it is, having worked on a project at $work. But curious to hear about others experience.)

letmevoteplease
Cool but please rewrite the text. Ctrl + F "land" gives 29 matches.
Xyra
Good point, fixed
mrbluecoat
Scry, keyword action that allows a player to look at a specific number of cards from the top of their library and then arrange those cards in any order, placing any number of them on the bottom of the library and the rest on top.
codexon
How did you scrape reddit comments? Doesn't this require expensive licensing from reddit? How do you handle comments that were deleted by users?
arboles
I'm out of my depth, search is just really cool. If Scry beats the current DeepSearchQA leaderboard why isn't it in the leaderboard?

Will it be added? Are you forced to use a traditional search engine to make it in the leaderboard?

https://scry.io/deepsearchqa

https://www.kaggle.com/benchmarks/google/dsqa

Xyra
good idea, I emailed them. And no search engine, only Scry or direct web fetch of an URL in a Scry result or from the agent's memory.
antoniojtorres
Fascinating stuff. Curious how you handle the ingestion of data from so many sources cost effectively
NoahZuniga
I was actually able to find something with this that I would have never been able to find with traditional search! This is great.