Ask HN: Have you used a good general purpose LLM-powered scraper?

I'm looking to either build or use a model or SaaS that could be used with a prompt like:

Find the 10 biggest realtors in [CITY], then extract all their listings within [BUDGET] in JSON with all the data you can find.

I'm looking for a tool that can use search inputs on any website, click next page buttons or handle infinite scroll.

Is there a tool like that on the market that actually delivers?What I've seen for now doesn't seem good enough.

3 comments

[ 0.17 ms ] story [ 20.0 ms ] thread

coderatlarge 2y ago ↗

One precursor question would be whether an LLM can extract the data you want from raw html even when copy-pasted manually. In my limited experience we’re not quite there yet, but I’d be curious to hear of others have different experience - or better yet, actual measurements against a baseline scraper.

FrenchDevRemote 2y ago ↗

Oh it definitely can. I tried with GPT-4 and Anthropic(claude 2), it was relatively good, although I had to tweak the prompts to get everything I wanted(sometimes it forgot images or urls, because my prompt was too vague)

You could also write a simple function that turn HTML into a json that only contained innerText, src and hrefs, and ask the LLM to only keep the relevant data

mdev23 2y ago ↗

I am working on an LLM agents tool that could probably help with this: https://recurai.com

Reach out and i'll try to get you setup: matt@recurai.com