Ask HN: Would an easier way to scrape 100s of websites be useful to you?
In the process of building AI agents, we've found that what we built could eventually be good at dynamically scraping data across a variety of websites (10s to 100s of different sites at a time)
Our understanding is that existing web scraping tools are bad at this because you need to write custom scraping configurations per site. Not only that, but when a site changes styling, it might completely break your automation. With agents however, you can provide a high level natural language overview of the data you'd like from a website or class of websites, and the agent system will deal with the details of traversing a page and fetching data automatically.
We’re curious how useful this might be for people. If you’ve experienced issues that this might solve or have already explored the space, I'd love to hear from you!
10 comments
[ 3.4 ms ] story [ 202 ms ] threadMy sites are periodically on the wrong end of scrapers, greedy by design or in error, occasionally needing to be manually blocked or even legals threatened.
Just because something can be done, done't mean it should. It also doesn't mean that you should make it easier. Any more than offering 'better' SPAM engines...
I don't know how relevant this is, but I was thinking that you could probably use some sort of AI to enhance OCR and convert written documents into some sort of semantic form like HTML or Latex. That would allow you to use books to scrape information, and written books still have a lot of untapped knowledge.
It seems like the demand for web scraping and such is to create datasets for ML training. And now you are using AI for scraping. So it is sort of a self-improving cycle
Interesting idea, we're definitely looking into coupling OCR and LLMs today but not for that particular case. I think raw language models with a good workflow are typically good enough to extract structured data from things like books
ML training is definitely one area we can see this being useful. General data aggregation across a large industry (clothing, retail, etc) is something we want to look into. Also RPA style workflows involving multi-click actions across a variety of sites
Scraping at larger scale is where it becomes challenging, the problems we want to tackle are: 1. You need at least a bit of technical expertise to do things like configure selectors properly 2. Websites typically have moderation in place to block scrapers 3. Scrapers are prone to changes in the site layout 4. Creating on the order of 100s of scrapers is difficult and time consuming. Creating this many will amplify the previous issue
Not to me but I would be curious if you found a way to mimic a real human browsing a site aside from Chrome Headless. Do your TCP packets and HTTPS requests look indistinguishable from real people? The reason I ask is that its a fun hobby for me to see if I can block a scraper without any proprietary tools.
Unfamiliar with blocking mechanisms, could you share some things you would do to block existing selenium scraping jobs?
So I guess to answer your question I would have to see some example packets. You could send some requests to the awful little blog I have in my profile if you were willing to share.
I was tinkering with custom figlet ASCII text at one point which automation can be made to solve, but unless I were hosting a popular site nobody would bother and I could just rotate through half of the figlet fonts, modulate the spacing, direction and letter overlap to make it fun. For now however I try to avoid anything that requires client interaction also fully accepting I am not doing anything as advanced as CDN's like Cloudflare do in their sleep. It's just a hobby for me.
[1] - https://www.zenrows.com/blog/selenium-avoid-bot-detection