HN Explorer

Do you need to scrape data from 100s to 1000s of websites?

2 points by asim-shrestha ↗ HN
We've been working on AI agents and in the process found what we built could be good at dynamically scraping data across a variety of websites (10s to 100s of different sites at a time)

Existing web scraping tools are bad at this because you need to write custom configurations per site. Not only that, but when a site changes styling, it might completely break your automation

We’re curious how useful this might be for people. If you’ve experienced issues that this might solve or have already explored the space, we’d love to talk to you!

5 comments

[ 0.20 ms ] story [ 173 ms ] thread
airstrike
you should post this either as a Show HN or Launch HN post, explaining why it's relevant. This current post feels spammy and requires readers to copy-and-paste the link*

https://news.ycombinator.com/shownew

https://hn.algolia.com/?q=Launch+HN

having said that, it seems like you may have already posted? https://hn.algolia.com/?q=reworkd.ai

asim-shrestha
Ah, none of those existing posts are relevant to this. Updated the messaging a bit here to sound less spammy - thank you

Additionally, we don't actually have something to show/launch. Was just interested in hearing perspectives of people that have had this problem or have tried to tackle this problem

airstrike
Got it. Maybe an Ask HN then? Otherwise the question may not be noticed in the hundreds of daily submissions. It may be worth resubmitting as an Ask HN. The question mark makes it sound like the start of an infomercial, but it may just be me ;-)
asim-shrestha
Gotcha, will do

Appreciate the feedback, very new to posting here