How did OpenAI crawl the web so effectively without getting blocked?

6 points by BrownSol 3y ago ↗ HN

4 comments

[ 0.23 ms ] story [ 19.6 ms ] thread

It didn't. Others did it, OpenAI used it. https://commoncrawl.org/

Very carefully

I'm pretty sure scraping is easier than building a groundbreaking LLM used by 100M people

It's not hard to scrape the internet