How did OpenAI crawl the web so effectively without getting blocked?

6 points by BrownSol 2y ago ↗ HN

4 comments

[ 6.0 ms ] story [ 18.6 ms ] thread

It didn't. Others did it, OpenAI used it. https://commoncrawl.org/

Very carefully

I'm pretty sure scraping is easier than building a groundbreaking LLM used by 100M people

It's not hard to scrape the internet