Ask HN: Is there a service that offers Common Crawl as an API? 7 points by georgehill 1y ago ↗ HN I am trying to do some data analysis work. I don't want the full dataset. I want only two things: give me the hostname, and give me all the pages or URLs with their HTML.
[–] phillipseamore 1y ago ↗ Not that I know of but there are various tools like https://github.com/alwalxed/wayurls [–] georgehill 1y ago ↗ thank you will check this out
[–] pluto_modadic 1y ago ↗ there's index.commoncrawl.org where you can ask for a domain with wildcards.
3 comments
[ 5.3 ms ] story [ 26.7 ms ] thread