Ask HN: What is the best way to get a data dump of HN?

3 points by c1sc0 3y ago ↗ HN

I’m working on a little text indexing side project & I think the content posted to HN would be a good dataset to work on. What is the best way to get a dump of all the url’s That have been submitted to HN? Asking for ideas before firing up a crawler. Are there existing dumps? APIs?

3 comments

[ 2.4 ms ] story [ 14.7 ms ] thread

yamrzou 3y ago ↗

There is a BigQuery dataset: https://console.cloud.google.com/marketplace/details/y-combi...

anigbrowl 3y ago ↗

Have you considered looking at the bottom of the page as well as well as the top?

krapp 3y ago ↗

You can find a link to HN's API in the footer of the page. Unfortunately, it's a bit awkward to work with, but it isn't rate limited.