HN Explorer

Tell HN: Hacker News BigQuery dataset is updated daily again

51 points by wietsevenema ↗ HN
As you might know, there's a Hacker News BigQuery dataset [1].

I recently noticed that the export was broken (since Nov 2022) and I worked with the team to help fix it. It'll now again run daily.

SELECT timestamp FROM `bigquery-public-data.hacker_news.full` ORDER BY id DESC LIMIT 1 # 2024-06-09 09:12:55 UTC

[1]: https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news

11 comments

[ 11.0 ms ] story [ 165 ms ] thread
dang
Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.
izyda
Any interest in getting this dataset on Snowflake too?

(I run a Snowflake funded data provider on Snowflake marketplace)

dang
The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API), so anyone is free to do this.
Raed667
a bit off topic: is there a way to display a HN page using the firebase API/sdk without doing N+1 requests ?

(sorry for noob question)

dang
No, unfortunately. We're going to make a new API that will be much easier, but there's at least one other major project that needs to get done first.
Raed667
Thanks for the response!
wietsevenema
Yes, that’s how it works. Same data as in the API.
zX41ZdbW
Daily is too infrequently.

I have this dataset being updated in ClickHouse in real time: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...

I also provide a way to export it or attach it to clickhouse-local and analyze it locally: https://github.com/ClickHouse/ClickHouse/issues/29693#issuec...

tarasglek
I love that this dataset is accessible via cors from web frontends. Got me into experiments with clickhouse

Same with github, much better than than throttled apis