There was a link to the API documentation at the bottom for as long as I can remember using HN, but I’m not sure if it always pointed to the GitHub-page linked in this post.
Wish API would show comment votes instead of only showing them up until 8 years ago when displaying them was turned off on the main site. Aside from allowing users to enable them via their on interface, do analysis, etc — it broke HN’s comment search based on comment popularity.
If it was up to me, all data would be publicly available aside from emergency recovery email and password.
This would include mod commutations and mod logs; unless it was security related, which would have a formal process with deadlines for release.
—-
To address you specific question as why the comment voting data would be of use, among other reasons, it would allow users to have custom filters. For example, posts and comments with high up/down votes tend to be flamewars. Mods also random boost/ding posts/comments too.
I hear you, but what is the endgame with this? Does it lead to a better HN?
Lobste.rs actually exposes even less and doesn't provide an API because the users haven't consented to such intrusions.
Personally, I don't want to be able to review every vote for or against my HN account. It wouldn't be productive and would likely lead to me not wanting to be here at all. It'd be like having a super strong mirror and then looking at your skin and being confronted with too much information about the reality of how nasty looking and imperfect being human is. Basically not a healthy pastime.
FWIW, if you're truly passionate about this, it's not difficult to setup your own site with the lobste.rs open-source code. Maybe there is a whole subset of like-minded folks.
The lobste.rs origin story may also be of interest to you. The TL;DR is jcs was being singled out and abused by pg on HN. Thankfully there is no evidence dang has (or would ever) do this, it was certainly toxic behavior on Paul's part. Being a moderator isn't for everyone, it's a Certain Thing, and net-positive mods who avoid power tripping are truly rare gems on the Internet. Open information would guarantee such abuses couldn't happen in the dark, but at what cost to the best aspects enabled by the HN ecosystem?
This is a website where people post links and other people comment below them. Most front page discussions don't surpass 50 replies. I don't really think we should care so much about this place to the point that we're willing to employ state of the art data analysis tools just to read a blog post and see if someone has critiqued it in the comments
For starters, that you only want HN to be used the way you want to use it; which for sure it not the way I want to use it and more than likely significant number of other users too. If that’s not gatekeeping, then what is?
> Lobste.rs is basically dead compared to HN — largely because it’s an HN clone.
My outsider opinion is that lobsters is dead because the sign up process is designed to keep people out. I appreciate there's also the network effect of people coming here because people come here, but I for sure would visit lobsters too if I could participate without having to know a friend to get past the velvet rope
In 2019 I set up a bot to store titles for HN stories as soon as they appear on the "new" page, so that I could then compare them to the current version.
I had also made a browser extension for FF and Chrome that compared the current version of any HN story to the stored one, and displayed a warning if they differed.
There was zero downloads of the extensions, so they fell into disrepair and don't work anymore, but the cron job for storing the titles is still active.
If there is any interest I could resuscitate the extensions, or some other system to make that information useful.
It's not going to happen. Showing comments led to constantly flamewars about unfair commenting. Even without the comment scores, the mods are still constantly having to remind people to stop arguing about scores. The decision not to display or make available other people's scores was litigated intensely; it is now part of the premise of the community.
By “showing comments led” do you mean showing comment points/votes? Also, if comment viewable comment points are gone forever, then the “popularity” filter should be removed from HN’s search and the old voting data removed from the public API.
To me, if users have concerns about “unfair voting” to me the solution is not to hide them. Also, if comment points aren’t important, why even allow them? They are obviously important and if a users is not available to respect guidelines not to complain about them, what’s difference between that an guidelines related to comments about being downvoted?
Like I said, this has been heavily litigated. So much so that I think you'd have difficulty coming up with an argument that hadn't been made already about it. Not making other people's comment scores available is, as I said, a part of the premise of the community. There are other communities that work differently. There isn't a single optimal set of policies.
You’re right, obviously the best way to secure the community was to obscure the votes of people that were upvoting toxic content instead of the mods/community just publicly tagging and flagging the related undesirable content and penalizing the voting weights of anyone upvoting that content.
If votes being public impacts voting patterns, listing top users, having usernames publicly associated with comments, being able to follow given user’s comments, etc — for sure does too. What’s the difference?
You're arguing here as if there is an argument to be had. I'm trying to help you understand that there is no such argument. Your claims have been considered and rejected hundreds of times over the last 10 years. It'd behoove you to consult the search bar below and read some of those old discussions. This isn't a case where the community made a rash decision and just doesn't realize how problematic it is; you're talking about what is probably the single most deliberate community management decision HN has ever made. It's not going to change.
A moment later
Maybe it'll help if you get your head around the goal of HN. There are a bunch of things you personally seem to want to accomplish with HN. But HN itself has just one goal: to nurture curious conversation. Other sites have other, equally valid goals: propagating the news as quickly as possible, or relentless focus on a particular topic, or getting the best Q&A pairs to the top of Google's search results. HN, though: curious conversation. That's the whole ballgame.
Anything that drags on curious conversation is a non-starter on HN, even if it might make a bunch of things better for you.
We don't have to guess about whether making comment scores available is a drag on curious conversations; it manifestly was, for years. People have an innate, visceral reaction to comment scores they feel are unjust, and they talk about them, and those conversations choke the landscape like kudzu.
A lot of things about HN make more sense when you accept the premise of the site, and understand that HN will make most sacrifices it can come up with to optimize for that premise.
Just to be clear, intentionally not responding, since you repeatedly in my opinion ignored my points and contrary to your claims, did not engage me out of intellectual curiosity.
I think tptacek is saying something meaningful in suggesting that there's already a long history to this discussion. Personally I disagree that the soul of HN is about curiosity. If you removed all the explicit comments about curiosity and showed the site to new tech people, I don't think there would be consensus on curiosity. If you showed a bunch of people the Rust subreddit, there would be consensus that it's soul is Rust.
Personally I see this site as the Reddit of tech with 20% of its flavor coming from YC companies. That is how it behaves, as opposed what moderators say the site is about.
'curious conversation' is a much loftier, woolier and syntheticier goal than 'Rust' and HN strives towards it imperfectly. The measure, though, is 'does this or that change bring us closer to it' not 'can people unfamiliar with the site achieve mindmeld with dang by staring at the front page'.
I’d like to be able to conveniently access my most popular comments. There are quite a few things in there that I might like to write longer-form articles about.
Which would be possible if HN was API first and all data beside email for recovery were public. Otherwise, HN is simply not designed to be responsive to user system design requests — but their have been a massive amount of alternative systems built by users that built things HN would have never made.
(Aside: I finally got round to scraping my comments in that way. My comments follow the sort of distribution you’d expect, ranging from −4 to +76, but there’s one outlier at +283, https://news.ycombinator.com/item?id=27274146. People really hate stale bots on bug trackers.)
I don’t want vote information to be publicly available. Hiding those numbers was good for the site. Users have ways to get their own data. That’s enough.
I used to query flagged comments from time to time using the API, just out of some morbid curiosity. It definitely used to work for that; I am not sure if flagged items are in the same class as dead ones, though, or if they have since deliberately eliminated this capability. I eventually found more interesting things to do :)
The problem was that many third party apps and clients were displaying flagged and [dead] comments as if they were normal comments. Then we'd get angry emails about all the horrendous comments we were "condoning" when in fact the users posting them were as banned as banned could be.
Would it be feasible to make a Firebase-to-Supabase connector?
Having a self-hosted private HN Algolia (perhaps via Elasticsearch) would be nice.
Algolia's forced fuzziness and limited number of results per page are a drag. Why can't they provide any search controls? Or at least support the "-" operator to exclude a term.
Firebase is owned by google. There are plenty of OSS solutions to create this exact realtime thing without getting too close to the elephant in the room.
> The v0 API is essentially a dump of our in-memory data structures.
TIL that HN doesn't use a "proper" / persistent DB. I wonder then how HN handles data persistence. Is it as simple as creating write-ahead logs + checkpointing every some duration?
No they don’t. This forum was created as a MVP of a webapp using PG’s Arc Lisp. All user and comment data are stored as tables serialized to text files or in memory. You can find an old version of the code at http://arclanguage.org
So they do use a database? In the form of text files. The comment I was responding to was edited, and before it was asking if they stored everything in memory and how they handled reboots.
I'm currently writing an HN posts score tracker (basically it plots score variations over time since they were in new for each popular post) and it's really handy that you can just subscribe with eventsource [0] and get realtime update when something changes without having to manually poll each post wasting resources. Not yet sure about the latency though.
I know but it does not provide detailed and raw data, making it impossibile to do a bunch of interesting in-depth analysis. Also, I like to build this kind of stuff to explore async rust capabilities.
Only using the official API is quite cumbersome, as fetching a story with comments requires one network request per each comment.
Instead, try considering the (more unknown) Algolia API[0]. It supports fetching the entire tree in one request. Unfortunately the comments aren’t sorted, so you’ll still have to use the official one (or parse the HN HTML) if you need that.
For my HN client[1], I parse the HN HTML (as it’s required for voting), and sort the comments from the Algolia API. It’s much faster than recursively requesting the official API.
I found this interesting, looking at the Firebase API. I guess one could get the min/max item ids for the child posts, and query the range of ids (startAt(), endAt()).
If you want to see a small example of this API being used, here's a utility to generate an RSS feed of replies to comments you post that you can self-host:
I wrote a fairly simple "Who is hiring?" browser (using this API) that fetched the post listings and stored them locally in an HTML file accessible by pointing your browser at file:///... . It supported filtering by "Remote", "Interns", and "Visa" with buttons, had an ad hoc Regex filter, and the ability to remove posts using a delete button and not see them again (or to restore all deleted posts) using the localStorage api.
It was indeed barebones but I'd like to think it helped me find a job. I would link the code (it's on GitHub) but today the "Who is hiring?" posts include a few considerably more advanced and capable "searchers" right in the top of the post:
I am definitely happy to share, but I didn’t expect any interest and I posted my comment under my HN pseudonym account. However when I link to the repo on GitHub that thin veneer of obscurity will disappear and I’d prefer that it did not. Being a programmer, I am quite familiar with workarounds and I am happy to offer you couple of different options:
1. You could add an email address to https://news.ycombinator.com/user?id=maxique and I can email you there - I’m happy to share my name and contact info in a one-off request, just not on an ongoing basis for anyone to see.
2. I could clone the repo onto my current machine and delete the .git dir, leaving the code and the README.md intact, produce an archive of that stuff, and link it here in this thread. While I’m sure it may be possible to use a code search tool to find the repo on GitHub after that, I am not terribly bothered by that.
I am open to other options as well if you have any suggestions. Finally, the code was written for whatever version of the Ruby interpreter (MRI) was current in 2016, in case that will dissuade your interest.
EDIT: I have a bias to action, appreciate interest in my work, and figured it was likely that you would see my reply at some later point and then I'd have to see your reply to my reply at some even later point, and we'd continue the loop, and eventually your interest would wane. So I went with option two. Here you go: https://we.tl/t-Bu257EdQn9
This is my first time using WeTransfer; let me know how it works out for you. I actually originally had a link that I hosted on file.io (which I've also never used before), but it looks like that gets automatically deleted after one download, which seems slightly excessive to me.
For a few years I am working on the side on a HN client. One thing that is really sad is that there is now way to handle authentication properly as a 3rd party. The user basically has to trust you with her/his password or you have to "steal" the auth token from a login.
137 comments
[ 2.7 ms ] story [ 230 ms ] threadhttps://github.com/overshard/newtab/blob/master/newtab.js#L1...
Haven't had any issues with it!
Related:
https://news.ycombinator.com/item?id=32535637
We're already helping to create a training set of slightly above average English speaker discourse for free.
This would include mod commutations and mod logs; unless it was security related, which would have a formal process with deadlines for release.
—-
To address you specific question as why the comment voting data would be of use, among other reasons, it would allow users to have custom filters. For example, posts and comments with high up/down votes tend to be flamewars. Mods also random boost/ding posts/comments too.
Lobste.rs actually exposes even less and doesn't provide an API because the users haven't consented to such intrusions.
Personally, I don't want to be able to review every vote for or against my HN account. It wouldn't be productive and would likely lead to me not wanting to be here at all. It'd be like having a super strong mirror and then looking at your skin and being confronted with too much information about the reality of how nasty looking and imperfect being human is. Basically not a healthy pastime.
FWIW, if you're truly passionate about this, it's not difficult to setup your own site with the lobste.rs open-source code. Maybe there is a whole subset of like-minded folks.
The lobste.rs origin story may also be of interest to you. The TL;DR is jcs was being singled out and abused by pg on HN. Thankfully there is no evidence dang has (or would ever) do this, it was certainly toxic behavior on Paul's part. Being a moderator isn't for everyone, it's a Certain Thing, and net-positive mods who avoid power tripping are truly rare gems on the Internet. Open information would guarantee such abuses couldn't happen in the dark, but at what cost to the best aspects enabled by the HN ecosystem?
References:
https://news.ycombinator.com/item?id=4452568
https://news.ycombinator.com/item?id=32112665
And yes, aware of Lobste.rs mod logs, but Lobste.rs is basically dead compared to HN — largely because it’s an HN clone.
My outsider opinion is that lobsters is dead because the sign up process is designed to keep people out. I appreciate there's also the network effect of people coming here because people come here, but I for sure would visit lobsters too if I could participate without having to know a friend to get past the velvet rope
In 2019 I set up a bot to store titles for HN stories as soon as they appear on the "new" page, so that I could then compare them to the current version.
I had also made a browser extension for FF and Chrome that compared the current version of any HN story to the stored one, and displayed a warning if they differed.
There was zero downloads of the extensions, so they fell into disrepair and don't work anymore, but the cron job for storing the titles is still active.
If there is any interest I could resuscitate the extensions, or some other system to make that information useful.
To me, if users have concerns about “unfair voting” to me the solution is not to hide them. Also, if comment points aren’t important, why even allow them? They are obviously important and if a users is not available to respect guidelines not to complain about them, what’s difference between that an guidelines related to comments about being downvoted?
If votes being public impacts voting patterns, listing top users, having usernames publicly associated with comments, being able to follow given user’s comments, etc — for sure does too. What’s the difference?
A moment later
Maybe it'll help if you get your head around the goal of HN. There are a bunch of things you personally seem to want to accomplish with HN. But HN itself has just one goal: to nurture curious conversation. Other sites have other, equally valid goals: propagating the news as quickly as possible, or relentless focus on a particular topic, or getting the best Q&A pairs to the top of Google's search results. HN, though: curious conversation. That's the whole ballgame.
Anything that drags on curious conversation is a non-starter on HN, even if it might make a bunch of things better for you.
We don't have to guess about whether making comment scores available is a drag on curious conversations; it manifestly was, for years. People have an innate, visceral reaction to comment scores they feel are unjust, and they talk about them, and those conversations choke the landscape like kudzu.
A lot of things about HN make more sense when you accept the premise of the site, and understand that HN will make most sacrifices it can come up with to optimize for that premise.
Personally I see this site as the Reddit of tech with 20% of its flavor coming from YC companies. That is how it behaves, as opposed what moderators say the site is about.
(https://james.darpinian.com/blog/scraping-my-own-hacker-news... gives one way of doing it, by crawling your /threads?id=‹username› page, which does contain the point counts for each comment by you.)
[1] https://github.com/superb-owl/hacker-news-comments
[2] https://superbowl.substack.com/p/commenting-on-hacker-news
Is there any way to query this API as if showdead is switched on?
Having a self-hosted private HN Algolia (perhaps via Elasticsearch) would be nice. Algolia's forced fuzziness and limited number of results per page are a drag. Why can't they provide any search controls? Or at least support the "-" operator to exclude a term.
curl https://api.news.ycombinator.com/
See the link at the bottom of every HN item page.
Edit: I took a look, it seems like Firebase is owned by Google now, but it was original a YC-funded startup, judging by this page: https://www.crunchbase.com/funding_round/firebase-pre-seed--...
https://www.ycombinator.com/companies/firebase
This repo started in 2015, and Firebase was bought in 2014. I'm really surprised; I thought it was way more recent than that.
TIL that HN doesn't use a "proper" / persistent DB. I wonder then how HN handles data persistence. Is it as simple as creating write-ahead logs + checkpointing every some duration?
[0] https://github.com/jpopesculian/reqwest-eventsource/
Gives a few more variables, and the different sample times show a different set of transient effects.
I wrote a script that uses your browser cookies to scrape them from the website here: https://github.com/superb-owl/hacker-news-comments
Instead, try considering the (more unknown) Algolia API[0]. It supports fetching the entire tree in one request. Unfortunately the comments aren’t sorted, so you’ll still have to use the official one (or parse the HN HTML) if you need that.
For my HN client[1], I parse the HN HTML (as it’s required for voting), and sort the comments from the Algolia API. It’s much faster than recursively requesting the official API.
[0]: https://hn.algolia.com/api
[1]: https://github.com/goranmoomin/HackerNews
Van Puffelen answers some of this here, but he doesn't get into specific queries: https://www.youtube.com/watch?v=66lDSYtyils
He also mentions here that Firestore offers a bit more for WHERE type clauses over Firebase:
https://stackoverflow.com/questions/26700924/query-based-on-...
https://hngraphql.fly.dev/graphql
https://gist.github.com/Q726kbXuN/15e61acc003bb6d46a458001fe...
It would be a shame for this vast trove of expert knowledge and nerd sniping to vanish into the ether after the next hard drive apocalypse or similar.
It was indeed barebones but I'd like to think it helped me find a job. I would link the code (it's on GitHub) but today the "Who is hiring?" posts include a few considerably more advanced and capable "searchers" right in the top of the post:
> Searchers: try https://kennytilton.github.io/whoishiring/, https://hnjobs.emilburzo.com, https://news.ycombinator.com/item?id=10313519.
It was fun; I enjoyed working on it.
I am definitely happy to share, but I didn’t expect any interest and I posted my comment under my HN pseudonym account. However when I link to the repo on GitHub that thin veneer of obscurity will disappear and I’d prefer that it did not. Being a programmer, I am quite familiar with workarounds and I am happy to offer you couple of different options:
1. You could add an email address to https://news.ycombinator.com/user?id=maxique and I can email you there - I’m happy to share my name and contact info in a one-off request, just not on an ongoing basis for anyone to see.
2. I could clone the repo onto my current machine and delete the .git dir, leaving the code and the README.md intact, produce an archive of that stuff, and link it here in this thread. While I’m sure it may be possible to use a code search tool to find the repo on GitHub after that, I am not terribly bothered by that.
I am open to other options as well if you have any suggestions. Finally, the code was written for whatever version of the Ruby interpreter (MRI) was current in 2016, in case that will dissuade your interest.
EDIT: I have a bias to action, appreciate interest in my work, and figured it was likely that you would see my reply at some later point and then I'd have to see your reply to my reply at some even later point, and we'd continue the loop, and eventually your interest would wane. So I went with option two. Here you go: https://we.tl/t-Bu257EdQn9
This is my first time using WeTransfer; let me know how it works out for you. I actually originally had a link that I hosted on file.io (which I've also never used before), but it looks like that gets automatically deleted after one download, which seems slightly excessive to me.
https://github.com/ashish01/hn-data-dumps