HN Explorer

104 comments

[ 3.9 ms ] story [ 177 ms ] thread
DonHopkins
OMFG somebody remembers OPML!
outcoldman
RSS is the only way to read news :D Reddit, HN - everything goes to my RSS ;)
DonHopkins
Then why not use RSS, or Atom, or CSV, or HTML, for that matter?

That OPML "outline" has no tree structure at all, except for the single top level container item:

    <outline title="HN Personal Blogs" text="HN Personal Blogs">
What is the point of using OPML for a flat structure?

Outlines are all well and fine, but a list of links to blogs is hardly an outline.

And isn't JSON the simplest most ubiquitous accessible standard format for outlines and even arrays of links these days, if not just raw-dogging XML?

More thoughts and opinions and history of OPML, RSS, Dave Winer, More, Frontier, Manila, Radio UserLand, XML-RPC, SOAP, Xanadu, spreadsheets, CSV, JSON, XML, etc:

https://news.ycombinator.com/item?id=21170434

https://news.ycombinator.com/item?id=16224154

https://news.ycombinator.com/item?id=20780928

https://news.ycombinator.com/item?id=35674125

https://news.ycombinator.com/item?id=36544675

NoboruWataya
I think it's just that OPML has traditionally been used to import/export lists of RSS feeds, and therefore most readers are capable of importing an OPML file whereas other formats would not be as widely supported.
DonHopkins
Why do RSS feeds need to be imported/exported as anything other than RSS feeds?

According to this thread, four out of five RSS apps on Linux Mint failed to import the OPML file, and the one that did was buggy:

https://news.ycombinator.com/item?id=36627431

Maybe I'm thick, but it bewilders me that someone wouldn't simply use RSS or Atom directly, instead of abusing OPML to mis-represent it (which is itself just as terribly designed as RSS, by the same person, Dave Winer -- don't even get me started).

Is this some kind of a masochistic corrupted data ingestion challenge from TikTok that I'm not aware of, akin to eating Tide Pods or washing your clothes in Campbell's Soup?

Does anyone even ever use OPML for outlines any more? Or is it like YAML's backflip, where its original name meant "Yet Another Markup Language" until somebody pointed out to their great surprise that it WASN'T actually a markup language, so they retroactively recursively renamed it "YAML Ain't Markup Language" on opposite day?

So does OPML no longer stand for "Outline Processor Markup Language" any more, but now stands for something else like "OPML Presents Mere Lists"?

What's wrong with exporting and importing RSS as RSS? Has its meaning also changed, flipping places with OPML, so it now means "RSS Serves Structures", so you use RSS for nested trees and OPML for flat lists now?

I suppose you could recursively encode outlines as nested RSS feeds inside the text content other RSS feeds as <![CDATA[ at each level, or since those can't be nested, exponentially doubling the number of escaped entities at each level.

I Wanna Be <![CDATA[ -- Sung to the tune of “I Wanna Be Sedated”, with apologies to The Ramones:

https://donhopkins.medium.com/i-wanna-be-cdata-3406e14d4f21

themodelplumber
RSS traditionally kinda sucks for discovery purposes though. (Unless you count these HN threads as the discovery part)

I do wonder what's a good way of doing discovery in software, FOSS-style. Using open source client side and free APIs if needed.

paladum
What about subscribing to OPML files hosted on URLs ? That way one can follow OPML list "hosts" (like the one on OPs repository) and discover new RSS feeds. I'm quite surprised that this feature do not exists in RSS reader I tried.
dagurp
Good. I'm tired of apps suggesting stuff to me.
award_
This is the way. I just swapped as many news, social (reddit/etc) and the like over to newsblur via RSS and I'm VERY happy. This OPML will go a long way to filling out my categories :-) Thanks a bunch for your efforts!
revskill
How do you update this list on a daily basis ?

Github actions is an option.

outcoldman
I am not sure if users of HN will be updating the original HN discussion in general, but yeah, GitHub actions can be an option for this.

For now, I just wanted to add those 600+ blogs to my RSS and filter them out slowly if I need to.

revskill
If possible, could you also add to description for anyone wants to add to your list, to go to original thread and submit ?

Currently it seems there're many people fetch from original thread, which makes it ne source of truth, less confusing for anyone to submit.

outcoldman
Yeah, the best way is to go back to the original post and submit comment there on top level. Make sure HN recognize that as a correct href and your blog has alternative link to rss/atom. And for now only users with 100+ karma are included.

I left the console.log there as well to show which blogs aren't recognized.

I am also open for PR. It was just a few hours project to get it going, but if people will find it useful, would be nice to get some tags from the blogs, so it would be possible to extract the list only for specific technology/topics.

BeetleB
> Make sure HN recognize that as a correct href and your blog has alternative link to rss/atom.

Any details on how to do this? My blog (https://blog.nawaz.org/) has an RSS feed here (https://blog.nawaz.org/feeds/all.atom.xml) but I don't know how to set it up the way you'd need it.

distcs
I don't know Go's flavor of regex but is this correct parsing code: https://github.com/outcoldman/hackernews-personal-blogs/blob...

Are you taking care that if the <link> tag spans over multiple lines (linebreaks in between), you're still parsing it right?

You do seem to be resolving relative paths with:

  feedUrl := matches[1]
  if feedUrl[0] == '/' {
      feedUrl = url + feedUrl
  } else if !strings.HasPrefix(feedUrl, "http") {
      feedUrl = url + "/" + feedUrl
  }
So I think this is good. Just trying to see what could be breaking for the other users who are complaining their feed isn't listed.
[deleted]
(comment deleted)
generalizations
Ah, shouldn't have used a throwaway to post my blog. Oh well.
outcoldman
You can comment on that original post again with normal account, and I will regenerate.
defrost
Good stuff, thanks.

Needs a bit of lint work still, eg: current lines 12 & 13 are dupes

    <outline type="rss" title="bookofjoe" text="bookofjoe" type="rss" xmlUrl="https://www.bookofjoe.com/atom.xml" htmlUrl="https://www.bookofjoe.com/atom.xml"/>
    <outline type="rss" title="bookofjoe" text="bookofjoe" type="rss" xmlUrl="https://www.bookofjoe.com/atom.xml" htmlUrl="https://www.bookofjoe.com/atom.xml"/>
but still appreciated as that's a minor quibble easily polished away.
distcs
When do you plan re-run your crawler? HN users are updating that original HN discussion every minute as we speak! It'll be nice to get an updated list sometime today. Possible?

And yes, thanks for your work and sharing it!

outcoldman
I am fixing some issues, trying to get more blogs from the post, so do it a few times in a day.

And hope at some point to actually build something larger from that. I love RSS and this is the best way to discover new content. I actually already have found a few interested blog posts, links, information.

sphars
There seems to be many valid blogs that got skipped due not having the `rel="alternate"` link in the HTML head (mine included). I just added it, as I'm sure others will, so it'd be a good idea to update the list regularly
zorrolovsky
Thanks for the effort, this is a fantastic resource.

I installed 5 RSS reader apps in my Linux Mint machine to test the OPML. 4 of them failed to import the file: Thunderbird, Lifera, Quitrss and another one I forgot the name. Gnome RSS did import the file but it became slow and buggy.

I think the file is too long for a typical RSS feed app to process it. Also, there might be some formatting issues a couple apps threw a "incorrect format" error (Thunderbird being one of them). Now trying CPod and giving it time... I hope in 20 mins or so it can work with it. I wonder if web-based RSS readers will work better with the file.

outcoldman
I run freshrss and was able to import it there.

Do you get any errors while importing it?

I believe also this XML maybe not valid. Will regenerate it a few minutes.

mrloop
I tried importing at feedbin.com, 524 of 598 imported successfully. The failures being a mixture of not found or connection errors and a couple of invalid URL errors
outcoldman
I just resubmitted the list, I believe around 500+ there is an invalid XML tag, which is fixed now. So all of them should be valid now. Forgot to do xml escape for the URLs
winrid
Oh, did it miss mine because I use rss.xml and not feed.xml?

https://blog.winricklabs.com/rss.xml

tommasoamici
No, it's because you should include this link tag in the <head>

    <link rel="alternate" type="application/rss+xml" title="XXX" href="https://blog.winricklabs.com/rss.xml">
winrid
ohmygosh. Thanks, fixed.
BigglesB
Ohhhh! Thank you! Looks like I need to do this too for mine! (https://biggles.games)
BigglesB
Ah, no, I had the atom feed set up already, just insufficient karma on HN... Oh well!
ivolimmen
I just gave you some.. who knows.
tommasoamici
>No feed: https://tommasoamici.com by tommasoamici ( 216 ) no feed found for https://tommasoamici.com

Uh, this made me realize I forgot to add the RSS feed link to the head tag when I recently migrated my blog to Astro, fixed now!

J_cst
Hi Tommaso! Your board games collection is quite noticeable!! Bravo!
victorbjorklund
Same haha. This was great to realize what i missed when rolling my own rss.
akhayam
Can we commit other people's blogs to this?

E.g. I am surprised to not see Julie Zhou's blog in this list--which, I believe, is one of the most influential blogs in the tech product space: https://www.juliezhuo.com/.

freediver
Is there an easy way to make a feed of feeds? So one RSS feed for all these?
outcoldman
That is good idea as well! That would require me to actually run the server (or github actions) to refresh the RSS feed, maybe something I would do later on.
freediver
Can you please ping me once this is up? We'd love to expose this feed in search results inside Kagi.
alanpearce
There are quite common and they're usually called planet <topic>, which might help to find useful resources or software
quickthrower2
A github action updating a static file is, well, webscale. And cheap/free to host.
the_common_man
What's the difference between rss and opml? Just different standards?
outcoldman
OPML allows your to export/import the list of RSS/Atom feeds from your RSS reader. So this is a list of 500-600 blogs that people submitted at that HN discussion, that you can import in your RSS reader.
DonHopkins
Apparently now RSS stands for "RSS Serves Structures" and OPML stands for "OPML Presents Mere Lists".
rambambram
Or 'Really Social Sites' and 'Other People's Meaningful Lists'. ;)
kevincox
OPML (when used in this context) is basically a list of URLs with a touch of metadata. It is the most common format for importing and exporting lists of feeds to move between feed readers.

So you generally don't subscribe to an OPML file, it just provides a list of feeds to subscribe to.

yard2010
HN is such a utopic place. I wonder when the inevitable enshitification will happen, and how?
hnzix
Fear not, dang is Hodor.
bennysaurus
It has its quirks on occasion and has changed a bit but has been around since 2007 and still going great.
Hamuko
Looking at how many "https://url.com//feed.xml" there are in the list, I have a feeling that the scraping logic needs some work. Is it just concatenating "https://url.com/" and "/feed.xml"?
RamblingCTO
It's also not a good idea. On https://andinfinity.eu/ I have a header for that and it clearly indicates that the rss feed lives at index.html. Standard hugo thing I guess.
outcoldman
I was extracting URL from the alternate link in the HEAD of the blog website. The issue is that some people will do "//example.com/feed.rss", some "feed.rss" or "/feed.rss". So I built pretty stupid URL resolving logic, when I just should have used ResolveReference from go runtime. So the last list will fix that.
fomine3
This is the first time to see useful OPML lists except export/import usage
altilunium
Now Is the Perfect Time for an RSS Renaissance (https://news.ycombinator.com/item?id=16907298)

I hope this list is constantly maintained and updated.

Additionally, do you have any recommendations for a good RSS reader that can properly handle this OPML file? I tried using Omea Reader, but it fails to fetch many of the feeds.

saeedesmaili
I was able to easily import the list into Inoreader.
altilunium
> File exceeds maximum allowed number of subscriptions! Your file contains 659 out of 150 maximum allowed subscriptions for your plan.

Alright.. I'm gonna build my own RSS reader then.

Edit : A kind of "decentralized" social media platform where we post on our respective personal blogs, and we may follow each other by using RSS subscriptions. If HN is one such platform, here's the chronological timeline : :https://gist.github.com/altilunium/c2fcbe1e23aeb1cb9564f2593...

While generating this list, I found at least 80 sites with inaccessible RSS link : https://gist.github.com/altilunium/8319dae345f701a2a1efd0152...

saeedesmaili
Ah, I'm subscribed to their Pro plan, didn't know there is a limitation on this on free plan.
splitbrain
If you want a more casual approach then to right away import 600 feeds into your reader, check out the feed options at IndieBlog.page - it gives you daily random posts not only from the 600 URLs of that recent thread but also from a similar one back in April last year and a bunch of other indie sources...

https://indieblog.page/rss

alanpearce
Seems like my blog was not even included in the console output, but it has been part of the thread for a while. https://news.ycombinator.com/item?id=36596022

I know I don’t have enough karma, but it seems like the log output should have logged something for that. I also wonder if you’d consider account age as an alternative to karma for anti-spam, as I’m a bit of a hardcore lurker here and I highly doubt that I’m the only one.

alemanek
You are definitely not the only one.
NoboruWataya
There seem to be some malformed URLs here?

> <outline type="rss" title="ntietz" text="ntietz" xmlUrl="https:&amp;#x2F;&amp;#x2F;ntietz.com&amp;#x2F;atom.xml" htmlUrl="https:&amp;#x2F;&amp;#x2F;ntietz.com&amp;#x2F;atom.xml"/>

outcoldman
Unfortunately that is on u/ntietz, if you look at the https://ntietz.com, for some reason they encoded the href

``` <link rel="alternate" type="application/rss+xml" title="RSS" href="https:&#x2F;&#x2F;ntietz.com&#x2F;atom.xml"> ```

kevincox
Is that wrong or just unusual? IIRC entity escapes are allowed in attributes.
outcoldman
Hard to tell, web is weird :D I contacted the author and they have fixed it!
fluxinflex
I've been playing around with Node-RED[1] for a while and thought I would recreate this using Node-RED (also being a big fan of Node-RED). The flow[2], i.e. code, is online to have a look at (editable but not deployable) and the feed[3] is cached and updated every hour or so.

It's only a small Heroku server so it might well be down or about to crash, I make no promises!

Thanks to the OP for the inspiration, I did take a lot of ideas from the original codebase :)

[1]=https://nodered.org

[2]=https://demo.openmindmap.org/omm/#flow/a8d90fc15ab6990b

[3]=https://blog.openmindmap.org/example/hacker-news-personal-bl...

award_
Thanks! I saw the posts and have been making a move to newsblur recently, this will help a lot with content!
yogsototh
That's really great! Thank you for that.

I saw some people asking for a good RSS reader. Personally I will use elfeed, and I also use elfeed-org so I could import the OPML file and export it into my existing RSS feeds.

outcoldman
I use freshrss with combination of Reeder for Mac/iOS.
Hamuko
My combination of choice too.

The fact that FreshRSS can fetch full article contents for truncated RSS feeds is precisely the thing you need in order to make RSS useful today. Not the biggest fan of the UI but this one feature makes the whole thing so fucking good.

thanatropism
Yay, I'm listed!
[deleted]
(comment deleted)
[deleted]
(comment deleted)
luxpir
Just a thanks for this. Wanted something like this from that list but didn't have time.

Loaded it into Feeder from fdroid (cheers to dev if here) and although it took a while, the result is much better than Twitter et al's output.

Might add my own blog to the list one of these days if people keep coming back to this as the reference.