Which lets you set filters so that you only see posts with a certain amount of activity, so you aren't spammed with every single HN submission in your feed.
hnrss is great. This is what most news sites get wrong, they might have RSS, but it gives you every new article and that can be over a hundred per day. And then they eventually conclude that no uses it and thus RSS is bad, when it's just their thresholdless implementation that's bad.
I'm so happy websites have kept RSS around, because even if it's mostly used for machine-to-machine feeds nowadays, we obviously piggyback on top of it and benefit. It could've been so easy for big sites to replace it with proprietary APIs for GitHub, YT, etc...
I provide an RSS feed for my blog. From my access.log, I see that there is a decent number of subscribers to my feed. Here is the data I could pull from it:
Feedly 91 subscribers
Inoreader 16 subscribers
Feedbin 6 subscribers
NewsBlur 4 subscribers
The Old Reader 3 subscribers
BazQux 2 subscribers
WordPress 2 subscribers
-------------------------------
Total 124 subscribers
I get about 4000-6000 hits to my website on a normal day. It increases by 5 to 10 times if a post hits the HN front page. When I publish new post, I see from the HTTP referer information in the logs that about 30 or so hits come from users who find a link to my post in their feed reader. These are very small numbers but they are good enough to keep the feed alive.
I came here to say exactly this: my numbers are similar to yours (and from headers only, like you[1]), and a decent proportion of my views come from feed clients. My blog generator takes care of the feed for me; I haven't thought about managing it in years.
I don't write very often but when I do it is usually tech or math. I didn't link to it here because it might look like self-promotion. In case you are interested to take a look, please see my profile for the link.
In fact, Feedly was the only large centralized feed reader that didn't do subscriber count reporting via UA string for the longest time. They made you do a GET request on some API endpoint if you wanted to get your subscriber count.
They implemented it some time back, but it's still kind of buggy - at least for my blog. Two or three different IPs periodically do a GET on my feed URL with the Feedly UA and they all report different subscriber counts.
Something I'd be curious about: how many of the hits on a new post are by RSS readers before the post hits HN vs. after? I know RSS is the most reliable way for me to notice a new post on blogs that update infrequently, and I wonder how many HN submissions come from people seeing a new post in their feed readers.
I have about the same distribution of RSS clients on my blog as well. Feedly seems to be the largest, with about 10x reported subscribers than their competitors. Newsblur, The Old Reader and Inoreader are all about the same, but far behind. Then there's a sea of various self-hosted readers like the TinyTinyRSS and desktop programs.
Note that these are all self-reported numbers from the user agent string - as far as I can tell, Feedly might be inflating their numbers to give an impression of a large user base and to appear on top of lists like this.
It is unfortunate that it's so tough to track RSS viewers. I'm always tempted to convert the feed to cut-back version just so I can see what posts are actually popular.
An idea I had is to just embed a pixel into the item body. You can self host it too: just analyze the apache logs (or whatever you have) for requests to that pixel.
Also love fraidycat, the approach it takes for keeping things “fair” is really nice.
It solves one of the problems I’ve had with other RSS readers around the overwhelm when adding a new source/neglecting an existing source. Opening up to see tens/hundreds of items in the backlog that you feel obligated to read isn’t fun.
I’ve got it set as my new tab page, which makes it easy to see new stuff at a glance, and then I can delve into the less important stuff when I’ve got time to waste.
I'm happily using Feedly ever since Google Reader got shut down. I genuinely would be missing out on a lot of content if people would stop providing RSS feeds, especially for blogs that only update very sporadically.
Seconded. I've been a very happy user of Feedly for years.
If someone posts an interesting, valuable article, what's the probability they'll post more? Very high! I'm happy to subscribe to their content and enjoy their insights!
Feedly is best of both worlds - you can use it as a plain linear feed reader, or let it use AI to sort your stuff by popularity and even discover feeds by keywords. Its price is on the steeper end tho, if you want to go pro.
>I genuinely would be missing out on a lot of content if people would stop providing RSS feeds, especially for blogs that only update very sporadically.
For me, that's the real value of RSS even if, truth be told, I'm more inclined to just find stuff through Twitter, here, etc. these days. I agree on Feedly. For all the hate on Google shutting Reader down, I've always seen it as an effect of not many people using RSS rather than a cause.
(I'm also sympathetic to the srsly Google? How much effort would it have taken to keep it going? argument. But, as has frequently been mentioned, working on Reader would probably have been career suicide or at least a guarantee of stagnation for anyone involved given the way Google operates.)
I might be wrong but my impression is that feedly exists precisely because google closed its google reader service and offered to export your subscriptions to feedly. It would be majorly funny if they bought it "back".
I am using inoreader.com and never looked back. Even the free version is great. I use the paid version and it gives me all what I need. I can only recommend it.
I pay for Feedly not because I'm interested in any of the Pro features, but because Google Reader taught me that you should pay for things you like so they will stick around.
Although your statement is generally true, I think the remaining flaw in that consideration is the subscription model. It makes you and your information source totally dependent on the whims of others. Sure, maybe they won't shut down, but maybe they will not refuse some insane buyout from a tech tyrant to capture the market and you and the information about you, if they aren't already selling that.
The general internet/application model of pre-fb is probably the only viable option going forward, where you pay or invest or buy into development projects like you would in one of those gofundme type things.
Sure. It's the old self-hosted vs cloud problem. I don't have the time or interest to host my own services, so I pay someone else to do it. That comes with its own risks, like you say, but I think the tradeoff is worth it. Having a functioning business model that isn't based on abusing their own users provides some insurance against those problems, I think.
I personally went the self-hosting route using FreshRSS and quite enjoy it. I couldn't really justify the monthly Feedly costs when I had server resources sitting around being wasted.
To me, one of the most important features of RSS is not having to reskim over titles I've already dismissed. I don't even mind that most feeds don't have any content, the links are good enough for my needs.
I really wish AP and Reuters would bring back RSS, I'd no longer have any reason to open a site just to skim headlines. Some might argue that is the reason news sites stopped offering RSS, but I don't think it really lowers my engagement with a site, I very often click around after opening an item that caught my attention.
Relying on the underlying HTML attributes of the page to generate the feed does make these feeds much more brittle, but it's one option when the sites choose to switch off their own feeds.
I still use rss too (switched to using a self-hosted tt-rss after trying out a ton of Google Reader alternatives). I love RSS and mourn the fact that it's not more popular. The main problem with RSS becoming so niche is that lots of blogs/etc. no longer support RSS (I guess I need to look into one of those mailing list -> RSS things).
If the website is still accessible, it's possible to generate an RSS feed using the main page URL. I work on a project called Feed Creator that does this. You give it the page URL and the item selectors: http://createfeed.fivefilters.org/
The most popular newsletter services do make them, though -- blah.substack.com/feed , for instance (don't remember Revue's or Buttondown's off the top of my head)
"Having only the content I want to see only be shown when I want to see it with the freedom to jump between readers as I please, all with no ads? For me, no other service comes close to the flexibility, robustness, and overall ease-of-use that RSS offers."
I always wonder why they aren't as popular now. It takes more work to add site to a reader, and more effort to even use one I suppose. And maybe it's also missing the social aspect that people seem to love?
I assume they'll be popular again some day like vinyl and static web sites.
Long-term Google trends graphs are kind of useless because they're a percentage of queries over a period of time when the demographics of the user base was changing as millions of new people got on the internet.
Clearly javascript has been slowly dying out. But wait, that can't be right.
They're also a poor proxy for usage because the existing user base with feeds already set up doesn't have to do a search query to continue using it, so if a large number of them lose access at once, it's barely a blip because they wouldn't be doing searches for it either way.
You probably didn't want to come across this way, but we should note that the absence of a better metric is not by itself a reason to use a bad one.
If a metric is sufficiently decorrelated with the thing we want to measure, it should be ignored altogether, regardless of whether we have anything better.
As far as I can tell, because RSS requires the cooperation of actors who have no incentive to cooperate. If I use an RSS reader to support content, then it is under my control when I read it, if I read it, and what format to read it under. I can view the content however I'd like, and no tracking is there.
All of these are unambiguously good things for me, and are anathema for the advertising industry. I cannot use RSS to read a site unless that site provides an RSS feed. Sites whose primary goal is to bring in ad revenue, rather than building a community, have no incentive to provide RSS feeds.
I'm perfectly happy just getting headlines with links, which seems most common these days, at least for what I subscribe to. It requires I go to their site to get the content where they can (try to) shove anything they want in my face and run whatever trackers they normally do.
This why I personally gave up RSS. I used to enjoy reading things in my RSS reader where I could set things up to be clear to me.
Once they started providing only links back to the WWW browser where everything was hard to read I gave up and now I just suffer reading in the WWW browser and frankly don't read nearly half as much online content as a result.
To me, the time saved not having to rescan headlines alone is worth it. Even if the title was just "New Article" for every item and I had to open each one to see what it was, the time saved would still probably be worth it.
So I take the opposite view of preferring to read on the actual pages.
I'm curious though, I've built a feed reader that caters to my preference, but I'd like to make it useful to more folks. If you could automatically open articles in "Reader Mode" (FireFox supports this for instance) would that make reading in the browser more palatable to you?
I understand that perspective. And I agree that it would be nice if all sites offered the full-text content in the feed. But on the flip-side, I still use RSS feed readers (Feedly) daily and simply click through the content.
The value of the RSS feed reader is still there because it allows you to monitor potentially hundreds or thousands of sites for content and then skim through to see if any new content was created today that interests you.
This makes it particularly useful for monitoring personal blogs or sites that only publish a few times per month or year. I personally LOVE reading people's personal blogs, but since most people only publish a post a few times a month (or less), I am unlikely to check their website every day. I also don't want to give them my email address and have them clutter my inbox with new posts. So the feed reader is nice because I can go once a day and see what new content exists. I skim through the posts and click through on a handful of content that interests me. If the content is long, I add it to my "Pocket" (getpocket.com) to read later.
In the feed reader I usually get a preview of their content (the first few paragraphs). I can read the content and click through if desired. So the feed reader still provides value for monitoring all these sites that have sporadic posting schedules.
I understand the value of reading the content entirely in the feed reader, but I don't mind clicking through the content once I know it is something that interests me. The feed reader allows me to skim, filter, and screen content so that I am only clicking through on interesting content (similar to how HN works, we still have to click through content). If content is interesting, there is value in supporting creators by visiting them on their site to get the experience that they designed and potentially supporting them with an ad or two. If ad use is egregious then I might retaliate by blocking ads for their site or simply removing them from my feeds.
Absolutely the same, and that's how I usually use RSS feed a anyways, as a source of headlines. Even there, though, it means that my choice of which articles to read isn't influenced by their snazzy new (and slow) layouts, their eye-catching (and emotionally jarring) images. It means that my daily habit is to check my RSS reader, rather than refreshing their site to see if anything has been posted.
With a limited amount of attention, even just exposing the headlines through RSS means that it is more my choice where to distribute my attention, rather than having my attention be drawn elsewhere through dark patterns and marketing tricks.
Yup, that is exactly the sense that I mean. Ad-supported web sites have an incentive to have information viewable only when viewed alongside advertisements.
> Having only the content I want to see only be shown when I want to see it
I would rather not do my own content discovery and content curation: I see those as literal chores that have to be done before I can consume content.
Like all chores, I'm not keen to do them. Despite years of practice, I'm still not confident I am great at them on my own, and I have the option to employ someone more skilled.
Twitter has done a good job of these chores, making discovery and curation easier (like the parent article, I feel frustrated that Twitter is now broken from this point of view); TikTok does a great job, surely better than I could do myself.
"I always wonder why they aren't as popular now. It takes more work to add site to a reader, and more effort to even use one I suppose"
I know for me, I didn't know what RSS truly was until like.. last year? I remember seeing the icon and all that on various sites growing up but I never clicked them as I didn't know it was a straight feed of content.
Whereas last year I read some article or blog about it dying and I decided to look it up. Now it's the only way I read or watch any content becuase it's exactly what I want: straight content with no bs.
A big one, that was the case even before social media, is the inherent geekiness of it. "RSS" is an acronym after a geek's heart, most people don't like it.
The subscription process is complicated. Go to the site, find the feed (which one? There might be a bunch! eg, main feed, topic feed, comments-per-post feed, etc), take that URL and paste it into your feed reader, then look at that and... you got linked back to the site you were on? Yay... There have been various attempts to streamline this via in-page "add to reader" widgets back in the day, browser plugins, etc. But it was never as frictionless as the Facebook/Twitter/Reddit "follow" button.
Responding to things is harder. There's no comment/like/whatever button right there, you have to go to the site, figure out what's going on there, maybe sign up for yet another damn account if you want to comment. Some sites don't even have comment sections! Or maybe they point you to FB or Twitter anyway. Again there were partial solutions back in the day, but nothing as easy as modern social media.
Last for today, it's not nearly as addictive. Some people (including the author and myself) like that about it... but it doesn't drive adoption nearly as well.
>I assume they'll be popular again some day like vinyl and static web sites.
I think they'll always be niche-popular with geeks, long-form content creators, open web advocates (the "EFF crowd" I guess) and so forth. But I don't think it'll ever be mainstream. At best it could serve as the substrate for something else that becomes mainstream.
Anyway, just my 2 cents from hanging around this space for a decade+.
> A big one, that was the case even before social media, is the inherent geekiness of it. "RSS" is an acronym after a geek's heart, most people don't like it.
Like WWW, RCA, GE.. (the latter two of the most recognizable and successful brands of all time). got it. (And no one cares what an acronym stands for anyway.. you don’t need to know this to make use of it, it has no bearing. RSS is pretty catchy as far as branding goes.)
The subscription process is simplicity itself and there are still some tremendously mainstream popular and well designed for a general audience readers like Feedly. I’ve almost never had to manually cut and paste a feed link (I think the cases were dealing with someone’s broken Linux desktop). Clicking on a link on an iPhone will open it in a reader or take you the App Store to install one.
RSS is simply missing the key feature of social networking “engagement” and these walled gardens are heavily promoted and in your face. It’s not a geekiness issue it’s an attention issue.
RSS as branding - well, I've tried to sell people on it for years. I actually worked at an RSS aggregator startup for 7 years, so I spent a while pushing it. You say "RSS" to most people and their eyes glaze over.
>The subscription process is simplicity itself
You must've found some different sites than I have, because even sites that have a feed don't make it obvious how. For instance, check out Substack, ex: https://steady.substack.com/p/america-the-beautiful
Can I get an RSS feed, and if so how? The giant "Subscribe" button isn't it. So I guess grab the page URL and paste it into Feedly and see what I get. I don't mind this (much), but "mass market" it ain't. And this kind of process is very common in my experience. Back in the day sites would have the little "feed" icon sometimes and that helped, but it was still a 2-3 step process.
Open this on an Apple or android phone. On iOS this will even prompt you to open the App Store to find a feed reader (#1 is still Feedly).
RSS icons were ubiquitous like fb and Twitter icons a few years ago, and anyone could use them. It wasn’t a geek thing, it’s just a market that faded.
I’ve never had the feed icon not work on a phone in the last 10 years or a typically installed browser on Mac or Windows.
I mean this is an absurd conversation. RSS was invented around 1998 and then it was an obscure “geeky” thing.. the only reason we’re having this tedious conversation (and arguably why you’d have had the opportunity to be involved in an RSS startup) is because it had a striking rise from obscurity in the late oughts.
Lol, my bad, you're right. It was wildly popular, random people on the street were talking about RSS, everyone in the general non-blogging populace was swapping their favorite feeds, it was amazing. That's probably why Google shut down Reader right at the height of that amazing, broad-based popularity and basically nobody talks about it anymore.
Even if RSS was popular, how could advertising companies like Google and Facebook make money off of it?
Those companies have no incentive that I know of to give their user a reliable (a.k.a non-personalised) feed that let them chose whether they'll get to click through or not. No, they want you to click through first, so you get to see the ads before you even start doom scrolling.
Google had a reader app, in fact the market-dominant reader app. They could sell advertising next to it, like they do with everything else (probably did, I'm not sure). They also owned a publishing platform (Blogger) and an RSS metrics platform (FeedBurner) that had paid tiers. So they actually did monetize it. Just not enough to bother with.
Facebook for a while was trying to be the commenting system for blogs. Which is not exactly RSS, but they were in an adjacent space and using it to drive traffic.
And yet it still died, or rather has slowly faded away back to the niche market that I originally mentioned.
Unless I screwed something up, it used to be the standard way to tell browsers where the RSS feed for this particular web page are. Back then it was very easy to use, even for Grandma™: just click on the standard RSS icon in your browser, next to the URL bar, and there's your feed. That was your subscribe button.
Then browsers stopped working. No more RSS icon, at least by default. I had to add an explicit <a/> link at the bottom of the page, which when cliked on shows an ugly rendition of plaintext XML.
Same. I actually built a personal Python app that turns RSS feeds into a daily newsletter when I quit most social media. It's been fantastic, in part because it's NOT infinite. It typically gives me something between 1-10 links and... that's it.
There's nothing to "check" and it's not watered down with provocative crap that an algorithm thinks would make me stick around longer.
What I particularly like about rss feeds, is that I can keep up with anything whatsoever, It is perfect for any serialized content, what I find important is not the notifications, that is just a side effect. But about choosing what to stay up to date with. you have control over the content you consume, when you want it. The same way you'd use a bookmark in books.
I've also recently switch my rss feed client into just my email client, installing rss2email, throwing all the content to email, means I can use any phone whatsoever and have exactly the same content.
I second that. If you have a lot of feeds, thousands, then QuiteRSS is one of the readers that will work reliably. It can be ram heavy with 1k+ feeds but it does the job.
In the article the author mentions one way to get RSS feeds from a youtube channel. This is how to get it from a youtube username,
Do you know if it's possible to make QuiteRSS automatically download and cache the pages of new posts it finds? It seems to wait until I click on a post before starting to load it (quite slowly).
my problem with quiterss is that it takes ages to save data after close. idn why but if you loose power in between next time either you have to update all the feeds or the state which were you in last time.
I personally find these articles amusing because I'm still wondering whether I should start using RSS. I've always kept in the back of my mind over the years that I really should check out which RSS reader is the best so I have it ready when I need one.
Youtube RSS still works well, me and my buddy use it to monitor things instead of using whatever interface youtube has decided today, combine that with youtube-dl and you have a great experience.
I use emacs elfeed for my feeds, however I found it frustrating that my different feed categories - news, views, tech, podcasts, etc - all get jumbled up out of the box into one massive rss firehose.
Eventually I realised I could simply use bookmark.el to quickly create filtered search versions of my feeds based on their respective tags, and now I'm much happier. After setting it up, a `C-x r l` (bookmark-menu-list) later and I get a list that looks like this
I could almost imagine running a small service that presented the contents of an IMAP folder as a feed, but I can't imagine actually using such a thing!
Feedbin[1], my feed reader of choice, has that. They give you a random email address that you can use to receive newsletters in your feed and automatically have them tagged.
I've mentioned this on HN before, one pattern I've noticed is a lot of sites tend to truncate their RSS feed so you have to click through to see the full article etc. This is mostly OK but kinda frustrating, as in the before-covid times I used to ride the London Underground a lot where no celluar signal is available in the tunnels - so my offline RSS reader would be useless.
So I wrote a little web service accepts an RSS feed, and then crawls each link in it, parses the HTML to extract the text using one of those text extraction libraries, and then "republishes" a new version of the feed (with the full article) which I then subscribe to in my RSS reader so I can read the articles offline, e.g.
example.com/rss?feed=foo.net/artcles/rss
I've never open sourced it though because I guess it's a bit of a grey area - the sites want you to go to the full URL so they can show you ads etc
Oh that's fantastic! I've been meaning to look into providing a Docker version at some point, will definitely look more at how you've done things here.
clicking through is what killed it for me too. I just never went through the trouble of setting up crawlers and stuff. Looks like open source is to the recue nowadays. I am pretty sure there was nothing as easy as that back in the day.
Thanks for sharing this!
I have used both of those methods when publishing... I do hesitate on the full article approach, but I don't run ads. (And to be clear, I do provide a whole-article feed)
One thing I don't like about the whole article RSS is that I get requests to narrate my whole website or blog experience, in addition to the normal writing about stuff.
In effect the full-article RSS audience has its own experiential needs, which include "tell me about what you added to the non-blog sections, tell me about the new tools in the sidebar, or the new article footer format, or even hey I never heard about the new design," and so on. Sometimes I also alter the CSS for a single post, when it makes sense.
Some of this narration is just expected if your main format is traditional, plain blog. But personally my blogs tend to evolve and incorporate other stuff. And for sites with even more of an experiential aspect than my own, I can see why RSS could drive the publishers a bit mad, increased exposure or no.
Because of this experience, personally I would rather try reading via a page-download tool or format, even automated w3m or something, and see if that's an improvement. You'd get the full article in plain text, but you'd also get some of the other stuff around the site, including links which you may even be able to mark for later download.
Newsblur has a setting per-site which allows you to have it use either the feed body or attempt to pull in the full text. When I was commuting on the subway (prior to ubiquitous underground coverage) here in DC, I used that heavily. One nice thing is that downloading it on your client avoids the site having to get in the middle of the copyright concerns about replicating their servers.
nice! I've been using newsblur for a few years now but I've never delved into the settings. is this option only available on the web version? i can't see anything on the android app
I still heavily use RSS, but usually unsubscribe from feeds that truncate (unless it is a very long article).
It isn't a religious thing - the vast majority of the time, the truncation is way too short to properly advertise the content. The mystery doesn't move me to click-through, quite the opposite - it is annoying, so I remove the annoyance.
I understand the tradeoffs for folks who want to be paid via ads, but the utility just isn't there for me - I do not lack for reading options, and if something is really good, it'll find me some other way.
I personally truncate the RSS feed for my site, because it's mostly interactive experiences, and I would rather them experience the interaction. I don't have ads on my site though. I've been trying to come up with a good middle ground though.
I use Feedbin, and read Feedbin on my Mac and iPhone using the Reeder app. Both are able to extract the full text (automatically on a per site basis if wanted) of most sites without issues.
I guess they just follow the URL, extract the body text the same way that sites like Instapaper do it, and give that back.
This exact problem applies to podcast analytics. "Downloads" is only a proxy for "listens" / ad impressions.
It seems to me like there needs to be a standard pingback endpoint (either in /.well-known or, better, right in the podcast/rss XML) that your client pings when it seems like you've consumed the content (scrolled through most of a web page, listened to 80+% of a podcast), but only if you've opted in to that.
Basically, for feeds where I have "full text" enabled, it fetches the article page with mozilla's "reader mode" implementation and extracts the full text.
It isn't really usable with aggregators such as planet kde/gnome, though.
> I've never open sourced it though because I guess it's a bit of a grey area
For whatever it's worth, I do click-through RSS feeds on my blogs just because I was too lazy to set up full text feeds and figure out how stuff like video would work and whether or not it would be a better end-user experience to alter the content in some way on style-heavy pages. Right now I'm constructing the feed myself. But if someone ever wanted to use a tool like you're describing on my blogs, I would have no objection. Don't feel guilty about scraping a site I build or building tools to do so, if anything it would make me feel less guilty about never getting around to setting up full-text RSS feeds. :)
I don't run ads or analytics anyway, but even if I did, as far as I'm concerned anyone who has the right to visit a page also has the right to download it, and to use automated tools to download it on their behalf[0].
I don't see any moral difference between using an adblocker and scraping a web page. Both are hitting the same endpoints and selectively displaying content that the user wants to see.
I don't think it's a grey area. They could show you ads in the HTML, they choose not to, and instead choose to let outside companies execute arbitrary code on your machine to track you. "Sites want you to go to the full URL so someone else can track your web use" sounds way less innocent and, to me, downright hostile. No one should feel morally obligated to allow that to happen.
One of the reasons I went from placing a full article in my RSS feed to truncating it into a descriptor and leaving the link as the "source of truth" was I found that there a bunch of intentionally crippled readers, like Thunderbird, that strip all the markup and turn my article into an unreadable mess.
Traffic to both the feed and to the main site increased noticeably when I reduced it to a descriptor.
This would largely have saved RSS for me. I'd argue there was a glorious era in late 2000s where you could almost replace your browser with a good feed reader and get vast majority of news/info this way. It was just so nice to have a single UI surface for all news stories rather than poking through many different websites.
I understand my using the sites that way contributes nothing really to their bottom line, I never saw any ads, but that of course was half the appeal. The early days of "web2.0" people often just naively dumped everything in the RSS feed.
I still find feeds from truncated useful as I can scan through the headlines quickly; I rarely want to read everything a publisher puts out there. When something catches my eye my RSS viewer, Feedbin, can attempt to fetch the entire one for me and if that fails visiting the site directly is tolerable.
A lot of web pages are just completely unusable or unreadable now.
It uses all sorts of manipulations to auto play video ads, pop ups, picture unders, user tracking, page interaction telemetry, etc.
All kinds of nonsense to display an article to you, which if it really is important, then another writer would have also written about it.
Some websites I just avoid completely, when I see the url.
These websites should just go with a pay-to-read subscription model instead. If their junk is interesting enough, then they’ll find enough people willing to pay a few cents to read it.
> I've never open sourced it though because I guess it's a bit of a grey area - the sites want you to go to the full URL so they can show you ads etc
I honestly would not mind much if sites just included ads in their .RSS feed? Some bloggers I follow have their rss 'sponsored' and will do a sponsor post once a week. If ad insertion is really so important a site could just have labeled ads as part of the content stream. It's not ideal, but I'd prefer it to a feed that only has 2 or 3 teaser sentences and then forces me to click through.
I actually took Wired and Slate off my RSS feed just because I got annoyed at how often I would click through and get a paywall. Usually I was using up my free articles just to see what the article was about without reading it.
That's an interesting use case. My POV has always been that the site itself is the best representation of the content and completely ignore the feed content.
For me it also depends on the content. Some sites I'd rather just read from in the feedreader such as a news site, but for things like YouTube videos I would rather watch on YouTube than watch via the embedded player.
However I find news sites are the most common offenders of the truncating which I guess I can understand.
At the end of the day if it's a openly available website and you are personally (through your own server) fetching the resources I don't think anyone has a right to complain.
Now if you were offering it as a service it might arguably be a bit more grey, but only if you're ignoring the robots.txt file
I think this kind of service is actually harmful to content providers. They can not benefit from the content they created if readers always consume it on the RSS readers. That's one of the reasons many blogger/website don't provide RSS feeds anymore.
The RSS aggregators should encourage readers to go to the original website. So when I got the chance to build my own RSS reader(https://feeds.pub) I only shows a title and link of an article to readers to drive them to the original website.
Do you allow for a feeds list format? In other words, can I just see a list of feeds and folders, and when I click on the feed, it will display a list of articles from that feed? Or is it like Twitter, where everything is squished together.
> a lot of sites tend to truncate their RSS feed so you have to click through to see the full article
This is unacceptable to me, and if a site does this I unsubscribe immediately.
Great tool you have! I was considering building something like this (but directly in a feed reader), by any chance do you intend to release the source for it so that people can run it themselves?
I thought the whole point of RSS was as an index to new content. It never occurred to me to put the whole article in the feed... surely a good summary and a link gives the consumer the choice to load the article or not, rather than having to load all the articles all the time?
I am not sure if there are many doing something similar to mine. My pattern is to Open all my interested Links in New Tabs. That is why I thought RSS Reader should always be within a Browser or as Web Services like Feedly.
I do offline Reading for long articles only. And Saved via Reading List on Safari that does automatic downloading. But it is quite problematic for me because most of the time I need to look up something related while I am reading on things that I dont understand or assumed wrongly. And this is still best done on a PC with Browser.
So I now only listen to Music, Reading Novel or WebComic / Manga for offline.
I used to live in London (8 years ago) and I did exactly this because of the Jubilee line. I had written a perl program rss.pl that I would give several URLs to generate custom RSS feeds for those websites that provided poor, summary only, feeds. I had it running in a crontab and just posted the feeds to my own webserver. This brings back memories.
I live in Tokyo now where there is internet access everywhere so I retired my perl program.
Out of curiosity, did you manage to implement auto crawler that can always find the correct html element for any site and remove rest? Or did you have to create per-site rules for it?
369 comments
[ 2.4 ms ] story [ 60.2 ms ] threadhttps://news.ycombinator.com/rss
https://hnrss.github.io/
Which lets you set filters so that you only see posts with a certain amount of activity, so you aren't spammed with every single HN submission in your feed.
[1]: https://www.yossarian.net/snippets#vbnla
They implemented it some time back, but it's still kind of buggy - at least for my blog. Two or three different IPs periodically do a GET on my feed URL with the Feedly UA and they all report different subscriber counts.
Note that these are all self-reported numbers from the user agent string - as far as I can tell, Feedly might be inflating their numbers to give an impression of a large user base and to appear on top of lists like this.
While https://rssbox.herokuapp.com/ is great, I wish more websites allowed RSS rather than resorting to hacky workarounds
It solves one of the problems I’ve had with other RSS readers around the overwhelm when adding a new source/neglecting an existing source. Opening up to see tens/hundreds of items in the backlog that you feel obligated to read isn’t fun.
I’ve got it set as my new tab page, which makes it easy to see new stuff at a glance, and then I can delve into the less important stuff when I’ve got time to waste.
If someone posts an interesting, valuable article, what's the probability they'll post more? Very high! I'm happy to subscribe to their content and enjoy their insights!
Related: I now subscribe to Hacker News users via http://hnapp.com/
Example: Jeff Geerling http://hnapp.com/?q=author%3Ageerlingguy
For me, that's the real value of RSS even if, truth be told, I'm more inclined to just find stuff through Twitter, here, etc. these days. I agree on Feedly. For all the hate on Google shutting Reader down, I've always seen it as an effect of not many people using RSS rather than a cause.
(I'm also sympathetic to the srsly Google? How much effort would it have taken to keep it going? argument. But, as has frequently been mentioned, working on Reader would probably have been career suicide or at least a guarantee of stagnation for anyone involved given the way Google operates.)
The general internet/application model of pre-fb is probably the only viable option going forward, where you pay or invest or buy into development projects like you would in one of those gofundme type things.
and about 170 other sites, at last count.
I really wish AP and Reuters would bring back RSS, I'd no longer have any reason to open a site just to skim headlines. Some might argue that is the reason news sites stopped offering RSS, but I don't think it really lowers my engagement with a site, I very often click around after opening an item that caught my attention.
* Reuters: https://createfeed.fivefilters.org/index.php?url=https%3A%2F...
* AP: https://createfeed.fivefilters.org/index.php?url=https%3A%2F...
Relying on the underlying HTML attributes of the page to generate the feed does make these feeds much more brittle, but it's one option when the sites choose to switch off their own feeds.
"Having only the content I want to see only be shown when I want to see it with the freedom to jump between readers as I please, all with no ads? For me, no other service comes close to the flexibility, robustness, and overall ease-of-use that RSS offers."
I always wonder why they aren't as popular now. It takes more work to add site to a reader, and more effort to even use one I suppose. And maybe it's also missing the social aspect that people seem to love?
I assume they'll be popular again some day like vinyl and static web sites.
1) Google makes really quite good RSS reader (Google Reader)
2) Everybody switches to it, competition atrophies
3) Google shuts it down
They shut it down back in March of 2013. If you look at the trends graph[1] for RSS it was dying out for years.
[1] https://trends.google.com/trends/explore?date=all&q=%2Fm%2F0...
Here's the one for javascript:
https://trends.google.com/trends/explore?date=all&q=javascri...
Clearly javascript has been slowly dying out. But wait, that can't be right.
They're also a poor proxy for usage because the existing user base with feeds already set up doesn't have to do a search query to continue using it, so if a large number of them lose access at once, it's barely a blip because they wouldn't be doing searches for it either way.
If a metric is sufficiently decorrelated with the thing we want to measure, it should be ignored altogether, regardless of whether we have anything better.
All of these are unambiguously good things for me, and are anathema for the advertising industry. I cannot use RSS to read a site unless that site provides an RSS feed. Sites whose primary goal is to bring in ad revenue, rather than building a community, have no incentive to provide RSS feeds.
Once they started providing only links back to the WWW browser where everything was hard to read I gave up and now I just suffer reading in the WWW browser and frankly don't read nearly half as much online content as a result.
I'm curious though, I've built a feed reader that caters to my preference, but I'd like to make it useful to more folks. If you could automatically open articles in "Reader Mode" (FireFox supports this for instance) would that make reading in the browser more palatable to you?
The value of the RSS feed reader is still there because it allows you to monitor potentially hundreds or thousands of sites for content and then skim through to see if any new content was created today that interests you.
This makes it particularly useful for monitoring personal blogs or sites that only publish a few times per month or year. I personally LOVE reading people's personal blogs, but since most people only publish a post a few times a month (or less), I am unlikely to check their website every day. I also don't want to give them my email address and have them clutter my inbox with new posts. So the feed reader is nice because I can go once a day and see what new content exists. I skim through the posts and click through on a handful of content that interests me. If the content is long, I add it to my "Pocket" (getpocket.com) to read later.
In the feed reader I usually get a preview of their content (the first few paragraphs). I can read the content and click through if desired. So the feed reader still provides value for monitoring all these sites that have sporadic posting schedules.
I understand the value of reading the content entirely in the feed reader, but I don't mind clicking through the content once I know it is something that interests me. The feed reader allows me to skim, filter, and screen content so that I am only clicking through on interesting content (similar to how HN works, we still have to click through content). If content is interesting, there is value in supporting creators by visiting them on their site to get the experience that they designed and potentially supporting them with an ad or two. If ad use is egregious then I might retaliate by blocking ads for their site or simply removing them from my feeds.
With a limited amount of attention, even just exposing the headlines through RSS means that it is more my choice where to distribute my attention, rather than having my attention be drawn elsewhere through dark patterns and marketing tricks.
Do you mean in the sense that content providers want to dictate more than just information delivery to consumers, or something else?
I would rather not do my own content discovery and content curation: I see those as literal chores that have to be done before I can consume content.
Like all chores, I'm not keen to do them. Despite years of practice, I'm still not confident I am great at them on my own, and I have the option to employ someone more skilled.
Twitter has done a good job of these chores, making discovery and curation easier (like the parent article, I feel frustrated that Twitter is now broken from this point of view); TikTok does a great job, surely better than I could do myself.
I know for me, I didn't know what RSS truly was until like.. last year? I remember seeing the icon and all that on various sites growing up but I never clicked them as I didn't know it was a straight feed of content.
Whereas last year I read some article or blog about it dying and I decided to look it up. Now it's the only way I read or watch any content becuase it's exactly what I want: straight content with no bs.
IMO it's a lot of reasons.
A big one, that was the case even before social media, is the inherent geekiness of it. "RSS" is an acronym after a geek's heart, most people don't like it.
The subscription process is complicated. Go to the site, find the feed (which one? There might be a bunch! eg, main feed, topic feed, comments-per-post feed, etc), take that URL and paste it into your feed reader, then look at that and... you got linked back to the site you were on? Yay... There have been various attempts to streamline this via in-page "add to reader" widgets back in the day, browser plugins, etc. But it was never as frictionless as the Facebook/Twitter/Reddit "follow" button.
Responding to things is harder. There's no comment/like/whatever button right there, you have to go to the site, figure out what's going on there, maybe sign up for yet another damn account if you want to comment. Some sites don't even have comment sections! Or maybe they point you to FB or Twitter anyway. Again there were partial solutions back in the day, but nothing as easy as modern social media.
Last for today, it's not nearly as addictive. Some people (including the author and myself) like that about it... but it doesn't drive adoption nearly as well.
>I assume they'll be popular again some day like vinyl and static web sites.
I think they'll always be niche-popular with geeks, long-form content creators, open web advocates (the "EFF crowd" I guess) and so forth. But I don't think it'll ever be mainstream. At best it could serve as the substrate for something else that becomes mainstream.
Anyway, just my 2 cents from hanging around this space for a decade+.
Like WWW, RCA, GE.. (the latter two of the most recognizable and successful brands of all time). got it. (And no one cares what an acronym stands for anyway.. you don’t need to know this to make use of it, it has no bearing. RSS is pretty catchy as far as branding goes.)
The subscription process is simplicity itself and there are still some tremendously mainstream popular and well designed for a general audience readers like Feedly. I’ve almost never had to manually cut and paste a feed link (I think the cases were dealing with someone’s broken Linux desktop). Clicking on a link on an iPhone will open it in a reader or take you the App Store to install one.
RSS is simply missing the key feature of social networking “engagement” and these walled gardens are heavily promoted and in your face. It’s not a geekiness issue it’s an attention issue.
>The subscription process is simplicity itself
You must've found some different sites than I have, because even sites that have a feed don't make it obvious how. For instance, check out Substack, ex: https://steady.substack.com/p/america-the-beautiful
Can I get an RSS feed, and if so how? The giant "Subscribe" button isn't it. So I guess grab the page URL and paste it into Feedly and see what I get. I don't mind this (much), but "mass market" it ain't. And this kind of process is very common in my experience. Back in the day sites would have the little "feed" icon sometimes and that helped, but it was still a 2-3 step process.
Open this on an Apple or android phone. On iOS this will even prompt you to open the App Store to find a feed reader (#1 is still Feedly).
RSS icons were ubiquitous like fb and Twitter icons a few years ago, and anyone could use them. It wasn’t a geek thing, it’s just a market that faded.
I’ve never had the feed icon not work on a phone in the last 10 years or a typically installed browser on Mac or Windows.
I mean this is an absurd conversation. RSS was invented around 1998 and then it was an obscure “geeky” thing.. the only reason we’re having this tedious conversation (and arguably why you’d have had the opportunity to be involved in an RSS startup) is because it had a striking rise from obscurity in the late oughts.
Those companies have no incentive that I know of to give their user a reliable (a.k.a non-personalised) feed that let them chose whether they'll get to click through or not. No, they want you to click through first, so you get to see the ads before you even start doom scrolling.
Facebook for a while was trying to be the commenting system for blogs. Which is not exactly RSS, but they were in an adjacent space and using it to drive traffic.
And yet it still died, or rather has slowly faded away back to the niche market that I originally mentioned.
It was, even Firefox users were using it. That's probably why Mozilla had to remove that functionality. /s
Then browsers stopped working. No more RSS icon, at least by default. I had to add an explicit <a/> link at the bottom of the page, which when cliked on shows an ugly rendition of plaintext XML.
There's nothing to "check" and it's not watered down with provocative crap that an algorithm thinks would make me stick around longer.
I've also recently switch my rss feed client into just my email client, installing rss2email, throwing all the content to email, means I can use any phone whatsoever and have exactly the same content.
https://quiterss.org/
No affiliation, just a satisfied user.
In the article the author mentions one way to get RSS feeds from a youtube channel. This is how to get it from a youtube username,
waiting this to be fixed.
Eventually I realised I could simply use bookmark.el to quickly create filtered search versions of my feeds based on their respective tags, and now I'm much happier. After setting it up, a `C-x r l` (bookmark-menu-list) later and I get a list that looks like this
Bookmark.el was written in 1993 by a Karl Fogel in 1993, and it still interoperates nicely with a random newer package like elfeed.There are a few different tools for getting the feeds to email, my own is a pretty simple golang application I run in a docker-container:
https://github.com/skx/rss2email/
[1] https://feedbin.com/
1. https://blog.newsblur.com/post/146752875548/newsletters-in-y...
So I wrote a little web service accepts an RSS feed, and then crawls each link in it, parses the HTML to extract the text using one of those text extraction libraries, and then "republishes" a new version of the feed (with the full article) which I then subscribe to in my RSS reader so I can read the articles offline, e.g.
example.com/rss?feed=foo.net/artcles/rss
I've never open sourced it though because I guess it's a bit of a grey area - the sites want you to go to the full URL so they can show you ads etc
An older version available on BitBucket: https://bitbucket.org/fivefilters/full-text-rss/src/master/
EDIT: full-text-rss + newsboat + nyt screenshot: https://lsngl.us/@alrs/105652475456016345
One thing I don't like about the whole article RSS is that I get requests to narrate my whole website or blog experience, in addition to the normal writing about stuff.
In effect the full-article RSS audience has its own experiential needs, which include "tell me about what you added to the non-blog sections, tell me about the new tools in the sidebar, or the new article footer format, or even hey I never heard about the new design," and so on. Sometimes I also alter the CSS for a single post, when it makes sense.
Some of this narration is just expected if your main format is traditional, plain blog. But personally my blogs tend to evolve and incorporate other stuff. And for sites with even more of an experiential aspect than my own, I can see why RSS could drive the publishers a bit mad, increased exposure or no.
Because of this experience, personally I would rather try reading via a page-download tool or format, even automated w3m or something, and see if that's an improvement. You'd get the full article in plain text, but you'd also get some of the other stuff around the site, including links which you may even be able to mark for later download.
It isn't a religious thing - the vast majority of the time, the truncation is way too short to properly advertise the content. The mystery doesn't move me to click-through, quite the opposite - it is annoying, so I remove the annoyance.
I understand the tradeoffs for folks who want to be paid via ads, but the utility just isn't there for me - I do not lack for reading options, and if something is really good, it'll find me some other way.
I guess they just follow the URL, extract the body text the same way that sites like Instapaper do it, and give that back.
Probably more RSS readers can do the same.
I haven't found a good solution, but I just suck it up and serve the full text anyways
It seems to me like there needs to be a standard pingback endpoint (either in /.well-known or, better, right in the podcast/rss XML) that your client pings when it seems like you've consumed the content (scrolled through most of a web page, listened to 80+% of a podcast), but only if you've opted in to that.
Basically, for feeds where I have "full text" enabled, it fetches the article page with mozilla's "reader mode" implementation and extracts the full text.
It isn't really usable with aggregators such as planet kde/gnome, though.
For whatever it's worth, I do click-through RSS feeds on my blogs just because I was too lazy to set up full text feeds and figure out how stuff like video would work and whether or not it would be a better end-user experience to alter the content in some way on style-heavy pages. Right now I'm constructing the feed myself. But if someone ever wanted to use a tool like you're describing on my blogs, I would have no objection. Don't feel guilty about scraping a site I build or building tools to do so, if anything it would make me feel less guilty about never getting around to setting up full-text RSS feeds. :)
I don't run ads or analytics anyway, but even if I did, as far as I'm concerned anyone who has the right to visit a page also has the right to download it, and to use automated tools to download it on their behalf[0].
I don't see any moral difference between using an adblocker and scraping a web page. Both are hitting the same endpoints and selectively displaying content that the user wants to see.
[0]: https://anewdigitalmanifesto.com/#right-to-delegate
Traffic to both the feed and to the main site increased noticeably when I reduced it to a descriptor.
I understand my using the sites that way contributes nothing really to their bottom line, I never saw any ads, but that of course was half the appeal. The early days of "web2.0" people often just naively dumped everything in the RSS feed.
It uses all sorts of manipulations to auto play video ads, pop ups, picture unders, user tracking, page interaction telemetry, etc.
All kinds of nonsense to display an article to you, which if it really is important, then another writer would have also written about it.
Some websites I just avoid completely, when I see the url.
These websites should just go with a pay-to-read subscription model instead. If their junk is interesting enough, then they’ll find enough people willing to pay a few cents to read it.
I honestly would not mind much if sites just included ads in their .RSS feed? Some bloggers I follow have their rss 'sponsored' and will do a sponsor post once a week. If ad insertion is really so important a site could just have labeled ads as part of the content stream. It's not ideal, but I'd prefer it to a feed that only has 2 or 3 teaser sentences and then forces me to click through.
I actually took Wired and Slate off my RSS feed just because I got annoyed at how often I would click through and get a paywall. Usually I was using up my free articles just to see what the article was about without reading it.
However I find news sites are the most common offenders of the truncating which I guess I can understand.
I'm maintaining the TT-RSS plugin feediron https://github.com/feediron/ttrss_plugin-feediron that fetches full-text data, so my thinking is this:
At the end of the day if it's a openly available website and you are personally (through your own server) fetching the resources I don't think anyone has a right to complain.
Now if you were offering it as a service it might arguably be a bit more grey, but only if you're ignoring the robots.txt file
The RSS aggregators should encourage readers to go to the original website. So when I got the chance to build my own RSS reader(https://feeds.pub) I only shows a title and link of an article to readers to drive them to the original website.
This is unacceptable to me, and if a site does this I unsubscribe immediately.
Great tool you have! I was considering building something like this (but directly in a feed reader), by any chance do you intend to release the source for it so that people can run it themselves?
Another reason, which I think is fair, is that with full articles in the RSS feed, the feed can quickly become a fairly large chunk of data.
I am not sure if there are many doing something similar to mine. My pattern is to Open all my interested Links in New Tabs. That is why I thought RSS Reader should always be within a Browser or as Web Services like Feedly.
I do offline Reading for long articles only. And Saved via Reading List on Safari that does automatic downloading. But it is quite problematic for me because most of the time I need to look up something related while I am reading on things that I dont understand or assumed wrongly. And this is still best done on a PC with Browser.
So I now only listen to Music, Reading Novel or WebComic / Manga for offline.