Show HN: I built a self hosted recommendation feed to escape Google's algorithm (github.com)
I created this chrome extension for myself where I track my own behavior locally and recommend myself content from platforms I want content from (youtube/twitter/quora/etc) in a feed. I made it public just in case anyone else was interested.
I would rather have control over my own algorithm and own the data. Also, it gives me flexibility. Turns out I do like these feeds just not when I don't own it haha. Let me know what you think of my implementation?
137 comments
[ 4.2 ms ] story [ 186 ms ] threadI have been working on something a little bit related recently, related to books. Kind of reverse-discovery, maybe.
I noticed that my ebook libraries in apps pretty much always end up in the same ordering, or sets of orderings. So I would tend to forget about interesting books I bought, the longer it's been since I purchased them.
I wrote some scripts that collect random thumbnail images from my various ebook and PDF collections every so many minutes and gather them into little collections. Like shuffling cards and dealing some out. It's very simple stuff but it's already been nice to have.
I also noticed that my collection of Gutenberg texts didn't have the greatest thumbnails, so I made a script to pick random excerpts from N different Project Gutenberg books at intervals. Then it joins then to a single text file. Kind of like visiting the bookstore to browse through some books a bit, see what's interesting.
Anyway, this interests-related stuff is enjoyable. I really like your idea for the discovery side of things, thanks for sharing it.
Do you people not use yt-dlp (or youtube-dl) because some of the sources they support can have more "extreme" content as well?
I'd understand it if the project was built by the same people who work at Gab, or meant for Gab, or something like that, so there really is an associated. But there is not, so why blame the project for it?
For all we know, the author could be reading Gab because sometimes you're curious to see what all the extremists are up to.
Seems like you are drawing conclusions that I, at least, never said.
> Do you people not use yt-dlp (or youtube-dl) because some of the sources they support can have more "extreme" content as well?
I don't think this is at all a fair comparison, as a simple glance at the front page can see. I also specifically tied my comment to there being only 6 options.
> For all we know, the author could be reading Gab because sometimes you're curious to see what all the extremists are up to.
Why would you want a recommendation feed for it then? Wouldn't you want to see it without the algorithmic distortion towards what you would prefer to see?
If I was building something like this, I'd start with something I use a lot (like Youtube), even if it required screen scraping, and then I'd move on to everything that had a dead simple API without low limits.
BUTTTT looking at the code, it would appear that they're screen scraping everything. But maybe it just so happened that the shape of Twitter's DOM is really really close to Gab's and it was super easy to add.
Or maybe this is a totally trojan horse approach to getting HN users on Gab. Who knows? But at least there's an easy way to turn it off. Not going to stop me from trying the extension, at least. (And as someone who tries to expose themselves to content bubbles that I consider insanely off-base, whether for education or entertainment, maybe I'll leave it on to see what kind of insanity is going on over there.)
Reading a site out of morbid curiosity doesn't mean you never want to follow a particular topic, and what is recommendation if not following topics without having to list them explicitly.
Because it shows a significant lapse in judgement.
> the author could be reading Gab because sometimes you're curious to see what all the extremists are up to.
Then they should say it. You can't assume good intentions when anyone willingly includes Gab because they'd be an incredibly rare outlier.
Ouch, this really hits me. I've seen so many issues lately because of this belief.
"Oh, but this one is justified!" they all say...
Viewing people as evil is different from guarding against evil input you might receive.
I don't have to assume everyone is a murder just because I also have a home alarm system.
If they are willing to publicly throw support (and jeopardize their user's opinion of them) at an extreme platform, what other explanation is there?
From Wikipedia:
> BitChute is an alt-tech video hosting service launched by Ray Vahey in January 2017. It describes itself as offering freedom of expression, while the service is known for accommodating far-right individuals and conspiracy theorists, and for hosting hate speech.
A society's level of tolerance can be measured by the least tolerant group it allows. If you allow nazis to remain at your private party, then you've joined the nazi party.
If gab, with its deep antisemitism, is tolerable to you, well...
I choose to not tolerate antisemitism, hence my comment.
However, all of what I just talked about is just a hypothesis and I'd be curious about how it works out for you and what traits it shares with other recommendation engines and what ends up being more unique. For example, does your algorithm just end up pushing for engagement (which also tends to push towards showing extremist content), or is it less likely to get caught up in rabbit holes?
I understand the purpose, but your browser of choice doesn't fit the purpose, it's just a bit sad :(
I wrote about this on my blog [0], but in a nutshell they slow you down, take you in unwanted directions and encourage consumption over action.
[0] https://suketk.com/feeds-considered-harmful
Case in point, as another commenter pointed out — it appears the OP feels GAB, commonly viewed as a far-right social media platform, should be listed as one of a handful of default sources.
World needs more systems that actually reflect how the user fits into the world, allows people to understand differences and find shared beliefs — not systems that create information bubbles, enable isolation, and increase the likelihood of extremists.
Is there such thing as a moderate-right social media platform? Does far-right mean "not supported by a majority of right-leaning people", or does it mean "extreme from the perspective of an average left-leaning person"? I'm not going to either defend Gab or put it down, but even if you deconstruct your statement, it's the product of an information bubble. And it can have different meanings depending on what type of information bubble you live in.
I mean the knee jerk response for me is Facebook, though I’d say it’s solidly right overall and not moderate-right.
Gab is certainly "far right", but that doesn't mean those views are uncommon. Centrist viewpoints have shrunk, pulling what was previously considered "far" viewpoints into the mainstream. I think it's undeniable that a website where many (most?) of the userbase would advocate against the peaceful transition of power in the US should be labelled "far right".
I don’t think anyone said those views were uncommon, just that they were far right.
...and saying that HN is "our own" - vs. "one modest Province in our Benevolent Overlord Paul's Empire" - is a bit of an issue.
"Pro-science" is not a position, and being anti-science means being anti-humanity.
Please consider the contradiction in this quote
From the other comment I figured I should probably point that out clearly.
What I'm trying to say is that all spaces are "echo chambers" in that they all implicitly have their own tone and angle that all the participants agree to uphold as they join. You don't discuss politics at the dinner table after all.
I'll agree though that it's still very much encouraging of new technologies even those that will be certain to cause problems. For example, people here are far more interested in the cool things they can do using WebAssembly than they are worried about how it will be used to deliver malware and violate people's privacy.
Whether we agree with that position or not is a whole different beast, but it absolutely is a real and tenable position that people can and do hold.
Wait, this is worded incorrectly right? The pro-science position would be the one claiming that science is the best (or at least one good) way to learn information about the world right? I'd agree in that case that a person can be pro-science.
pro-science also very often veers into anti-humanity as well. we like to pride ourselves that we’re keenly aware of how many unanswered questions we have and the current limits of science and technology — yet it seems increasingly common that when we’re chasing our love for science and technology, more and more we forget how these very topics can strip humanity from a given situation or conversation.
do i think religion is the answer when society’s pro-science stance leads to anti-human situations? absofuckinlutely not. (particularly organized religions. i’d be more open to religions if people could practice it as a personal thing. but sadly the religious always seem to want to inflict their personal beliefs on others.) the answer probably lies more in the region where the science minded get a firmer understanding that the human condition requires us to understand humanity more, not solely science. understanding other humans is a subject in which many of us are kinda lacking. i truly believe they were wise to require a certain number of humanities courses in university. honestly, seeing how terribly many of us in science fields struggle with understanding other humans, we could probably use a few more in the requirements.
in an attempt to tie this rant onto the topic: i’ve maintained for years that even in the best of our current iterations, algorithms are just plain terrible when compared to human curation. billions and billions spent and years and years later, still terrible.
i’m sure some will argue but i strongly believe this is true for just about everything, from news algorithms, music, movies, literature, art, shopping etc… etc… the best recommendations are still coming from other humans, by a significant amount.
As for what a moderate-right platform would be, I'd actually point right here at hackernews. It allows discussion around more right-leaning topics, but draws a clear line at outright hate. There's progressives too (hi) but you'll find no shortage of right leaning people either (like OP apparently). The sheer number of libertarians here alone is enough to move the needle right of center.
I sort of consider myself a libertarian who sees the need for a social safety net, regulations, strong consumer protection, education, infrastructure, emergency services, and healthcare. You know, a democrat. :D
As for not having a left/right spectrum of political views at all, I think it's somewhat inevitable in a (de facto) two party system, but it's worthwhile to acknowledge that virtually nobody is going to fit comfortably on one side or the other. People have tried for ages to come up with more nuanced multi-axis representations of where people fall within political spectrum, but even then it often just works out to carving out specific flavors of left/right.
That's literally every platform. There's no nazi-or-not test to take before you're allowed to use any of them. All platforms allow varying degrees of racist rhetoric ranging from dogwhistles to death threats and I'm not aware of a single platform that doesn't have racists in healthy numbers.
I think a better definition of a far-right platform would be found in its intended audience or in the percentage of racist users/content.
I’m not American BTW. Just an external observer who’s just flabbergasted how much a developed country can revolve into questioning of such obvious facts in the name of “both sides.”
> In 2012, D'Souza contributed […] Two years later, D'Souza pleaded guilty in federal court to one felony charge […] He was sentenced to eight months in a halfway house near his home in San Diego, five years' probation, and a $30,000 fine.[29][30] In 2018, D'Souza was issued a pardon by President Donald Trump.[31]
Some simple math reveals the pardon came long after his “incarceration”, and the sentence including probation would have been fully served by now (2012 + 2 + 5 < 2022).
Where are you getting your information?
The report concludes that the investigation "did not establish that members of the Trump campaign conspired or coordinated with the Russian government in its election interference activities"
https://en.wikipedia.org/wiki/Mueller_report
And since you brought up prosecutors, let me ask the obvious question: why have none of the investigations against Trump resulted in charges?
Pot/kettle/black
https://www.acslaw.org/projects/the-presidential-investigati...
- Special Counsel Mueller declined to exonerate President Trump and instead detailed multiple episodes in which he engaged in obstructive conduct
- The investigation “identified numerous links between the Russian government and the Trump Campaign” and established that the Trump Campaign “showed interest in WikiLeaks's releases of documents and welcomed their potential to damage candidate Clinton”
- Russia engaged in extensive attacks on the U.S. election system in 2016
- The Special Counsel investigation uncovered extensive criminal activity
Yes, obstructive conduct. Not collusion with the Russians.
The very fact that Mueller detailed episodes of obstructive conduct and not episodes of collusion reveals how little evidence there is to support allegations of collusion.
And as for the obstruction, remember that an FBI lawyer pled guilty to doctoring an email that was submitted as part of a FISA application used to surveil Trump campaign adviser Carter Page.[1] And Igor Danchenko, the primary researcher of the infamous Steele Dossier, is awaiting trial for lying to the FBI about the dossier's sources.
So is it really a crime to "obstruct" an investigation that is based on lies? Such an investigation should be shut down.
1: https://www.cbsnews.com/news/former-fbi-lawyer-kevin-clinesm...
If we say that Gab isn't far-right, it's kind of like saying Grindr isn't just for gay men. That may technically be true, but in practice it's not.
Also, judging Gab by its ownership, if you look at the Wikipedia page for Gab, and go to the Antisemitism section, Gab has made multiple antisemitic statements officially. Straight from the company. The owners of the site are far-right.
Gab only exists because far-right people get banned on other platforms for breaking hate speech rules.
The far-left as a group barely exists compared to the size of the far-right. Far-left would be advocating for violence to dismantle the neoliberal system, these would be your anarchist-types.
In real life, I've never met anyone on the "far-left" when you describe it this way, while I've met plenty of openly racist and LGBT-phobic Republicans who are pro-violence: the kind of people who want to, I dunno, be a part of a coup and enter the Capitol building? The kind of people who cheer when Donald Trump tells police to rough up the suspects a little more.
What Republicans call "far-left" is just progressive center-left people nonviolently advocating for equality. They don't get banned from the generally non-political platforms because their speech isn't hateful and it isn't violent. Facebook/Twitter/TikTok happily take their money and call it a day, they don't care about politics beyond the money those politics bring in.
Platforms like Parler are much more blatant than Gab about banning liberals who join simply for existing. I guess you could say that Gab stays true to its policy against censorship, but that policy is there to benefit the far-right over anybody else, and I think that's an important detail.
And, by the way, if you head over to Gab’s homepage it’s all right-wing political stuff. None of the suggested homepage topics are non-political. It isn’t teenagers dancing or funny skits or cat videos like TikTok. I think that’s evidence showing that Gab isn’t just some impartial algorithm. It’s curated.
In the US, the right wing is often bound up with co-opting Jews via the term "Judeo-Christian" and supporting Israel, even it its worst abuses of the Palestinians. They often call-out left-wing antisemitism, which often tries to take the form of "anti-Zionism" (but whose information is often traced to antisemitic channels).
So antisemitism doesn't by itself demonstrate far-right leanings -- even though the far right is also full of literal Nazis and those who call them "very fine people". But Gab is certainly happy to play host to those, too.
First, I 100% agree with you that what the mainstream right categorizes as the "far left" is really anyone "center left" that they disagree with (or even slightly right, as many mainstream neoliberal Democrats fall on the "political compass" (though take that with a grain of salt)).
However, there are absolutely folks on the far left advocating for violence as a means to their goals. I see more of them online, obviously, than in real life (though I do know folks who do black bloc, destroy property, etc.), but it's really hard to know to what extent that's an artifact of their tactical differences with the far right. Many on the far left (including anarchists, out-and-out communists or AnComs, and even some DSA types) embrace anonymity, decentralization, and _not_ documenting their efforts as core tenets of achieving their goals.
On the other hand, the far right's strategy is to beat their chest and embrace highly visible patriotic pride, which they then use to create associations between say, the flag, and their more extreme political beliefs. Then, they dog whistle and gaslight the other side about the full extent of their own and their opposition's goals as they move the goalposts towards the right to distort the mainstream view of the center.
Theoretical unknown numbers aside, I believe the latter tactical posture has been significantly more successful in the far right achieving their goals in the political mainstream. I also believe the far left's tactics have hurt them significantly in being accepted by the mainstream (and in talking to many of them privately, they often see this as a point of pride, that they're unwilling to compromise with the "normie" Democrats who e.g. sold them out throwing Bernie under the bus during Clinton's ascent to nomination). E.g. even as a small minority of the overwhelmingly peaceful BLM protests in 2020, just a few guys in black bloc throwing molotovs enabled the mainstream and far right to immediately write BLM off as a violent, communist, antifa movement that was a threat to democracy, capitalism, and the American way of life. So why throw them at all? What good did that do? (And could it have ever been enough to offset the negative PR?)
Even the slogan that many on the mainstream left embraced--defund the police--was a terrible messaging position that AFAICT almost always required immediate clarification of why that didn't mean "we want to abolish the police, all criminal statutes, and let society run wild doing whatever they want". I think it would've been a lot harder for the right to attack them if they'd have gone with something like "fund community outreach [for POC and mental health]". And then they could have even found their $$$ in the police budget later (a trick Republicans use all the time). Hell, they might have even found alignment with Republicans in the wake of mass shootings to actually address mental health on a national scale (something I'd argue is critical to our long-term well being as a country, regardless of your stance on their relation to mass shootings).
ANYWAY, I got a bit off the rails there, but far left filter bubbles--with calls for violence, personal attacks, doxxing, and all the rest of it--absolutely exist on Twitter and Reddit (among other places, I'm sure). In particular, just pop over to subs like /r/GenZedong, /r/COMPLETEANARCHY, /r/Anarchism, /r/AnarchismZ, /r/196, /r/2624, /r/JusticeReturned, or many many others. (I'd also argue that these transgressions are largely tolerated on those platforms in a way that the far-right is not, leading to an increase in demand for an increasingly fractured ...
The site received extensive public scrutiny following the Pittsburgh synagogue shooting in October 2018. The perpetrator of the attack, Robert Gregory Bowers, had a history of making extreme, antisemitic postings on the platform, as well as messages indicating an immediate intent to cause harm before the shooting.
A story about how Republican Ron Johnson is a liability because he acknowledges that Joe Biden won the last US election is currently in the top 5 on Gab's frontpage. Many similar stories are highly ranked as well. It seems quite self-evident that Gab's main audience leans conservative (in the American sense) and that conspiracy theories à "stolen election" are quite popular there. These do seem quite "far right" to me even if they might have mainstream support at the moment.
Another of your default options is "BitChute" which is a platform for similar content.
To my understanding, Fox News is pretty commonly accepted as a news source that's right of center, and even they've run articles describing Gab as far right.
I know it's uncool to do political bikeshedding on HN, but if you'll indulge my curiosity for a minute, how would you describe Gab's politics if not "far right," and are there any spaces you would personally label that way?
[0] https://en.wikipedia.org/wiki/Gab_(social_network)
[1] https://www.foxnews.com/politics/social-media-platform-gabs-...
Gab is well past "right wing" and well into "company replies to the complaints about antisemitism with press statements reading 'JESUS IS KING'" territory. The site's content basically parrots whatever today's Republican most extreme talking point is.
I just opened Gab's front page in an anonymous browser, and the first 3 posts were a "Let's go Brandon" joke, a salute to Hobby Lobby for defending Christian Nationalism, and a post about "Ron Johnson is a RINO because he thinks the President won the election." If you can read through this and think "this isn't far right," your perspective has been significantly influenced by whatever media you're consuming, and you need to look into deprogramming yourself.
https://antisemitism.org/wp-content/uploads/2021/10/Screensh...
Which isn't some dogwhistle, inapt wording, etc. It's straight up, plain Nazi phraseology:
https://en.wikipedia.org/wiki/Jewish_Bolshevism#Nazi_Germany
Otherwise it is creating information bubble by excluding far-* or whatever.
I wonder if theres a way to temper a user, such that you expose them to more vitriolic bullshit on either side with some survey based feedback, and pull back again to allow for recovery, you go to and fro until you filter some undesirable components, idk maybe you have this pseudo-personality profile you target or you just want to minimise usage of a set of biases and you include lessons alongside it.
Today's super polarised identity politics was „mental hazard worth avoiding“ 20 years ago. Maybe the society at large should have paid more attention to it.
I think my failing in this context is more to do with apathy inspired by everyone looking like assholes or idiots. Watching people interact poorly doesnt inspire positive behaviour by itself. It puts you in a position where the norms are unhealthy, and youre the one left with the additional mental work of struggling against those norms and making them better.
Maybe its necessary for society though, in some balance, because of what youre saying. Its easier to deescalate when there are people in the room that arent seeing red and blue and its easier to fall into dogma when youre only chilling with your political color.
Assuming the above is valid, maybe these questions are worth considering - How likely is consuming a bunch of blue v red perspectives to prompt a healthy and thought out response?
Whats the process in going from uninitiated to someone with a healthy, well adapted stance?
How do we make it more likely that people make it through that process, across social strata?
IMO the importance in reading all sides (IMO there're much more than 2 sides, even in US, let alone Europe and the rest of the world) is to learn that all sides deep bellow have some good ideas, some bad ideas and then there's reasoning behind them. And different contexts where some bad ideas actually are an improvement and good ideas would be net negative.
Of course, it's takes energy to read into that. It needs smart and thinking population to work. Personally I think classic philosophy is a good starting point. Pre-modern-era history. It gives some perspective how different contexts shape people. How things change in a sort of circular way. How societies were built and how they crashed.
It's the curse of democracy. On one hand, it's great. On the other hand, it's damn hard work to keep it well oiled.
At the end of the day, I think there's no need to „unify“ people for the sake of it. Different people live in different contexts and naturally they come to different conclusions. What we need is to get people into more intelligent mode. All camps need to get to the root of their issues and think of a constructive way to fix them. Rather than just destroy the world and hope the issue fixes itself. And then we need more autonomy at all levels so people living different contexts can try out their ideas at small scale.
Same pattern with Brexit or Trump.
> World needs more systems that actually reflect how the user fits into the world, allows people to understand differences and find shared beliefs — not systems that create information bubbles, enable isolation, and increase the likelihood of extremists.
Wasn't the past world of BBS, NNTP, IRC just nice? Everyone had space for themselves, there was community moderation, sure some smaller or bigger attacks on each other, but it always was somehow nice and in balance, methinks. I particularly miss NNTP, we could have this discussion there.
No. It was just tiny and very non-uniformly sampled so you didn't notice where it was heading as easily. E.g.
https://en.wikipedia.org/wiki/Serdar_Argic
Then the question is what makes it far right: far from our point of view, or far from the normal point of view in the US?
Because I strongly suspect that what we consider normal is not the normal and so what we might think of as far right is not that far from the center.
If not, I would love to see this add a "forward to webhook" option. I would be happy to write up a real backend that parsed the content and indexed it.
Actually, there are lots of OS projects for this: https://github.com/quickwit-oss/tantivy, https://github.com/valeriansaliou/sonic, https://github.com/mosuka/phalanx, https://github.com/meilisearch/MeiliSearch, etc...
I would think that with the thousands (tens of thousands?) of pages I would index just browsing each year it would be relativity easy to find ways to automatically expand the index to include links that appear multiple times in those pages or some other heuristic
im wondering if it's harder to get the same quality data as with short video, but still curious to know how they got recommendations that much better. is it just more/better data into the algo, like with rewatches etc tracked? or actual tech improvement?
bc yt shorts and ig reels don't have the same recommendations even with about the same data.
- Have you thought about adding edit functionality to your content sourcing under feed settings to add or remove sites? Wikipedia, Stack Overflow, and other sites come to mind.
- Have you thought about adding an report export functionality of links recommended? Basically to help an individual understand why their copy of the algorithm is recommending specific items.
Ok, so, my understanding is that self-hosting is running a web service for personal use. This isn't self-hosted. It's a Chrome extension. It's like saying that you're "self-hosting" Microsoft Word because you have a Word window open on your desktop. (Or, slightly more similar, you're "self-hosting" magit because you have emacs installed locally.)
IMO, selfhosting here was descriptive in that the Chrome extension is storing all data and generating all recommendations locally, without the use of a managed service that would presumably be storing all of my history in someone's cloud instance. (I'm sure you're a great person, OP, but I wouldn't want to give you my entire browser history.)
Yes, it's shipped in the Chrome web store, so OP could theoretically ship different code than what's showing in the open-source repo, but you actually _can_ selfhost even "further" by building the extension yourself (even if installing unsigned extensions in Chrome is harder than ever, I think it's still possible, and certainly Edge still allows it via Developer Mode).