I'd been thinking recently about how to write a "train your own recommendation algorithm" feed reader and fediverse reader, but I was unsure how those sorts of algorithms work. It looks like cosine similarity is a good place to start.
I wonder if you could get anything useful from just upvoting/downvoting entries in your own feed for a few days/weeks this way, or if much more signal would be needed for a real algorithm.
This sort of algorithm requires quite a lot of data to work. If you had a large collection of many people's votes/bookmarks, that would work. It's not enough to just use one person's preferences as there is nothing to compare them to.
Although I guess you could use link data (or some other source) to base suggestion off.
I've thought about it a lot and I really thinking having these sorts of "interest" or "friend" groups are the way to go. A major problem with major search engines like Google is they're really trying to do it all in one.
We can keep a generalized utility search as the default, but I wish I could somehow tell a search engine "show me stuff you'd show to my friends from this community". I.e. what would HackerNews' community search group look like? What would this friendgroup that for some reason includes a lot of people who work in botany but that isn't the defining feature of the group. Maybe they all also happen to like board games, are allergic to social media, and are weirdly well-read on continental philosophic traditions.
I think the key is that people have exit. My main gripe with things like TikTok and twitter is the algorithm tries to optimize results for what you'd like most. But people don't really work that way. People grow over time. People join different communities, put on different hats, try different identities. Sometimes it happens over time or sometimes people are in phases of their life where they're constantly in between multiple communities. I would feel more comfortable with the "profile" these algorithms made for me if I had some sort of say in it and could switch between at will or do an "incognito"/fresh slate mode whenever I want
Hey, I love the idea behind marginalia. I love the fact that it doesn't require javascript, and in fact lets you search only for sites that work without javascript! So incredibly awesome!
Unfortunately Marginalia itself requires javascript. Could you please do something about this?
I would use Marginalia a whole lot more, except that if I do more than two or three searches an hour, all I get is this:
search.marginalia.nu
Checking if the site connection is secure
Enable JavaScript and cookies to continue
search.marginalia.nu needs to review the security of your connection
before proceeding.
... and then I just change my search engine back to Yandex in frustration. After a few days I tend to forget about Marginalia. At least until it pops up here on HN.
You're probably unlucky enough to share an ASN with a spammer. I wish I didn't have to degrade service like this for some people, but the alternative is not serving any traffic at all because the site is constantly DOSed by bots.
Cloudflare and other providers add this in front of websites for DDOS protection purposes, and it is triggered based on different factors, one of which is location. I regularly get this on an Australian forum that discusses SOHO routers (and Javascript-based validation always fails so I have to solve a captcha instead).
I imagine "spammer" here is not an email spammer, but either e.g. a comments spammer (a bot trying to post links in comment forms to game google), or a generalized thought shortcut meaning something like "a bot/botnet crawling a website for nefarious purposes in a high-traffic way with abusive disregard for the site's computational and bandwidth resources".
Both HN and XKCD probably have a fairly broad audience with decent overlap. Seems to happen with popular websites, they lack strong specific correlation with anything.
Yeah, seems like a difficult thing to tune with recommendation algorithms.
I think it essentially boils down to the Friendship Paradox. I've mostly ignored the problem as I'm more interested in the fringes of the graph, but I'll note even the likes of Spotify and Youtube struggles with this.
You mentioned rich data earlier, and as an amateur sitting on the sidelines I think that's the key. Once I fed my entire library into LibraryThing, its recommendations were orders of magnitude better than, say, Amazon's, who only knew which books I'd bought through them. In fact they were so good it was mostly recommending books I'd already read, but didn't own.
My toy, which relied on scraping <ul> and <ol> lists of people's favourite bands from personal websites, had almost nothing to go on by comparison. last.fm might have been able to do something good with the data they collected, but they seemed to die on the vine once they were acquired.
This is really neat! It reminded me of the defunct sets.google.com (yes, very different things, but there is enough similarity that it came to mind).
If unfamiliar, google sets would return other terms related to the term the user input. E.g., input: cisco output: brocade, juniper, hp, arista etc.
@marginalia_nu, if you are ever looking for another project to do with your scraping data, a sets.marginalia.nu could be super useful.
On topic, the top results returned for my queries were spot on. Some others were initially surprising, but probably correct. Creepy website similarity or an on-the-fly web ring creator.
1. As I understand a 10M bit vector was created for zombo.com target, features were domains(10M of them), and values were booleans of whether site links to zombo.com .
So on the fly to save space 10M bit vector was converted to a 64 bit bloom filter. So no false negatives and a certain percentage of false positives - so far so good.
2. Those 10M sites all have already precomputed bloom filters. If I understand correctly features would be 64 now and we calculate cosine similarity on those?
1 -- I'm using the link graph for my search engine, which consists of the domains I've crawled, as well as domains I've not yet crawled. It's these linked-to, but not-yet-crawled domains it will discover.
2 -- The bloom filter is used to not calculate the cosine similarity directly. It's used to eliminate cases where the vectors are trivially orthogonal.
It's this part in the code example
if ((a.hash & b.hash) == 0) {
return 0;
}
During vector construction, these bloom filters are calculated something like this
hash = 0;
if (values.size() < 128) {
for (int v : values) {
int bit = computeHash(v) % 64;
hash |= 1 << bit;
}
}
else {
hash = ~0L;
}
If there is no overlapping bits between the bloom filters, it is possible to skip performing the calculation. If there are overlapping bits, the calculation needs to be performed. This only works for small vectors, but that is a relatively common case. For large vectors it's just a single logical AND and a comparison instruction, so the optimization is extremely cheap.
I don't perform the calculation on the fly, but rather I've pre-calculated the similarity between each domain and store up to like 75 results given they have good similarity. This took about a day.
In the 1910s, propagandists would send push-surveys that included a question "(who else/what other organisation) do you know who should get this survey?" in order to map out socio-political graphs and plan their campaigns.
HREFs sound like much less manual work.
Edit: > ...able to find similarities even among websites that Marginalia doesn't index
Yeah, I joined LinkedIn specifically because even without my input it was way too good at figuring out my actual social graph. With my input, it no longer is :)
I feel like people can only get attention for something if they express how disturbed by it they are. If this is creepy how creepy is a search engine? You can find the websites of "democracy advocates in Hong Kong, Putin-critics in Russia, gay people in Uganda" using any search engine. People put stuff on the web because they want it to be found.
It reminds me of the hand-wringing about the youtube recommendation algorithm. Billions of people discover amazing stuff is not a story, a few people discover disturbing stuff is.
You can fairly easily hide from a search engine by essentially doing reverse SEO techniques, by blocking crawlers or feeding them different results, or just hiding behind login screens.
The problem with the similarity technique is that it can find associations between sites that do not link to each other, regardless whether and how they try to hide. It's enough that someone thinks they are related.
29 comments
[ 1.8 ms ] story [ 65.4 ms ] threadI wonder if you could get anything useful from just upvoting/downvoting entries in your own feed for a few days/weeks this way, or if much more signal would be needed for a real algorithm.
Although I guess you could use link data (or some other source) to base suggestion off.
We can keep a generalized utility search as the default, but I wish I could somehow tell a search engine "show me stuff you'd show to my friends from this community". I.e. what would HackerNews' community search group look like? What would this friendgroup that for some reason includes a lot of people who work in botany but that isn't the defining feature of the group. Maybe they all also happen to like board games, are allergic to social media, and are weirdly well-read on continental philosophic traditions.
I think the key is that people have exit. My main gripe with things like TikTok and twitter is the algorithm tries to optimize results for what you'd like most. But people don't really work that way. People grow over time. People join different communities, put on different hats, try different identities. Sometimes it happens over time or sometimes people are in phases of their life where they're constantly in between multiple communities. I would feel more comfortable with the "profile" these algorithms made for me if I had some sort of say in it and could switch between at will or do an "incognito"/fresh slate mode whenever I want
Unfortunately Marginalia itself requires javascript. Could you please do something about this?
I would use Marginalia a whole lot more, except that if I do more than two or three searches an hour, all I get is this:
... and then I just change my search engine back to Yandex in frustration. After a few days I tend to forget about Marginalia. At least until it pops up here on HN.Also, what does any of this have to do with email spam? We're talking about HTTP(S) here, not SMTP, right?
My toy, which relied on scraping <ul> and <ol> lists of people's favourite bands from personal websites, had almost nothing to go on by comparison. last.fm might have been able to do something good with the data they collected, but they seemed to die on the vine once they were acquired.
If unfamiliar, google sets would return other terms related to the term the user input. E.g., input: cisco output: brocade, juniper, hp, arista etc.
@marginalia_nu, if you are ever looking for another project to do with your scraping data, a sets.marginalia.nu could be super useful.
On topic, the top results returned for my queries were spot on. Some others were initially surprising, but probably correct. Creepy website similarity or an on-the-fly web ring creator.
1. How does marginalia build up the 10M bit vector for a site it has not seen before?
2. How do you do a cosine similarity on a 64 bit bloom filter?
======== Example: https://explore2.marginalia.nu/search?domain=zombo.com
Results seem quite good.
1. As I understand a 10M bit vector was created for zombo.com target, features were domains(10M of them), and values were booleans of whether site links to zombo.com .
So on the fly to save space 10M bit vector was converted to a 64 bit bloom filter. So no false negatives and a certain percentage of false positives - so far so good.
2. Those 10M sites all have already precomputed bloom filters. If I understand correctly features would be 64 now and we calculate cosine similarity on those?
2 -- The bloom filter is used to not calculate the cosine similarity directly. It's used to eliminate cases where the vectors are trivially orthogonal.
It's this part in the code example
During vector construction, these bloom filters are calculated something like this If there is no overlapping bits between the bloom filters, it is possible to skip performing the calculation. If there are overlapping bits, the calculation needs to be performed. This only works for small vectors, but that is a relatively common case. For large vectors it's just a single logical AND and a comparison instruction, so the optimization is extremely cheap.I don't perform the calculation on the fly, but rather I've pre-calculated the similarity between each domain and store up to like 75 results given they have good similarity. This took about a day.
HREFs sound like much less manual work.
Edit: > ...able to find similarities even among websites that Marginalia doesn't index
Yeah, I joined LinkedIn specifically because even without my input it was way too good at figuring out my actual social graph. With my input, it no longer is :)
Lagniappe: https://www.tandfonline.com/na101/home/literatum/publisher/t...
Maybe if I'm looking for even more I could put my feeds in and get more content.
It reminds me of the hand-wringing about the youtube recommendation algorithm. Billions of people discover amazing stuff is not a story, a few people discover disturbing stuff is.
The problem with the similarity technique is that it can find associations between sites that do not link to each other, regardless whether and how they try to hide. It's enough that someone thinks they are related.