Ask HN: Let's build an HN uBlacklist to improve our Google search results?
The low quality of results has been a problem from a while now and has become worse lately thanks to all those StackOverflow and Github clones. So I was wondering if we could come together and contribute to a single blacklist hosted somewhere and then import it into each of our browsers. Who knows? We might end up improving the quality of the results we all get.
Lists to get rid of the StackOverflow and Github clones already exist. [1]
I would love to contribute to a project like this, but won't be able to be a maintainer due to time constraints. Would greatly appreciate it if someone could host this. A simple txt file on github would do.
What do you say, HN?
[0]: https://github.com/iorate/ublacklist [1]: https://github.com/rjaus/awesome-ublacklist
283 comments
[ 1.9 ms ] story [ 295 ms ] threadCan we just nudge them to do so under the threat of an influential minority leaving due to their use case being affected?
To the original replyer: you could wait for Google to do something, but if they were going to fix the listicle issue, and it were fixable on their end, they'd probably have done it by now. I'm disappointed in the situation too but if there is a workable solution on our end it would be silly to ignore it because fixing the problem is someone else's job.
To the OP: I worry that the number of domains pumping out crap might be far greater than we know, and that might hamper the effectiveness of this. If the collaborative block list ever got big enough you might also have to deal with spam. But I think it would be a great thing to try. This is one of those issues that annoys me, but it's just below my action potential threshold. My biggest objection right now is the spammy recipe websites.
Or somewhat related, the "if you touch it you own it" problem.
So to overcome this learned helplessness effect, we'd need a good strategy to prevent the deterioration of duty of whoever is "supposed" to be fixing a problem, and/or a way to cut out the derelicts entirely.
https://www.science.org/doi/10.1126/sciadv.1600451
I'm sure you're right about the number of spam domains, but Pareto suggests that blocking even a small percentage of them might provide a large gain.
https://en.wikipedia.org/wiki/Pareto_principle
If we manage to block them, we might be able to get a results page with good sites upfront and the other meaningless content below it. I assume Google will also surface good content along with the bad, so our blacklist might enable the good stuff to reach the top.
The spam problem, I'm sure of yet, but we might either be able to block enough of it to be satisfied or it won't pose a problem for most searches that are currently giving bad results.
https://github.com/sanketpatrikar/hn-search-blacklist
* code-search-blacklist based on this jhchabran's repo [1]
* pinterest.com based on SwiftOnSecurity's interesting seo analysis [2]
Maybe your repo could be an opinionated list of things developers find annoying about google search results, where others might value sites like pinterest in their results.
[1]: https://github.com/jhchabran/code-search-blacklist
[2]: https://twitter.com/SwiftOnSecurity/status/12588753334467174...
At the risk of sounding pretentious, I call it "socialist", since they spend their lives telling others what to do or what is good or not for the rest of us but they rarely do anything about it. Surprisingly, this is the group that is really worried about poverty and climate change and do as much as I do for it, with the difference that I do it by myself, the few times I do it, not requiring the rest to do it.
It is always someone else who will do it. Though the other day I had a conversation with a non-socialist person that had that same attitude ("other should do it") towards what OTHERS should do. I really dislike that attitude, no matter where it comes from.
Point at hand: when I want or promote something, I am the first one to do it no matter others do it or not. The rest, no matter the ideology, all b*llshit.
As imperfect as I am, I try to do what I think is good (and sometimes my imperfection prevents me from doing it) but I do not spend my life telling other people why they are worse than me and telling them what they should do or not. The most I have for someone is good suggestions, never requirements.
I cannot see how that would be a vice of capitalism since capitalism is basically non-intervention and freedom.
I don't think socialism is the word you're looking for. If you're trying to say that the described behaviour ("it's everyone else's job") is common in, or a byproduct of, a socialist system then it would make more sense (though it could be argued).
But that doesn't make it a correct "approximation" of socialism. Socialism is:
>a political and economic theory of social organization which advocates that the means of production, distribution, and exchange should be owned or regulated by the community as a whole.
https://www.lexico.com/definition/socialism
I did not mean anywhere every socialist is like that or that all peopple that do that are socialist.
On top of that, that is just a personal opinion. Nothing else.
I described an attitude I dislike and I often find, like it or not, at that side of the spectrum: we have to eliminate poverty, stop the climate change and do all good things in the world. At the same time I write from my big luxury Iphones, my big house (as I promote equality) and, of course, tell the rich to pay the bill as if their money belonged to everyone else. Mine no, I am no rich... I am socialist to point to who should do it, not to do it.
I have never, ever seen a person with a capitalist mindset pointing to the wealth of others or saying that everyone should have the same be equal for the sake of being and they are quite more restrained about telling all the others what to do. Also, they are often quite more frugal people.
A centralized economy is not flexible enough to supply the changing demand or to readjust.
The big leap forward was created, for example, to overtake England (I recall, correct me if I am wrong) in Steel production. In practice what happened is that people run desperate for their lives and died of hunger. Because they did not have the means to do it.
There are way more examples, being this one maybe the biggest disaster, but in essence there is an impossibility to calculate prices and gather information of good quality in a socialist system.
https://en.wikipedia.org/wiki/Distributism
Not with public capital and under coercion, which is how a socialist system works.
It’s often possible to work around issues in lower layers, but it's usually at least worth raising it upstream to get it fixed 'properly'.
It'll help me when I dont have a blocklist active, and it'll help new programmers who arent familiar. It'll reward good sites with extra traffic and discourage new spammers entering the market.
In the worst case, if Google really can't or won't address tge issue, understanding the upstream problem more fully can help make a better workaround.
This reminds me of the Linux gamers who claim that they can influence game development companies by purchasing games with Linux ports, but wind up being less than 0.5% of sales of most games with Linux ports, which leads manufacturers to ignore that customer base almost completely.
I personally am kinda happy where Linux gaming has come to be. Sure it could always be better but I remember times where there were only like 3 games for Linux and you had to compile them yourself..
Have you searched anything on Google lately? The answer is "no". Their new job seems to be to stuff your results with anything even remotely related (and sometimes related in a way that only machine learning can see) so you have things to click on.
Edit: with the lone exception of "find me this bussiness nearby".
It's very noticeable if your search contains a short keyword that has to interpreted in the context of the other keywords. As an example, if I search for 'ARM assembly' plus another keyword (macro, syntax etc) it will see 'ARM assembly' without the extra keyword has way more high ranking results and happily show me how much it knows about armchairs that don't require assembly. Ignoring the fact that the extra keywords are there specifically to limit the search results.
It's tiring, a lot of time I previously spent browsing the limited but valuable results it returned I now have to spend mangling the keywords enough to outsmart their ML/NLP interpretation and get it to admit I am actually asking for the thing I am asking for so I can finally get to the part where I have to solve the modern captcha: click all the results that are:
1. Not stolen/rehosted 2. Not a "Hello World" level Medium blog 3. Written by an actual human
Or more fundamentally perhaps this is just the system working as Google intended?
I'll be happy to be proven wrong, but I think Google is now fully in the 'optimize for engagement' camp. If that's what they're doing, it's by definition not spam (from their point of view) if people are clicking on it more than the non-spam results.
Again, only my guess as to what's going on. I don't see another good explanation for them only serving cloned Stackoverflow and top X lists for basically everything now.
That hasn't been true for a long while.
Users are there for an answer to their query, not to be directed anywhere. Frequently this comes in the form of search vertical response that doesn't lead the user anywhere.
From Google's point of view, you want the user where you can show them ads, full stop. Google does not exist to provide you with answers to your queries, it exists to make money for shareholders.
> When you don’t have viable competitors both of those are improved by worse search results.
I don't think "worse search results" is the way to think of it. From G's point of view they're better because they make them more money.
> Can we just nudge them to do so under the threat of an influential minority leaving due to their use case being affected?
Many influential people have tried and nothing seems to have transpired from it.
Google.com is the most popular website. I don't think the leaving of any minority group we manage to create would even matter to Google, let alone force them to fix the issue. Not that I discourage using alternatives.
Start with changing default search engine to DuckDuckGo or something else, install uBlockOrigin and Privacy Badger to disable tracking, and gradualy reduce using every Google or application, starting with Chrome.
Be the change you want to see.
1. DuckDuckGo too is affected by these SEO-gaming sites, so maintaining a blacklist will help us make that experience better too.
2. There are times when only Google can find us what we're looking for, so this will prove useful when we go back to it.
I sense Google is too big to cater to us like this. Despite a steady decline in quality, Google is still the dominant search engine and the competition isn’t even close to its market share. Not only would they not notice many of “us” leaving, the amount of change they would have to implement in order to satisfy our desires would end up changing the product for the rest of the market. On some level, the product managers must be satisfied with the metrics as they stand since Google is continuing with their current course.
I doubt many advertisers like the status quo very much either. They basically have to pay for ad placement to ensure the first results for their product aren't ads for competing products. On mobile when I search for Boox the first result linking to them is an ad. Same for Kobo. In other instances I'll search for company or product and a competitor ad is the first to show. So vendors get stuck paying for ads when their own site should probably be the first organic result, above the ads.
Google doesn't make money from people finding what they're searching for. Google makes money by keeping people searching.
No. This is a classic mistake of intellectual types, who are impressed by each others' cogent arguments. But there is a much wider pool of people who are not, and among whom the intellectual types actual have very little influence, due to being boring and hard to understand (plus, it has to be said, kind of snobbish about how smart they are).
Now, you might reason that Google is full of smart people who should care about cogent arguments. But that assumes as an unspoken premise that Google's internal goal is to maximize the quality of the service and profit from being The Best. They passed that goal years ago are now so awash in money that it's cheaper to just squash competition than to innovate. They can be moved by threats to advertising revenue got up by angry crowds on social media (a market when they have little direct power), but Google would probably be delighted if grumpy nerds wandered off somewhere else. If they need talent or access to some compelling technology they can just throw a pile of cash at the problem.
On a much smaller scale, if anyone is interested, I maintain a black list focused on those code snippet content farms that gets in the way when you're searching for some error message or particular function here https://github.com/jhchabran/code-search-blacklist.
[1]https://www.cyberciti.biz/tips/about-us [2]https://twitter.com/nixcraft
Glad you said something though, I wouldn't have looked at it twice without a human attestation.
That doesn't really intersect with my observation in any way though. As a stranger I don't see your intent when I see your website, I just see your website.
The .biz definitely does not help, since it hints to me that it's just another one of those worthless reposting sites, as someone else commented below.
I'll be honest, I don't remember how I came to that conclusion but I suspect I encountered an unsatisfactory answer to a question I was looking to answer, saw the .biz and drew my conclusions.
The noise to signal ratio for most of my queries is so high that I have to start judging a book by its title, not even its cover.
That's really frustrating. I'm building a faster search engine for programming queries and just added your site cyberciti.biz as a recommended and curated source of Unix/Linux material. Hope more devs get aware of your work and you (and your collaborators) receive the credits deserved. Thanks for your work of many years.
When I see your site pop up in my search results I know the content is going to be more reliable than most of the others. Thanks for the effort you've put into it.
The other comment made me remember there was captcha too, right? I had been using my own rented server as a VPN for all my internet access. But I'd have never blocked it for a public list - I've read the 'about me' page.
Except they so obviously don't. They lie through their teeth and their good at marketing, that's it. They've zero technical ability.
In short, if you find what you are looking for, they get few ad impressions and make less money. Meanwhile, if you have to click through half a dozen spam results first, there are many ad impressions, and they make more money.
Until that incentive structure changes, Google will not be interested in solving this.
> If I end up having to solve a problem that's someone else's fault, yeah, sure, it's unfair.
> But if I end up having to live with a problem because you insist that we wait for the responsible party to fix it, knowing that they never will, that's more unfair.
[0] https://twitter.com/lkbm/status/1478425875578302464
...assuming "support" is a substantive action and thus a finite resource, and not just an upvote on a social media page.
https://news.ycombinator.com/item?id=29546433#29549855
https://github.com/sanketpatrikar/hn-search-blacklist
Google doesn't deserve my time or my eyes.
Are you only going to filter obvious spam and sites that republish other’s content, or are you going to block sites that are “harmful” or disseminate “disinformation”?
Who will get to decide which media bubble I’m in?
Someone, give me the late 90’s cesspool of un-curated internet back! Everything is so… trite now.
It only took 7 days for that other search engine project on HN last week (Mwmbl) to add hard-coded weights for certain news websites, so it does show how careful you have to be with this stuff.
https://github.com/mwmbl/mwmbl/blob/a41088ca9ad7fdcac952a3be...
And the issue is because it creates filter bubbles and introduces bias into how information is discovered.
"dangerously political", as if search ranking is not already political.
> Who will get to decide which media bubble I’m in?
Whoever you rely on for your consumption needs. Your search engines, content aggregators, and news sites already do. None of us have much agency online and pretending like we are operating objectively and rationally is delusional.
Imagine you had a position in a huge market that was as close to unassailable as there has ever been. Imagine also that you have a controlling position over the mechanisms that allow people to participate in that market.
Now try to make a case against optimizing for squeezing every last cent out at the cost of the user experience.
In 10 years we will regard Google the way we regard cable companies today. Maybe even worse since we need to be able to search for answers more than we ever needed cable TV.
If comparatively tiny operations like DuckDuckGo or Brave can launch decent search engines, I think there's hope of reducing Google's dominant position in the market.
I find it difficult to believe that relatively beginner NLP projects get posted here all the time, yet no one has adapted that stuff to create a new search index.
Personally I don’t know enough to really do this well, but I can tell just blocking sites from Google’s results isn’t the way.
- https://github.blog/changelog/2020-10-01-the-default-branch-...
- https://about.gitlab.com/blog/2021/03/10/new-git-default-bra...
However, recent versions come with "main" as the default.
And since 2.28, released more than a year ago, we have the option to change the default:
If I have to log in to an account only to search, that'd be a no-go for me.
I don't think I'd want to download a list of the most blocked sites and plug it into one of my tools though, for some of the reasons you mentioned.
A google search showing some of these leech type sites:
https://www.google.com/search?q=%22code+that+protects+users+...
For me, "farath.com" is outranking stackoverflow.
https://you.com/search?q=code%20that%20protects%20users%20fr...
To be fair, not sure what other results we'd expect if we're going to search for a specific, plagiarized phrase.
Edit: actually, upon review, you.com does indeed give one extra useful result within the top three. So one point to gryffindor.
Yeah, I posted an exact search solely so people could see examples of the copycat sites. It's not a good example of a real life query in any way, and not useful for comparisons. It is interesting that Google puts one of the copycats above SO though.
Google ("code that protects users from accidentally invoking the script when they didn't intend to")
Link: https://www.google.com/search?q=%22code+that+protects+users+...
SO - #2
Bing ("code that protects users from accidentally invoking the script when they didn't intend to")
Link: https://www.bing.com/search?q=%22code+that+protects+users+fr...
SO - #2
Brave Search
Link: https://search.brave.com/search?q=%22code+that+protects+user...
SO - Not on page
You.com
Link: https://you.com/search?q=%22code%20that%20protects%20users%2...
SO - Doesn't load
DuckDuckGo:
Link: https://duckduckgo.com/?q=%22code+that+protects+users+from+a...
SO - #2 (seems to depend on refresh)
Basically they're all the same. Google is faster, but the order of the results is identical.
If you did a large scale analysis in this manner I doubt Google would lose.
It was helpful solely to show what some of these leech sites are.
Searching for (without quotes): What does if __name__ == "__main__": do?
Is probably a better test of which search engine has better results for the real-life query. Google might still win, but it should do a better job of screening out the spammy sites. It used to be better at this.
It’s obviously a very difficult problem.
If what you are saying is true it should be trivial to give sample queries so we can test across all search engines and see if it’s true.
None of this is in dispute, so I don't feel compelled to dredge up old screenshots or examples.
Comparing the page layout to competitors isn't especially helpful when Google has 90%+ market share in search. Google is defining the standard for others.
Can't say I really get your point. You're willing to complain and go on this long tirade, but not to give a single query example? The one you gave in your original post was already debunked easily enough. You're saying these things as if they are a fact - I disagree with you, it is in dispute. If we're going to just take random claims are facts, then I'll say Google's results are better than ever.
Google's market share isn't really relevant. It's very easy to just use Bing or any other search engine. I actually use Bing half of the time since it offers rewards.
As for a query with a shit ton of ads and widgets? Try vegas hotels, or anything else with lots of widgets and ads. The travel space has a lot of them.
I don't happen to have a historical screenshot history of SERP results specific to StackOverflow copycats, no. I'll concede that.
What, lol. I'm not making any claims. You're the one saying Google is becoming complacent. I'm asking for proof of this, lol.
>> I believe they've become more complacent since there is no competitive pressure, yes.
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/newsguidelines.html
So they at least have to keep their quality even with the competition for the average person (HN readers are not the average person).
https://www.eweek.com/news/microsoft-bing-beat-google-in-soc...
I think someone or an entity has been engaged in a consertive effort to manipulate the results if its not something more nefarious in Google and Bing's domain.
Very few entities have the resources to do this either, its not something a ragtag band of goat herders could, thats for sure!
This seems pretty suspicious? Is it reporting the first time Google crawled the main domain farath.com? How is that relevant information?
0: https://web.archive.org/web/20080607010730/http://www.farath...
1: https://who.is/whois/farath.com
Though the proposed solution borrows heavily from concepts long used in email and Usenet spam, there are a few critical distinction in SEO SERP[1] spam which both make a widely-crowdsourced listing less applicable and less necessary.
In the case of email, your inbox is an unlimited resource to the spammers --- there's effectively no limit to how much spam they can throw at it. As there are also an effectively limitless set of source addresses (by either domain name or IPv6 addresses), and because email/Usenet spam is itself a quantity/numbers game with rapidly shifting origins, collectively-source and curated blocklists have value.[2]
A SERP is itself a finite resource --- the default is to display 10 results, and not making it into the top ten provides little reward. Moreover, high ranking search takes some effort and time to achieve, it's not like in email where a new server can spin up and immediately start deluging targets.
My experience with annoyances matching this sort (stream-based social media is one example) is that blocking a relatively small number of high-profile annoyances hugely improves signal/noise ratios. And I think that will be the case with SERPs as well. There are a half-dozen or so sites which tend to dominate results in most cases, and those can be individually blocklisted (if the capability exists). If more appear, they can similarly be removed.
The other factor is that quite a few sites which some people find exceedingly annoying and spammish, others find appealing. Coming to agreement on what to block, and classifications of such domains / sites, is likely to be difficult and/or contentious. There may be exceptions in specific instances (hence: specific classifications of unwanted results), but less so in the general case.
I might be wrong. The case of DNS adbloc, with PiHole as the classic example, shows that very large lists can be compiled and used. My own Web adblock / malware block configurations have typically had from ~10k to ~100k of thousands of entries. That said, the really heavy lifting is typically done by a much smaller fraction of the total. Power laws and Zipf functions work in your advantage here.
________________________________
Notes:
1. Search engine results page, that is, what you see in response to a query.
2. Even in the case of email spam, the principle value is largely from curated lists, usually by experts, e.g., Spamhaus.
I suggested a single list to prevent repetition and to limit the imports one needs to make to one.
For malware and the like, repurposing extant DNS-based blocklists as in for uBlock Origin / uMatrix should be viable, and not require an additional curation effort.
Note also that we're looking at a browser extension, and as such, very large lists and memory load would probably carry significant negative impacts.
I think empowering individuals to curate the web would create stronger social and financial incentives to improve online indexing (I.e: Shopify vs Amazon). 20 years ago we could approximate quality from backlinks from credible sites, in the age of social media it seems this signal has shifted towards what creators, influencers, and online experts endorse.
ELI5?
On the other hand it is absolutely ridiculous to conflate the difficulty of occasionally adding a domain to a local filter, and assisting to build a random unproven search engine. People volunteer their development effort for projects they personally find interesting or challenging. If you want more developers advocate for the project don't try to scold people for wanting to spend a small amount of their time refining a solution that works for them.
Plus there’s no private Google API here, just an extension that removes search results from the page. I suppose you could say the extension APIs are from Google (Chromium) but they’re certainly not private and are commonly used.
> I’m not sure why you think that a domain blocklist would be harder than custom search engine development.
I didn't say this. The custom search is already created. Helping it's development is much easier now. AFAIK it's main problem is the lack of hosted servers.
Fair enough about a custom search engine though, I misinterpreted your comment.
[0]: https://github.com/stroobants-dev/ublock-origin-shitty-copie...
Not sure if the X's were to denote wildcards or if you're after porn actresses named Ruby, but either way it might be worth adding a term to the query to clarify
Leaving it to "the algorithm" to decide what to show you based on what it "thinks you want" from previous interactions turns the tool into no better than Twitter or Facebook optimizing engagement by showing you what they think you should see.
For reference, Ruby is only has 6% on the Stack overflow dev survey https://insights.stackoverflow.com/survey/2021#most-popular-...
> DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google).
https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
Their "hundreds of sources" are for very niche topics and appear in separate boxes. The 10 links per page come from Bing which would make them Bing-based.
Kagi isn't trying to take significant market share from Google or DDG, but find a niche (probably Hacker News type people). The one thing I find they're lacking is maps and location based results (e.g. you can't search for "lunch near me"). If you're interested, I'd recommend checking out their FAQ: https://kagi.com/faq
If they're not ready to charge for use yet, a limited preview makes more sense than a public free service.
https://kagi.com/signup?invite_code=adfreesearchengine
Edit: This is really promising and interesting. They show a summary of the link when you hover over the crystal ball icon next to it, together with a button for "Block" and "Boost" - i.e. you can vote with your account. These metrics could be used for ranking and kill affiliate marketers.
[0]: https://news.ycombinator.com/item?id=29772136
https://search.brave.com
Like ddg it's just Bing with a gimmick, but it's decent.
Is that true? I've read images are served by Bing, not sure about the web search.
Coke/Pepsi create, market, manufacture, distribute their soda.
Dr. Pepper just creates and markets their soda. They don't manufacture or distribute their own soda (believe it or not, Pepsi manufactures & distribute Dr Pepper soda).
Instead of a blacklist, a whitelist like a curated list of links à la Yahoo would be better approach, or something with upvote/reputation/karma as on HN.
I never took any further than "what a cool idea".
Side question, is there a max number of sites you can block on that list?
Edit: fruedian misread. I thought this was through ublock origin.
[1] https://github.com/darekkay/config-files/blob/master/adblock...