316 comments

[ 0.23 ms ] story [ 272 ms ] thread
This is such a great idea, and a mystery as to why no search engine has dared to put it on their engine. Care to enlighten me?
google used to have the personal blocklist extension

not sure what happened to it...

I think it was a stop gap solution because Google was overrun with spam sites, like Expert Village, before Google did the Panda update.

https://en.wikipedia.org/wiki/Google_Panda

Isn't that still an issue? For some of the obscure searches I'm still seeing sites that copy content from github/stackoverflow/reddit
It's still a huge problem. Try searching for a product or product category. For example, search for "best winter coat" on Google. Half the sites are low quality affiliate link farms who don't know anything more about the product than what's in their Amazon description.
Those sites are so annoying. I want some know-it-all moms on a forum telling eachother what to buy - not scam sites pretending to review stuff.

Some years ago I got moms. Nowadays Google seems to serve crap.

These days I just search Reddit to get real people's opinions on a product. Way more trustworthy than Amazon reviews or random blogs.
The URLs usually have a certain smell to them as well. "bestcoat.com", "topreviewsz", "theexpertreport2021.net" etc.

CHECK PRICE ON AMAZON CHECK PRICE ON AMAZON CHECK PRICE ON AMAZON

Yes, Google used to have this feature. I was probably one of the tiny percentage of the users who used it. My guess is it's one of those features where very few people used it. So the cost of maintaining a feature like this on a scale like Google search with tiny percentage of users is not worth it.
Yep! It was called “personal blocklist (by google)”.

I don’t think anyone was maintaining it. So when the SERP changed its layout, the extension broke and Google removed it :(

What do you mean it broke? I am using it as we speak and it still blocks great
It hurts Google to allow the product to choose what customers have access to it.
To add to this...

A lot of those super annoying ad-filled websites you get directed to from Google Search... Guess who's providing those ads and making money when you accidentally click an ad while trying to get off the website?

Extend this to map search - If only I could exclude all starbucks, 7/11, even Denny's from my map searches for 'coffee shop'. No hate to those who like the coffee from there, but I want a proper coffee shop and if I could filter out certain chains, I would do it in a caffeinated heartbeat. This applies to not just Google maps though.
At least you can -exclude those names. Gas station convenience stores come up for "grocery store" searches, yet they're all named differently.
Google maps doesn't seem to support `-exclude` like Google/DDG search does.
You're right. I noticed this on Bing Maps before, but hadn't noticed Google following suit.

It makes these sites so much less useful. Their unique location catalog is the reason I usually prefer Google, then Bing, over Here Maps or whatever OSM thing. But it's harder and harder to query it to receive anything useful.

Maybe Google will make you to be able to exclude white men owned businesses in your map searches but excluding brands is not something they seem willing to do

https://www.google.com/business/identity-attributes/ "You can now help your business stand out on Google Maps and Search by displaying the Black-owned attribute, in addition to the Women-Led..."

btw, I don't see a big distinction here if I actually would like to boycott Starbucks so I better boycott Google Maps then.

>You can now help your business stand out on Google Maps and Search by displaying the Black-owned attribute, in addition to the Women-Led

Wow, that's insane. I've always assumed this stuff is made up by conspiracy nuts (like the "War on Christmas"), but there it is right on their page!

This frustrates me to no end. Usually, I search for "independent coffee shop" to narrow results, but then still have to zoom in until my nose is touching the pavement before I can see anything other than large chains. Even worse, if there aren't enough "popular" results, I'll get things that are very much not coffee shops, like Target or Arby's
This would be an interesting app to build on OpenStreetMap data.
This is a great, pithy expression for exactly what is driving the toxic behavior--nicely phrased.
I see what you did there ;)
(comment deleted)
It was a major feature of blekko's search engine.

Google had it for a while, too.

Are you paying for Google search? No. So, its not in Google's interest to do what's best for the enduser (to an extent, after which it hurts them as user engagement dwindles and people move to another search engine).
Yes, you are paying.
i suspect google does not support user configurable spam filtering because spam filtering is one of their big value adds. their job is to return relevant and spam free results in a user friendly manner. if they made client side filtering a thing, and it allowed their spam filtering efforts to shrink because "the users can always filter it themselves" then their overall product would suffer. i imagine also it's a matter of internal pride, if they do their job correctly, you shouldn't need it.
Doubt it, it's more likely that they want to maintain full control over results. They likely have certain results they don't want you to filter. Also, relevance can change over time. You might filter out something now that made sense to filter out now but not in the future after the site changed and got reindexed.
right, their value proposition to you, as a user of their search engine, is that if you type in a few terms they'll present a carefully curated list of locations on the internet that are not only relevant to your search terms but also are useful to you, specifically, at that moment.

supporting user controlled blocklists would implicitly acknowledge that their system can miss the mark. i suspect they focus on not missing the mark rather than providing supplies for their users to paper over problems with their system.

i'd be really surprised if there are "certain results they don't want you to filter." separation of editorial and business in publishing has been around for a very long time and it would be pretty hard for them to defend if it came out that they were using their own ad revenue as an input to how they rank and filter results.

edit: i would also suggest that if they were boosting results that drove ad revenue for them, they'd have been caught by now. there are a lot of people out there who make a living experimenting with how to boost a google ranking. if it was as simple as "use google as an advertising partner on your site" i'm sure we'd be hearing about it.

'if it was as simple as "use google as an advertising partner on your site" i'm sure we'd be hearing about it'

There used to be the same kind of conspiracy theory around G analytics. Hey, guess what?

They mean no way to have a persistent black list you don’t have to set every search.
Couldn't you do this with Chrome search engines?

https://zapier.com/blog/add-search-engine-to-chrome/

Not practically due to the query length limit.

If you take 102 site blocklist from a top comment and turn it into a site exclusion search like this:

  -site:code-examples.net -site:kotaeta.com -site:answer-id.com -site:code.i-harness.com -site:living-sun.com -site:qastack.jp -site:qastack.ru -site:qastack.it -site:qastack.mx -site:qastack.com.br -site:qastack.info.tr -site:qastack.in.th -site:qastack.com.de -site:qastack.fr -site:qastack.cn -site:qastack.com.ua -site:qastack.co.in -site:qastack.kr -site:qastack.vn -site:qastack.net.bd -site:qa-stack.pl -site:qastack.id -site:www.coder.work -site:www.it-swarm-ja.tech -site:www.it-swarm.jp.net -site:www.it-mure.jp.net -site:www.it-swarm-ja.com -site:www.it-swarm.com.ru -site:codeflow.site -site:codeguides.site -site:overcoder.net -site:coderoad.ru -site:www.generacodice.blog -site:www.generacodice.it -site:www.generacodice.com -site:www.javaer101.com -site:voidcc.com -site:siwib.org ...
And then try adding that to a Google query you see this warning message on Google:

> ".... (and any subsequent words) was ignored because we limit queries to 32 words"

If you try it on duckduckgo you just get an error page.

Every time I end up on the pinterest from google, I hit back. Google knows that, but it will still show me pinterest as the top result.
It used to be built into Google. I was just telling someone that Google has been down hill since they removed it, and they were so young they didn't remember it being there.
Most likely because extremely low number of people (by Google's definition) would use it and there are other things to prioritize.

For a feature to be interesting to just 1% of Google users, it would need to appeal to 35 million users and this is unlikely. Even if it did, features that touch 1% users rarely get to the top of queue (unless those 1% users are "whales").

Google actually did use to have this functionality. They almost certainly removed it because it hurts profits (an advertising company that allows it's targets to define content?).

I've been using this extension for a good while now, and it's probably saved me a stroke or two.

For a quick and dirty DIY solution, add something like this to your uBlock filter list:

  google.*##.g:has(a[href*="thetopsites.com"])
  duckduckgo.*##.results > div:has(a[href*="thetopsites.com"])

The linked project goes way beyond this though, very neat!
Thanks for the tip.

That works well enough for me, and it saves having another browser extension to configure!

> it saves having another browser extension to configure!

Not just that, it saves having another browser extension to trust. A large number of the sites I get to are through search results. A malicious extension that has access to modify search results could do a lot of damage.

Hell, I know people who get to their bank and other sensitive sites by typing the name of the institution into Google, rather than typing the URL. Being able to modify search results would be great for phishing.

This is why I always do banking/TDAM in Firefox safe mode. I'd rather be tracked (or even hacked by some malicious script through the bank's fault) by my bank, than hacked by an extension I installed, because I wouldn't have much liability in the former but I would in the prior.
You think they're going to accept liability just because it's their fault? In order to get that far they'd have to investigate and then find themselves at fault.
Of all the bad things banks do, not accepting liability for theft or loss of funds doesn't seem like a very common one. I've never had trouble with any bank reversing fraudulent transactions or withdrawals.
At the risk of making an unsubstantive comment, this is a huge help to me, and I've been trying to figure out how to do this for years.

Thank you!

Glad to be of help!

Forgot to mention that you can also block arbitrary text with the `:has-text(/regex/)` selector.

Block Medium's cookie notification on any domain:

  *##body>div:has(div:has-text(/To make Medium work.*Privacy Policy.*Cookie Policy/i))
Or apply it to search engine results:

  google.*##.g:has(*:has-text(/word combination to block/i))
  duckduckgo.*##.results > div:has(*:has-text(/word combination to block/i))
i tried pinterest and did not work for me

google.##.g:has(a[href="pinterest.com"])

google.##.g:has(a[href="*.pinterest.com"])

dunno. i primarily use duck which apparently does not show that many pinterest results and even that one didn't work. i would have to investigate

try this:

google.##.g:has(a[href="pinterest.com"])

If you're typing strings which include the * character, you must either escape it with a \ (\*) or write it in a code block.

I STRONGLY recommend the latter.

I think what you're trying to write is:

  google.*##.g:has(a[href=*"pinterest.com"])
(I'm inferring that based on the italic/roman text in your pattern line. I'm probably inferring wrong.)

HN's markup interpreter will style any line indented by two or more spaces as code, and not interpret special symbols. The lines will wrap, however (wrapping is a relatively new feature).

I like the idea, but does it come with a default list? I'd be interested in something like this but that includes most known blogspam sites, with a global list like say pihole provides.
Are you referring to default black list?
Kinda - not necessarily on/blocked by default, but a curated, maintained global list available to use, if that makes sense.
My feeling is that by blacklisting like 100 sites you get 99% of the SEO spam.

In practice it should be a winning battle, since it takes time and money to spam site up in ranking and low effort to blacklist it.

I will try this out.

So you're saying Google could clean up the index by simply banning 100 sites?
I don't think it works at the provider level - one person's junk is another person's treasure, and there's always going to be content that could swing one way or the other.
For my typical querries atleast. I haven't tried yet.
I just installed the firefox version and it doesn't.

I wish there was a predetermined list with things that are known to be poor results, like pinterest on google image search.

Not really, an easy way to discover and use other people's blacklists (like ad blockers try to have) is the biggest thing missing from this extension.
Given the nature of domain-level spam, tuning your own list is likely to be both fairly quick and effective.

High-ranking search results don't come easily, and there will be a relatively small set of domains occupying any such slots. Even with turnover, that's going to be viable. Remember, the competition here isn't to flood your inbox from any random domain on the Net, but to secure one of the first 10 SERP results. Odds are good that blocking 10--20 domains will be more than enough.

Your list will also likely vary from other people's.

Can the title be changed to: Ublacklist - Blocks specific sites from appearing in Google search results
(comment deleted)
Changed now. Thanks!

Submitted title was "Ublacklist – the missing button on Google search, IMO". Submitters: please don't do that. It's against the site guidelines: "Please use the original title, unless it is misleading or linkbait; don't editorialize." - https://news.ycombinator.com/newsguidelines.html

Oh yeah! Adding pinterest to the list now!
Recently installed this because I'm learning React and there are so many awful results in the first page of Google search for anything related to React. Many sites that are actually scraped from Stack overflow with no reference to the original post, or Medium posts that are paywalled. Bye bye.
Too late to change name to "ublocklist"?
Why would they do that? The extension implements blacklisting domains.
Part of an industry wide effort to replace the term "blacklist" due to connotations.
GitHub and some other big companies put effort into this, but other than that, it’s a pet project for people with too much time on their hands.

Give it a few years and nobody will remember what the big deal was.

I believe this is also a cultural/political effort to create problems where they don’t exist.
I am part of an industry wide efforts to add more of this term.
"ublocklist" could easily get confused with "uBlock origin", a very widely used browser extension.

Even "Ublacklist" could give the misleading impression that it's somehow related to uBlock or uMatrix.

(comment deleted)
Non-creative name: "SearchDenylist"

Reasons:

* I think of the "u" prefix in this narrow technology niche (browser-based filtering) as almost a trademark of the great project now called "uBlock Origin".

* "Blacklist" isn't a term to be making new uses of in the US. (Given ongoing history of racial injustices, and how the terms "blacklist" and "whitelist" have been used in our field to designate "bad" and "good", respectively. Someone can try to argue the etymology, but we're beyond etymology in connotations of the terms now, and it's just insensitive to insist on saying "black" to mean "bad".)

So I'm not trying to start a flamewar or anything, genuinely curious to learn more about your view here.

Is your viewpoint that "blacklist" is a racial term in itself, i.e. is used in a racial sense? I've never seen that, and my understanding is that's not the case historically, but call me out if I'm wrong and that's your point.

Based on your comments on disregarding etymology, my understanding is that you're not making the first point. I'm reading your viewpoint more as: "we shouldn't use 'black' as an abstract connotation for 'bad' meanings in words", (presumably due to 'black' being also associated with a group that has a history of racially-motivated prejudice against it, but let me know if this last part is wrong).

I'm also curious if your view is that no colors that have historically described groups should be used as negative descriptors, or is this specific to black people in English? For instance, is your view that phrases like "white bread" to describe boring people, or "brown noser" to describe suck-ups should go away as well?

Additionally, should abstract phrases/words involving the term "black" should be done away with entirely, or that only ones with negative connotations should be avoided? For instance, is it okay for "black ties" to associate "black" with formality, or for "in the black" to be used for businesses that are very successful?

I think I disagree that "blacklist" needs to be associated with black people any more than "white noise" noise needs to be associated with white people. The color black has a long association with a variety of different things, from death ("Black plague") to authority ("Black Rod"), from luxury ("Black Card") to villainy ("Black Hat").

One of the many associations of "black" is with a particular racial group, but that doesn't mean we have to attribute all the associations of that word with that racial group.

It seems to me that if the word itself has no history of racial origin, isn't used today in scenarios of racial prejudice, and the association is not related to a racial group, it seems unnecessary to avoid it.

Like I said, though, I'd definitely love to get a different perspective here.

Yes, I think we shouldn't use "black" to mean "bad".

Because we've had so tragic much of that, in various other ways, and it's time to just say stop that, we mean it, unambiguously.

I might be able to explain the thinking better, were it not 2am, but here goes... Imagine a little kid, of any ethnic/racial heritage, who's picked up on some of the prejudices around "black people" and class. Now try to try to teach them a computing class with the terms "blacklist" and "whitelist", and what they mean. Now you can guess where their thoughts are going, so you try to address that, by saying that you could also say the much more clear "denylist" or "excludelist", and also not have to explain that you don't mean to be offensive or hurtful, but you just really like calling it "blacklist" so much, that overrides all other considerations.

Thanks for the explanation, I can totally empathize with someone making a link between a phrase that means something negative and a term which they identify themselves with.

I'm curious about the whole host of other abstract terms which could connote a similar negative meaning if we associated their terms with racial meaning, though. I listed some above, but I'll elaborate here:

* White bread (boring), or white noise (something not worth listening to)

* Brown-noser (a suck-up)

* Yellow journalism (sensationalist news, doubling up on an offensive term for Asian people if we choose to racialize the term)

If the little kid in the example above picked up on racial biases against any of the previous racial groups, and associated that racial group with one of the phrases above, I would expect that they'd be similarly upset.

It seems reasonable, then, to explain to anyone concerned that these are not racialized terms, rather than trying to prevent offense from terms that bear no relation to the perceived slight. I have no objection to the terms "denylist" or "excludelist", but "blacklist" also seems perfectly fine when used in a technical context.

In much the same way, the terms above could all be replaced as well:

* White bread -> Plain bread

* White noise -> Full-spectrum noise

* Brown-noser -> Suck-up

* Yellow journalism -> Sensationalist journalism

* Black comedy -> Taboo-related comedy

If one wanted to use any of the terms on the right, I'd be more than happy to read them (just as I'm happy to read "blocklist" or "excludelist"). It does not follow, though, that I should stop using the terms on the left -- the fact that someone might be offended if they assume that the phrase is racially motivated (especially if it conforms to other racial prejudices they've experienced) is definitely terrible for that person. The solution, though, is to make it clear that such phrases are not racially motivated, rather than to assume that such terms are implicitly racialized.

tl;dr: There are lots of phrases, across different groups that could be offensive if they were assumed to be related to that group's identity, but which aren't actually related. We could choose to remove these phrases from our vocabulary, or we could clarify when terms refer to extremely long-standing color associations vs. when they refer to racial concepts that use the same language. I have no qualms with people that choose to do the former, but I see no reason to be any less okay with people that choose to do the latter.

Nobody gives a fuck
> isn't a term to be making new uses of in the US.

And even if that’s true, why should iorate, a Japanese developer, give a single damn about that? The world is way bigger than the U.S. A few countries’ names can’t even be mentioned without offending certain American sensibilities. Stop exporting this bullshit please.

Personal Blocklist is another option on Firefox
First usecase I can think of is blacklisting those code example websites which are polluting Google searches.
I guess this is useful if you want to filter a number of sites from your results. -site:example.com still works.
Yeah it seems smarter to ask a result without the websites you don't want to see than to filter them out after.
There's a specific extension doing this for w3schools. Congratulations are in order, I guess...? For being so horrible for so long to have an extension created long ago to nuke you, that is. https://chrome.google.com/webstore/detail/remove-w3schools/g...
Could someone explain why they're hated? I don't know much about E3, but used them a few times without issue
I am not sure of the current state of affairs, but in the past (5-10 years ago or longer) w3schools was known for having outdated or sometimes flat out wrong information. They often rank or ranked higher than MDN which is usually the best and most comprehensive source on the subject. So a front end dev would typically want to always see MDN over w3schools.
My w3schools workaround is appending `mdn` to any search for something related to web standards
I have been part of the beta test for kagi search engine (kagi.com) and my favorite feature is you can prefer/mute websites. So for example I prefer MDN so the site is ranked higher if it shows in the search results. Note I’m unaffiliated with them but the product/team is great so rooting for them to succeed.
I love the preference idea... I'm going to implement that in my personal search engine.
I'm using Kagi at the moment and not only does it have nice features like this but it is also a fantastic search engine.

Maybe the best way to explain how fantastic it is is that I think it has a bang operator like DDG but I have never tested it because when I look at the results they are obviously better than DDG or Google.

pretty much.

their contents was never the best or most relevant back then. but they gamed SEO to position themselves higher than most. they are doing better today? good for them. for some of us who have witnessed the whole thing happen, they will be ignored for a long time.

(comment deleted)
(comment deleted)
They’re the best resource for learning web technologies. They’ll tell you in one plain sentence what you need to know, with simple, concrete examples, whereas MDN, on the other hand, is a baroque mess which, after paragraph upon paragraph of rambling legalize will often leave you more confused than when you started.
That's exactly my experience too. I've learnt so much from w3schools. Also being able to quickly try stuff out and tinker with it and see the effect instantly is so handy.
Yeah, as someone (re)learning modern CSS/HTML, I really don't understand the hate they get. For a beginner (at CSS/HTML anyway, I'm an HPC engineer otherwise with lots of low-level experience), I think it's a great resource, with good examples too.

I can only assume they used to be different?

In terms of the site design itself, it's always been like that - the w3schools site has always had the clean, readable presentation (and it's a pretty easy argument to make that it's much more approachable for a newbie than MDN). The main complaint was with the content. In the past there were some pretty big problems with content on w3schools. Often common footguns (of which there are many in Javascript) were not covered and newbies were encouraged to do things that were bad in the short and long term. IIRC one of the more egregious examples of this was sample code that taught developers to write code vulnerable to SQL injection. Supposedly most of this has been fixed, though I don't really use w3schools that much so I can't really vouch for its integrity or lack thereof these days.
They used to publish vulnerable code as examples without any context. For example...

XSS: https://web.archive.org/web/20060722051641/http://www.w3scho... / https://web.archive.org/web/20060722073110/http://www.w3scho... / https://web.archive.org/web/20060722072840/http://www.w3scho...

Unauthenticated sessions: https://web.archive.org/web/20060716141638/http://www.w3scho...

MySQL injection: https://web.archive.org/web/20110412041949/http://www.w3scho...

Nothing directly wrong with this one but prepared queries were not well understood at the time: https://web.archive.org/web/20060718041049/http://www.w3scho...

They were very easy to learn from and thus widely popular, but missed critical security context.

For learning, they're OK. However, most of the time when you're doing real work, what you really want is specs and complete APIs. At that point w3schools in the results becomes a distraction. I'm totally blocking them.
Exactly. I do coding for hobby, just html, css, js; & in past php too. If I quickly need to find some function to do xyz or a tag or property, or to find the capabilities of some function, its params, w3schools is easy to comprehend than mdn (which very rarely I have been to).
I think it's similar to PHP, they're criticized by the vocal newbies who are chasing the newest and shiniest.

For me personally, w3schools has been one of the most reliable sources of reliable solutions that work not just for the latest Chrome build but across many different browsers and platforms.

Given the cumulative nature of HTML, "outdated" is another word for "reliable and established", IMO.

They were much worse before. See the old version of w3fools website: https://web.archive.org/web/20110412103745/http://w3fools.co...

They're better today, but it's still... let's say an equivalent of empty calories instead of a healthy MDN meal.

I don't see how MDN is a healthy meal when they document only shit which works exclusively in Chrome and Firefox, while claiming everything else is "deprecated" and ignoring all other browsers.

More like synthetic junk food, if you ask me.

Some examples would be useful, otherwise it's hard to tell what situations you're talking about.

But looking at a random entry, they list Samsung Internet, Safari, Opera, IE, and others. That doesn't look like ignoring other browsers.

Much lower quality information and shittier website than MDN yet they somehow outrank them on a ton of queries.

I've pretty much always prefix web related searches with "mdn" these days.

w3schools is unusable without ublock, makes the browser/computer crawl
You can also achieve that with this uBlock filter. Just add the following to "My Filters".

  ! Hide w3schools from Google search
  *://www.w3schools.com/*
  google.*##.g:has(a[href*="www.w3schools.com"])
Someone please add support for DuckDuckGo!
Wait, it already works there.

Amazing!

it has it, says so right on the page
what happens if the first N pages google returns are full of only blacklisted sites?

this is literally the end of the web. (day of the spider predator animal)

Some basic filters for stuff that pops up all the time and is never what I'm looking for:

/.pinterest.\../

/.torrent.\../

/.spankbang.\../

/.mp3.\../

/.proxy.\../

/.block.\../

/.addiction.\../

/.recovery.\../

/.detox.\../

/.clinic.\../

/.rehab.\../

/.rarbg.\../

/.hydra.\../

/.123.\../

/.stock.\../

/.linkedin.\../

/.subtitle.\../

/.shopify.\../

/.dreamstime.\../

/.depositphotos.\../

/^(m|mobile|it|es|ja|fr|pl|at|ru)\.\w./

Also recommend TLD filters:

/.\.(tk|photo|dev|page|science|ninja|monster|st|cricket|store|website|help|bg|party|work|xyz|link|camera|computer|click|cc|club|pw|space|info|rocks|review|faith|date|win|racing|site|online|tech|news|life|design|nyc|guru|photography|global|today|solutions|media|world|biz|gq|rest|ga|casa|buzz|cam|ml|fit|ly|it|vip|php|pl|ru|app|name|tr|asia|surf|top|cf|tw|uz|am|krd|is|cz|pt|pk|cl|il|in|hu|hr|si|ee|bid|vn|kw|ro|mx|me|market|nu|how|red|guide|mobi|cafe|tn|xxx|icu|fun|pro|desi|id|live|one|ir|forsale)\/(.*)/

(modify for your personal needs, it'll save you many, many hours in the long term)

dot NYC is full of spam? A pity. Hard to get voting or ferry info here without it. :)
Then edit the list instead of blindly using it? And surely you know the local addresses for such things, or do you search for them using Google every single time you need to travel or vote?
God damn it. I don't want heroin detox; I want heroin.
you really don't...
Is there a word or expression for when people fail to spot fairly obvious sarcasm?
You're losing lots of quality content by removing those TLDs. I'm using 3 of the listed ones just because they're cheaper and found similar ones to be a good indicator of "this is less likely to be blogspam, and more likely a personal space".
Ya, I use them for personal space. My last name is five letters ending in "st" and I managed to snag that three-letter st domain which I was pretty happy about :) I know a lot of people seem to hate them--and a lot of non-tech-types don't always seem to understand that they are domains--but I think they're fun, and I love finding peoples' personal homepages (mine is currently borked, of course, but not because of the st domain).
There is also a lot of junk on those TLDs, especially the inexpensive ones, outweighing the ratio of good-to-crap found on the big few.

Blocking them in search lists doesn't stop you finding them indirectly by them being mentioned in content elsewhere, or from even more direct referrals, so the non-junk ones can be found those ways. I tend to find useful content on .com et al as much those ways as by direct search these days anyway.

I'm not sure how spankbang got on that list but it's hilarious. Is it really a common problem like pinterest and similar stuff on that list?
If there is such thing as being "too good" at SEO, Pinterest is "too good" at SEO. It's a common complaint with image search that Pinterest will cannibalize the results. What will often happen is someone will repost an image from a website or blog to Pinterest and then when you look for images of that thing, you get the image that was reposted to Pinterest instead of the OG image.
It's supposed to be an adversarial system. Why does Google still provide Pinterest search results, given that if you follow them, you're not allowed to see the image?
I doubt Google wants to, but their users keep clicking on Pinterest's SEO'd results for "cute cake pop" instead of whatever non-Pinterest result Google serves up. Basically Google being outdone at their own game in this niche area.
Google special-cases their own stuff all the time. Why haven't they done that for Pinterest?
Who says they haven't? As to why they haven't totally blocked Pinterest, that would be pretty controversial given Pinterest paints inside the lines.

edit: oh, I see the confusion. People misunderstand what Google image search is, how it works, and are blaming Pinterest for Google's ineptitude.

> given Pinterest paints inside the lines.

Do they? They advertise images in their search results that you aren't allowed to see when you follow the link. It's some of the most paradigmatic black-hat behavior there is.

You can blame Google for that, how on earth is that Pinterest's fault? Google is the one choosing an arbitrary thumbnail out of hundreds of images to represent "pinterest.com/cake-pops". Then Google uses that thumbnail to refer people to Pinterest, but doesn't tell Pinterest what thumbnail they used. Pinterest's hands are tied here. This is Google's fault, 100%.
Well, considering that a user can't view the image, Pinterest is showing images to Google differently than to the average user. What Pinterest does by hiding the images is specifically against Google's policy. There's no question that Pinterest deliberately went out of their way to create the currently existing paradigm, to suggest their hands are tied and they are somehow a passive victim of Google rather than an actively malicious actor is ludicrous.

It may well be the case that this is Google's fault and they are a malicious actor here as well as Pinterest, but Pinterest is blatantly malicious and does not draw within the lines.

You're seeing the same images that Google does, it's just that these are dynamic pages that change over time. Google might only crawler pinterest.com/cute-cupcakes/ every day, for example. Meanwhile, Pinterest refreshes the content of that page every day, but at a different time than Google crawls it. Boom, you click on the image google showed for "/cute-cupcakes/" and what Pinterest was showing is no longer there to be found. Both Google and Pinterest are misbehaving here. Google is using Pinterest to beef up its image search. Pinterest is using Google to get traffic. But Google won't share any of its precious data with other companies unless they pay for it, so Google doesn't pass anything like a hash or X-ORIGINAL-IMAGE-URL in the referrer headers when you click on a link, so Pinterest's hands are tied, sort of.

Anyway plenty of bad behavior to go around. Both Pinterest or Google could probably solve the issue with better engineering. I put a bit more responsibility on Pinterest, but I'm not giving Google a total pass here either.

People will correct me if I’m wrong, but I believe the practice of showing an image in search results and then hiding that image behind a login wall when users click the link would qualify as cloaking [1], a practice Google specifically prohibits [2].

[1] https://en.wikipedia.org/wiki/Cloaking

[2] https://developers.google.com/search/docs/advanced/guideline...

Unfortunately this and similar rules are not well enforced. The only high-profile case I can remember was when they delisted BMW for a short while for blatant keyword stuffing.
This happens because Pinterest is better at categorizing images than Google is. Google should just buy Pinterest and integrate them directly into their image search. Or partner with them so this doesn't happen. Or they could do what Bing did and just copy Pinterest feature by feature.
interesting, does that mean that there is a part of 'active' pinterest users who take time to add tags to pics, letting pinterest to categorize a lot of pictures and let the google ranking higher?
At the moment it's one of my favorite websites of "this type", because they haven't done the content purge yet like the main sites did.
(comment deleted)
Ah yes the notoriously spammy .dev TLD.

I guarantee you that .com, .net, and .org have far more spam and low quality sites than this stupid list.

Yeah, some people are morons and stuck in the original "If it's not one of the old domains you're not a 'real' website" mindset. Kinda makes me glad to know he won't accidentally stumble onto Flutter though, since bother their package manager and main site are .dev :P.
I'll add that if you're getting unwanted ads for that many drug addiction recovery-related terms constantly, perhaps your habits do merit examination.
The original generic TLDs have many highly-ranked legitimate sites.

The problem with extended generic TLDs is that the spam sites have managed to become highly ranked. Yes, there's good stuff there, undoubtedly. But search engines can't find it. At which point it may as well not exist.

I'm not a fan of outright blocking at this level without good cause, but it could well be worthwhile to block an entire TLD in cases.

I'm using .xyz domains for my side projects because they are cheap. Guess they are because they look "spammy" to some.
Interesting thing about .XYZ is that if you try to send someone a link with a .xyz url via SMS it usually just gets quietly spam filtered out by the carriers and never received.
.dev blocks a lot of personal blogs
Almost all personal blogs fall under that tld filter.
That's a whole lot of TLDs to block. .me is very common for personal pages/blogs/emails. .app is used for a lot of legitimate sites as well.
This "recommended" TLD list would block my bank site, government tax service and most of news sited since my country TLD is blocked here. Hilarious.
It is only a suggested list. Edit as needed depending on locality. I don't expect it would inconvenience you if it blocked my local bank or government tax service.

Always review blocklists rather than applying them automatically as your needs may vary from their creators'. No one list is going to be right for everyone, so criticising one for not being right for you is a bit "me me me me".

You need to escape your * characters, by prepending them with \ like \*, to stop HN interpreting them as markup intended to control italic/not.
For those interested in this functionality, Kagi search engine (currently in beta) has this built in natively.

The counter-part feature (domain boost) is also available.

https://kagi.com

(comment deleted)
> Our searching includes anonymized requests to traditional search indexes like Google and Bing.

Are they sleeping with the directors of Bing and Google Search to be allowed to do that, or do they use one of these shady companies that proxy google search through hacked computers ?

There is an API for both.
Just a few month ago HN consensus was that "blacklist" is a bad bad word so it should be named UdisallowList.

Funny how this virtue signaling movement died within a few month.

Has one of you smarter people looked thru the code to attest (honor code, I won't sue you) the extension is not shady with it's use of "Access your data for www.google.com" and it's "may use" of "Access your data for all websites"?
I hope someone with experience could do that. Also it passed apple store review process which give me some feel of safe ( hopefully it is not misleading)
Their review process has been shown to be a theater, it's meant to instill a dependency; the only way to know it's safe is viewing the source code or asking someone to do it for you.
I have wanted this for such a long time. Thank you.
As others have said, it can also be done with a uBlock filter.

For example, here are my filters that block and remove terrible copycats/translations of StackOverflow https://gist.github.com/quenhus/6bd2c47e5780f726f0c96c0a2ee7... . I hate these copies that somehow manage to be better referenced than StackOverflow.

I didn't even think to try and do such with uBlock, and this precompiled list gives me all the more reason to do so. Thank you!
An uBlock filter list is the correct solution to this problem. I want to keep browser extensions which have full access to all website contents at an absolute minimum. I only use extensions which are available in the official Arch repos (like firefox-ublock-origin).
That is specifically the only thing I've ever used this for. The worst offender hides the "solution" behind a paywall.
That's wonderful, thanks for that list. I hope that someone else have figured one of github/gitlab issues clones
This is amazing. I tried extending this to remove all pinterest results too (the real cancer of the internet) but the filter doesn't seem to be working.. Any tips?

    google.*##.g:has(a[href*="pinterest.*"])
PLEASE MAKE THIS WORK.

caps intended.

It doesn't work because = is looking strictly for a substring, it doesn't do globbing. Something like [href="pinterest."] probably gets closer to what you're expecting.
(use \* to escape the *)
Nuking Pinterest off of the Internet is what immediately came to my mind as well!
Seriously, how does something so hated continues to exist?

Is some evil billionaire secretly bankrolling Pinterest as a cruel joke?

Anecdotally, when I've mentioned my hate for Pinterest appearing in search results, several of my coworkers have reacted with surprise. They use it regularly, and mentioned something about pinning interesting results. Our individual minds boggled at each other.
Google will have all those signals that people like using Pintrest and will keep it on the first page.
You know, maybe it's really good! I never considered this possibility because of its hostile UX.

Any service that pops up a login and won't let you access any content without logging in I just nope out of and have for many years. Especially user-hostile on mobile (Twitter and Reddit websites work really really hard to force you into using their apps and/or logging in on mobile, much more than on desktop). But maybe we're missing something and Pinterest is super awesome. Maybe I've been using this anti-user UX pattern as a signal for "crapware" but it's not accurate. Maybe fantastic services are hiding behind this pattern.

I'm not gonna sign up to find out but it's interesting to think about.

Or maybe I'll setup a VM for this and finally get FB/Insta/TikTok/Pinterest/Twitter, check em all out, and find out what the rest of humanity has been up to.

This is a classic example of the HN bubble. Nobody of my "tech" friends uses Pinterest, everyone who's not in that group uses it heavily for finding furniture, clothes or recipes. It's usually the app they use instead of googling for something.
My former manager worked there..
As much as I hate walled garden sites like Pinterest, Quora, Instagram it is huge for looking up clothes, recipes, jewelry.

So it serves a function for non-tech people. Its format works for them.

Using it daily for inspiration for radio controlled cars and trucks I scratch-build from styrene. I also use it for interior design ideas, fashion and if the odd pitcure of a VW T4 van build pops up I tend to save to a collection for when I start my own conversion in the spring.

I love it.

The only people who hate Pinterest are computer nerds, which are an extremely tiny minority of the population, and not Pinterest's target user anyways). Everyone else either likes it or doesn't have a strong opinion on it.

Pinterest is very popular in my friend group (which contains zero computer nerds outside of myself).

> Seriously, how does something so hated continues to exist?

Something hated by billions of nonusers (but not to point of outlawing it) and liked by 25 paying consumers can happily survive as a business.

And Pintereset is actually liked by many people who have user accounts there.

would also add IG and FB to the list.
Sooooo so many hobbies have moved from forums to first Facebook, and the last few years IG. One of my girlfriend's workflows is research on IG, buy on Etsy.
Pinterest is a hate/love relationship. Sometimes it's like a kind of archive of things which have disappeared on other sites (imagine certain clothes you cannot buy anymore). I actually like that they really make a copy of the content. But of course this is totally non tech related.
I wouldn't call it an archive, it's more of a fragment. There's usually no context or link back to the source so it merely exists as evidence that you're not insane.
haha, I like this reply "... evidence that you're not insane" :D
I am using these:

    google.*##.g:has(a[href*=".pinterest.*"])
    google.*##a[href*=".pinterest."]:upward(1)
Thanks this seems to be the closest. On the search results page the Pinterest results still appear, but without a link to the site, so you see floating paragraphs. On the image results page the pinterest results do seem to disappear.

I tried setting the upward() to 2, but that got rid of all the image results.

Excellent, thanks. I am so glad to have that pinterest shit out of my search results. Companies that try to take over your life, even in small ways, should all fall apart.
Thank-you so much! It's been a disease on my browser for too long
Ok, genuine question - Why do people hate Pinterest so much? I have no feelings towards it either way (I don't have an account on it but I have viewed some things over time)
My hatred is due to the fact that if you image search for something, e.g. a product, click on what you think is the product’s site, however you end up on someone’s Pinterest board. From there, there’s no way to get back to the original site. And the biggest annoyance is that the search results always seem to rank better than originals.
Imagine the following scenario: you need an image for an internal presentation, so you don't care about copyright. You find an image in the Google search results that looks promising. You open it, but end up on the dreaded Pinterest site.

You'll know for sure that there is no chance on above average quality or size. Hovering over the image will overlay a "join us" message. If you click to image you get a "join us" modal. When you right click, you get a custom menu, that does have a "Save image" option. But clicking that will just get you the same "join us" again. You'll need to inspector hack the image out, which can quickly become annoying if you need more than one.

Beneath it will be a whole bunch of also promising looking images, but scrolling down a bit will quickly get you, you guessed it, a "join us" modal. Clicking on any image there will get you to another page just like it, but this one often opening with the "join us" modal already open.

If you use Ctrl-click, to save your position in the overview that you inspector hacked the scroll block modal out of, tough luck: that behavior is modified as well. It will just open the new page like a regular click. Go back and you're at the top again.

The site feels like a collection of dark patterns hijacking your image search results.

You're right, it's been a while since I went there. I forgot about all that shit.

I'm sold.

Death to Pinterest

The irony is that the Pinterest users who are putting this nonsense on their profiles are usually uploading copyrighted images to begin with.
Exactly this! If pinterest had a moral leg to stand on I might have some sympathy, but they have no implicit rights to the media they're spamming search results with.

Imagine if Instagram or Facebook tried the seo garbage pinterest is doing to completely take over image search results... they'd get shut down hard, and there would be screaming matches between c class movers and shakers.

I honestly think there's gotta be a kickback or individual level corruption involved, no other site would be allowed to break Google's image search functionality and reputation. It's not like they can't simply downrank and spread out the results. The pinterest situation is fishy af, and it's been years. Google image search used to be useful. Now it's annoying.

Would you submit this to an the awesome list I’ve made for uBlacklist?

I love uBlacklist, but I felt the community aspect of it was missing, and vital to its success. I felt bootstrapping lists and GitHub repos was the most instantaneous way to enable a community.

I comment each time it hits HN, but beyond that haven’t seen much uptake / contribution towards building blacklists.

https://github.com/rjaus/awesome-ublacklist

And awesome list is a great idea! I have a huge blocklist of those fake machine-translated e-commerce sites that just redirect to AliExpress. I'll try to remember to submit it later today
Isn't this equally possible with a uBlock blocklist?

I already trust gorhill, and a bunch of Firefox maintainers, with my life and my passwords.

I'm guessing you haven't tried it. uBlacklist is purpose-built for search filtering, so the UX is far better - It pretty much feels like an official part of Google Search. Next to each result is a single-click block button and the results can be shown/hidden with a button in the top options bar. uBlock gives you all the building blocks, but you have to assemble and manage them yourself.

As for trust, that's a more general issue. Personally, I can't imagine someone would go through this much trouble just to ship a trojan and I find the probability of their GitHub and AMO account being compromised quite unlikely since there are far more lucrative targets for such an attacker - including uBlock Origin. I guess there's always the possibility of auditing the code and installing a self-signed version with no automatic updates.

> Personally, I can't imagine someone would go through this much trouble just to ship a trojan

I agree it’s not very risky, but I can absolutely imagine this.

Thanks a lot. Pinterest and the Stack Overflow translations are a great start. I'm using that + *://*.quora.com/* now and feel like my web experience is already a lot better :D
I do that to remove Medium websites from my Google search. It's a disgrace to the free internet and the amount of garbage on it is staggering.
Thank you. Came here to say the same thing. It's a blight on the Web.
Sometimes there are good articles on it, scribe.rip is pretty handy for reading those few.
Is there a way to filter out sites that use their own domains but are medium websites ?
Thank you for this filter. Much appreciated!
Thanks! These garbage sites have been sneaking towards the top search results the last months.
1. How do I subscribe to these lists in uBlock?

2. Often, the original website (Github, Stackoverflow) is not present in my search results; in such situations, I prefer seeing the copycats to seeing nothing at all. Wouldn't using these black lists be counterproductive for me then?

github-wiki-see.page is a new one I came across today.
I don't know what to do with it. You can't find Github Wiki results on search engine because of Github's robots.txt. The project is thus quite legit. Source: https://github-wiki-see.page/
Will Google kick that out of the Chrome store?
Google literally had an official Chrome extension called “Personal Blocklist (by Google)” serving this purpose in the past.
had? What happened to it?
It broke, then eventually got delisted. Presumably the person working on it left and no one cares to work on something that clearly won’t lead to promotions? There’s a “Personal Blocklist (not by Google)” now.
Google used it to detect the SEO spammers, improve the algorithm for a while, and now it seems that spammers have found a way around that again because in the mean time maintaining it wasn't worth it inside Google.
(comment deleted)
I would love something like this for Google News, to block stories by keyword. Supposedly the functionality exists within Gnews itself to do this, but it doesn't work, never has, and I expect it never will.