Dear Google: please let me ban sites from results
Given the current high-ranking thread about spammy sites in Google results, it strikes me that a very simple solution would be to let logged-in users blacklist sites.
Bam, no more wareseeker or efreedom.
This would solve a lot of people's complaints in one fell swoop.
There are greasemonkey etc. scripts to do this, but they're tied to a single browser on a single machine. A global filter (like in gmail) would be so much more useful.
Would this be particularly hard to do?
216 comments
[ 2.8 ms ] story [ 276 ms ] threadhttp://www.google.com/cse/
Usability-wise, though, it's not nearly as much use as a 'ban' button next to each result would be. But it shows Google already have the infrastructure and code that would allow this -- they just need to make it instant to use.
EDIT: The other downside of this is you lose a load of bells & whistles, e.g. previews, "pages from the UK" (without typing), icons for images/news/etc. Time will tell if I miss those.
"Standard edition - ads are required on search pages"
There's an ad-free premium version as well, but you definitely can get it for free.
http://www.antimoon.com/ce/
It includes only sites which are known to use (mostly) good English. It's designed for English learners/teachers who want to find correct usage examples without risking exposure to Yahoo-Answers-style English.
Do people realise that if Google is your referrer, you can scroll all the way down and see the solutions to the question?
Then I think they went through a stage of not revealing it at all.
But in any case, it's far less useful than SO/SF etc.
Actually, I sort of sympathize with the predicament EE faces. They want to show up high on Google search results, because that's how they get new customers ... but they don't want to give away their content for free.
Here's an opportunity for a search engine start-up: Allow users to search (by default) for only free content -- but also allow them to search, if they so choose, for content that's behind a paywall. Pay-for-access sites would love something like this.
I have no sympathy at all for them. All the answers are community-generated for free, aren't they? So they're trying to charge for other people's generosity. And plenty of sites on the web give away better content than theirs for free, with no attempt to trick people into paying.
Their business model is broken and I'm surprised they've lasted this long.
I do hope those working on the algorithm are taking note.
http://googleblog.blogspot.com/2008/11/searchwiki-make-searc...
Traditionally google seem against human powered editing (as this would be), but I think as the black hat SEOs run rings around them, its needed badly.
What I'm trying to get at is, with all things equal, let's say Stack Overflow and efreedom's SEO is on par with each other, shouldn't SO's reputation/inbound link ranks automatically trump things?
stackoverflow has been my first port of call for programming queries for quite some time. If Google wasn't so filled with scraper junk that probably wouldn't be the case.
https://chrome.google.com/extensions/detail/gledhololmniapej...
SO is not editing the material for SEO, they just have whatever content the users generated.
So yeah it can be done.
Not only do you have to bypass the captcha, but also the heuristics that google could (does?) employ to detect fake accounts.
If the votes were weighted by account activity then the spammers would not only have to sign up and vote; they would also have to keep all these accounts busy with fake queries or other means.
Keeping all those accounts active, cycling out dead accounts etc. could take quite an ongoing infrastructure investment.
Moreover google could give significantly more weight to accounts that look very real (i.e. actively using multiple google products for a long time) and thereby further devalue the botnet impact.
It's yet another arms race of course. But one with fairly good odds I would think.
If you have a botnet, you shouldn't have much trouble doing that.
Of course, anything can be gamed... But it's better than nothing.
If I were to create a web-app with user-generated content, I would try to create a "trust" system based on a "reputation" / "real person" metric. It may come from his StackOverflow / reddit / HN reputation, his Google Account activity, Facebook activity...
Before buying a product based on a review I read on a forum, I always look at the author's post count / history / reputation. That's why I like HN / Reddit and their Karma systems. We need to automate that
Hundreds of millions of searches and visitors in any one keyword niche: not so often.
Many websites live off a handful of visitors a day coming from a few core keywords and associated long-tail traffic. For a keyword that only gets 100 searches a day, it wouldn't take many down-votes to affect the rankings of the relevant sites.
Why do you assume a flawed implementation?
Naturally there would be thresholds. There is no reason to devalue a site that's only displayed in 100 search-results/day at all.
The sites we want to hit are orders of magnitude worse at polluting the results. We're talking about the Mahalo's and expert sexchanges.
For example, for a low competition keyword, 1 link can make a big difference to a site's rankings. For a high-competition keyword it takes many many more links to change anything. It's the same algorithm affecting the rankings in both cases, just in the latter case there's a lot more data being used as input.
That's the way they seem to do things, so I'm guessing there's a fair chance they'd take a similar approach to the influence of down-votes on rankings, if they went down that road.
There'd have to have some sorts of thresholds, but there'll always be points above which the algorithm can be gamed, just like pretty much every other aspect of their ranking algorithm can be gamed.
Right now, in a low-volume-keyword niche, a sleazy operator can kill the rankings of competing sites pretty easily by buying them a few dirty links and letting Google's algorithm do the rest. If a site hasn't got lots of quality links pointing to it, and let's face it, the vast majority of sites don't, it's pretty easy for someone to kill its rankings.
A lot of people say that it's impossible to affect the rankings of someone else's site, but that's simply not true. You can't easily affect the rankings of a big established site with lots of good links, but a little small-business site? The reality is it's pretty easy. I don't think Google have any interest in quashing that particular myth, because the reality is actually kind of scary for small business operating on the web.
(Apologies if I'm going off on a tangent there. You are absolutely right in pointing out that I was assuming a flawed implementation.)
Search "viagra": http://www.google.com/search?q=viagra
This is the third result (after wikipedia + viagra.com which have a hardcoded boost): http://www.genericviagrarx.net/
Here's a sample of links to it: http://www.google.com/search?q=link:genericviagrarx.net
I.e. there are thousands of forums / blogs where you can post your link to (even in a proper context, the more popular the subject the better). Hire a couple of offshore workers for $5 / hour, and in a week you'll have thousands of links pointing to you.
That site you point to has many more links than Google shows. You can check yahoo to get a more realistic count:
http://siteexplorer.search.yahoo.com/search?p=www.genericvia...
But even that is probably only indexing a small fraction of the links that are likely to be pointing to that site. Generally when a site has a few thousand spammy links, they've usually got tens of thousands more that haven't been indexed.
Now I'm not for a second saying that that site should be ranking where it is, just because it has a lot of links. I really wish Google did a better job of discounting spammy links.
Basically they were leveraging off other people's work and charging for it.
1) - It doesn't have to be extremely painful, just painful enough, such that true loathing is needed as motivation. This way, we filter out frivolous decisions. A few seconds pause would be enough.
2) - We need to let the reduced ad revenue do the job for us through the market. Anything else will be gamed much to everyone's detriment. Just empower people to remove the annoyance, and let the money do its thing.
Re 1, painful? WTF? The whole point is to make it quick and usable. I can already blacklist sites the painful way, by adding them to a Google Custom Search page. The whole point is I'd like a quick add-to-killfile button, like email clients have had for decades.
Nobody links to an SO question, except perhaps mentioning it in a blog post. Scrapping sites create lots of links to the question, even if they are low quality ones
Everyone is ripping off someone's content.
And just to be accurate here, SO content is creative commons (created by the community). Are those just cheap words?
These add significant value to the original content IMO
However most of the time that I and most people come across SO it's the top search hit for a technical question. The answers are seldom unique, but instead are usually pulled almost word for word from the online manual for the same.
That is overwhelmingly the value of SO -- it benefits by collecting aggregated content from all over the web. It is, hilariously, by design exactly what many in here are complaining about. In that case instead of an automated process it's a mechanical turk.
Hey I don't care because it gets me my answer, but I can see the hilarious paradox in some of the complaints.
I was once searching on a problem -- one that I had faced years earlier but had then forgotten -- and the top link was a SO QA. The answer sounded oddly familiar, when I remembered that it was actually from a blog entry I had posted two years prior.
In my experience though the sites that are taking the content are ad ridden messes which remove value rather than add anything.
Maybe it's like that in your field, but in Mac dev questions you're fairly likely to get answers from established OS X developers, and even Apple employees.
Also there is this form for reporting spam sites: https://www.google.com/webmasters/tools/spamreport
Integrating the above into standard search results would be difficult unless it was restricted to users with a good "karma". That might be possible in our increasingly socially networked world
Perhaps we need to frame the discussion differently, considering what the searcher wants, rather than "spam-free hits".
If Google use that information to gradually adjust their ranking overall, then fair enough -- won't affect me, I can't see them anyway.
EDIT: Even if they don't let that affect everyone else's results (because of gaming), then I still don't care, I still don't see the crap in my results ever again.
Had they not had such a terrible site design and revenue scheme, they might have actually made the transition from scraping bottom feeder to respectable site.
Good to know.
This sure looks to me like real, original content.
EDIT: how about the courtesy of an explanation for the downvote?
This would be an awesome feature.
People are also a lot less tolerable of spam in their inbox than they are of irrelevant search results.
If they're not looking into integrating that nicely into the existing search results page (not a separate form that the average user will never find or use), especially after all the internet chatter about it recently, then they definitely should make that a top priority in 2011. I definitely don't want them to do a rush job on it though. I don't want competitors to start reporting each other as spam in search results to try and game the system even further. I'm assuming they have anti-gaming measures in place for Gmail, so they won't be completely starting that from scratch...
At best G could use the information as a list of potential spammers and filter domains manually, but I really can't see this being automated without giving the SEOs another weapon.
As a workaround, try searching for "[any widget] sucks" and "[any widget] good".
EDIT: tying this to other discussions on the topic, it's a symptom of Patio11's observation that natural language search doesn't work very well. If you want to find something, you need to paint a picture of what it looks like, rather than asking a question about it.
So for example, I could define -site:efreedom.com as an operator to be applied silently for every search I make.
That's the only negative I can think of - other than that, I say bring it!
I can even do that already with Google's Custom Search, all that's missing is a little 'block this site' button. Instead I have to go and configure Custom Search manually for each URL mask.
Presumably it would do the same exact thing as '-site:foobar.com'.
I never said anything about it affecting other people's results...
If the best and brightest (arguably) on the planet can't figure out how to filter out search with algorithms, what makes us think we can mimic true human intelligence any time soon. (I think it will happen, just not as soon as some claim)
Basically, it's the difference between PvP and PvE.
Why?
99% of users are non-tech oriented.
Those users will not really be aware of the specific problems with the search results, they won't understand the concept of a good vs bad result and they certainly won't bother to tweak/ban/filter their results.
The 1% that do care and are currently being vocal about it will start filtering their results and they will perceive that the problem is solved. They will stop making a fuss.
So now, the complaints have gone away, but 99% of users are still using the broken system, so the good sites that create good original content are still ranking below the scrapers and spam results for 99% of the users.
The problem must be solved for all (or at least the majority) of users.
(And you can't take the 1%s filtering and apply it to all users in some kind of social search because the spammers will just join the 1% and game the system)
I think 99% of email users have not been adequately trained in why or how they should report spam, and even if they were I think most of them would still not care enough to actually do it with any regularity.
When pushed many may acknowledge that they know it exists, they will probably even be able to find the button when asked if given a chance. But they won't remember to do it when they see spam, they'll just ignore it and move on to the messages from people they know.
After all, the real goal is giving people a better, more relevant experience, detecting and removing spam is just one facet of that. Whether it's email or search.
However I don't think it's as simple as changing the button text. Even a process driven UI, like a wizard style interface where you have to click next to progress through each step might work, but users very quickly become immune to dialogs. They don't read them, they just evolve the actions that get them to their goal the fastest, and the users goal does not include reporting their spam.
She's been applying it ever since quite efficiently.
Shift+e for "report spam" would be perfect. Same key, just a bit chorded.
With search results, the spam is more often than not Made for AdSense sites that the average user doesn't realize are pure garbage. Then there are the mass-produced content sites like eHow that most technical people realize are worthless, but the average user loves. It isn't often you see Viagra sites popping up in searches for woodworking. It does happen occasionally though.
So no, I am pretty confident a majority of users would not utilize effectively a feature like that.
I posted this as an entirely self-serving request for a way to make my results cleaner.
Also: If the majority of users do like eHow, then that's a sign it's not spam. But a lot of 'power users' consider it spam. This is an argument FOR personalised blacklists, not against them.
And when it's about search result. People are browsing and clicking through adsense filled "landing page sites". Most of them think that it's their fault that they couldn't find the thing they were searching for.
Perfection being the enemy of good enough, and a common and valued and traditional mechanism to delay product shipment.
And Google might well be able to utilize information from that 1% of users that have sorted that out - 1% of a Really Big Number of searches, factoring for the folks looking to game the search results (downward, in this case) - to provide feedback back into their search results.
I disagree. Let's call it 95%.
>Those users will not really be aware of the specific problems with the search results, they won't understand the concept of a good vs bad result and they certainly won't bother to tweak/ban/filter their results.
So have only people that have enabled the advanced features of Google search ban sites. All of a sudden only people that "get it" are the ones that can ban.
>So now, the complaints have gone away, but 99% of users are still using the broken system, so the good sites that create good original content are still ranking below the scrapers and spam results for 99% of the users.
So we need to use the votes to stop the spammers.
>(And you can't take the 1%s filtering and apply it to all users in some kind of social search because the spammers will just join the 1% and game the system)
Sure you can. If you couldn't then Reddit would be a wasteland of adds, but it isn't. They only have 4 or 5 engineers there and they can write code that will stop vote rings, let alone Google.
It's actually a pretty simple exercise to stop vote rings, unless the anti-vote ring code is open sourced, but even there it should be possible.
What I think really needs to be exploited is a ring of trust type aspect, I'd like to have the Hackernews ring where all us on here work together to remove the spam from our results and let's Google see what are taking out, maybe that will help them improve their algorithms.
Why not apply that reading level algorithm to users gmail data and public social network profiles, estimate the users IQ, and then those at the top of the pile are given "result burrying" moderator privileges.
Confirmed user accounts (cell phone verification) combined with other algorithms, such as profile age and activity, could make spamming sufficiently complex to de-incenvize all but the most illicit spammers.
Users at the bottom of the IQ pile (non-logged in users based on past search data and geo-location socio-economic status) don't even get the option to bury results. Which, by the way I think is more like 20% of US internet users than 99%.
And no, this doesn't solve the problem.
Plus, we are talking about a company whose core business demands that it can identify groups of bad-faith voters. Given time, they may find a way to incorporate this data safely into the ranking data (if anyone could, it would be Google).
And I know there are extensions to do this (mine mysteriously stopped working recently), but doing this on the client-side in a way that's bound to a single browser install just seems wrong to me, especially for Google.
Facebook, on the other hand, has developed a system where nearly every user activity creates a new easily processed and meaningful connection between users or out to the web itself. And those connections are probably closer to representations of some kind of trust than "I email that person a lot".
Anyway, I'm not saying the sky is falling for Google, just that search appears to be changing for the first time in awhile.
As mentioned above, then introducing the shared-ranking via the social graph would be the next logical step. It could be something opt-in'd to ease adoption.
Then, ideally (and this is my personal 'white whale' problem) it would be great to imagine something where the user could whitelist through no action of their own rather than have to do any work to block, i.e. use the result set 'hit' of what's clicked in the results to act as a personal ranking upvote.
There's some interesting engineering issues of per-user indexing though, but hey, you wanted to work at Google right?
efreedom is monetised by Google ads. Might seem like a problem to Google.
Let's say it starts with personal blacklists. Then trusted lists that you can subscribe to (AdBlock-style). Then word spreads and enough people are using it such that AdSense revenue drops 20-30% or more?
(IME, CTR on ads is much higher on these content-light sites than it is on more reputable sites.)
It's to Google's benefit that people end up on these pages, see a ton of ads, and then click on one out of confusion or desperation.
Let it learn what I think is a good result for my needs.
If you make it a little bit social, make sure you weight other people's opinions by how much they agree with my own in other areas (making it harder for sockpuppets to muddy the waters)