HN Explorer

Google algorithm change launched

688 points by Matt_Cutts ↗ HN
Earlier this week Google launched an algorithmic change that will tend to rank scraper sites or sites with less original content lower. The net effect is that searchers are more likely to see the sites that wrote the original content. An example would be that stackoverflow.com will tend to rank higher than sites that just reuse stackoverflow.com's content. Note that the algorithmic change isn't specific to stackoverflow.com though.

I know a few people here on HN had mentioned specific queries like [pass json body to spring mvc] or [aws s3 emr pig], and those look better to me now. I know that the people here all have their favorite programming-related query, so I wanted to ask if anyone notices a search where a site like efreedom ranks higher than SO now? Most of the searches I tried looked like they were returning SO at the appropriate times/slots now.

202 comments

[ 2.4 ms ] story [ 205 ms ] thread
badwetter
Bravo!
Matt_Cutts
I'll be curious to collect programming-related queries where we're not returning Stack Overflow or some other site that we should. Computer science and programming queries are easy for engineers to assess and say "Ah, here's something we need to do better on."
dminor
What's the best way to get these to you (after this article has dropped off the front page)?
Matt_Cutts
I'll still circle back to this page for quite a while, or you can tweet them to me. If it's long feedback, you can blog it and tweet a link to the blog post.
cryptoz
This comment made me chuckle. I agree all this should be discussed out in the open, but blogging and then tweeting links to the blog post....what happened to email!? :)
Matt_Cutts
I get way too much email already, but the main reason is that email is a poor use of my cycles for doing support. In the 10 minutes that I would use to reply to one person, I could make a webmaster video that 1000+ people would benefit from, or 1/6th of a blog post that could answer other peoples' question for a year or more. Even Twitter is 1 to many instead of 1 to 1. I do get and reply to a ton of email, but when possible I try to communicate in ways that will help multiple people at once.
mceachen
This is the best legitimate advertisement for twitter that I've ever read.
Natsu
As the guy in charge of fighting webspam, maybe this will help you find or improve upon some algorithmic means of detecting when lots of unrelated wikis start to get exploited to feed up advertisements.

http://www.bay12forums.com/smf/index.php?topic=73045.0

Apparently, for a short time (until the admins fixed it), a wiki for a game I play was hijacked to spam Google. Actually, said wiki (magmawiki.com) is under attack right now from spam user accounts. I don't run the place, but I thought I'd pass the information along because it might give you something to examine and because I have to imagine that anything that helps Google identify spam-hijacked wikis would be helpful to everyone.

[dead] flawawa2
"We" is Google, could someone add that to the title? I was really confused what kind of algorithm would scan submitted pages at HN and rank based on that...

On topic: Thanks! I would have thought Google already did that. Don't you also weigh in the age of the content? So if you have seen something earlier somewhere else it would be the "master" and rank higher?

alexsherrick
thanks matt this is awesome!
[dead] mebigfatguy
now if i could only get java 1.6 javadoc to show up, instead of java 1.4.2 javadoc, when i type in any arbitrary java class name.
bhavin
I would like to point out one interesting thing I noticed today. I was looking for "gcc optimization flags for xeon".

Following query is from google.com and contains no efreedom on the front page. http://www.google.com/search?hl=en&q=gcc+optimization+fl...

Now, the same query from google.ie (ireland site) contains 2 efreedom on the top page!

http://www.google.ie/search?hl=en&q=gcc+optimization+fla...

Why this strange search behavior to a query which has no relevance to user's location?

P.S. I was logged on to my google account while searching, not sure if that has any effect whatsoever.

bhavin
UPDATE: I am even more surprised to see that when I logged out of my account and searched again on google.ie, one of the efreedom disappeared!!

So, does this mean that Google search now takes into account my past search result clicks (or rather mis-clicks)? Or ranks contents in someway that proves efreedom is somehow more relevant to me?

Matt_Cutts
If you're using personalized search, we do look at past clicks to change your ranking. So if you clicked on efreedom in the past, that could affect your ranking. You can add "&pws=0" (or use incognito mode in Chrome without logging in) to turn off personalized search and see whether that's the factor for you.
andybak
If I'm clicking around to crappy sites to figure out why they rank highly I don't really want them to get even higher in my personal search ranking!

Is deleting them from search history enough to correct this effect?

bhavin
Adding &pws=0 turned the personalized search off and hence got rid of the efreedom.

Also, I cleared my web history and searched again. The results are same now signed in or signed out. Problem solved! :)

Thanks!

Matt_Cutts
Glad to hear it. &pws=0 is handy to diagnose whether personalized search is causing something to rank higher for you.
underdown
It's fairly trivial to add a custom search engine to chrome or ff that automatically appends
sjs382
In Chrome, do this by adding an engine with the following URL, and make it default if you want:

  {google:baseURL}search?{google:RLZ}{google:acceptedSuggestion}{google:originalQueryForSuggestion}sourceid=chrome&ie={inputEncoding}&pws=0&q=%s
jjclarkson
Wow, you learn a lot around here if you're paying attention! I often wondered whether or not my results were being personalized. Good to know.
mol2103
Google has been doing this for some time now.
mbrubeck
The two links show the same results for me now (logged in). Perhaps the index/algorithm changes just took a while to be deployed to all servers.
Matt_Cutts
The high-order bit for me is that Stack Overflow is #1 for both pages. We can do localization based on TLD: the query [bank] will return different results on google.com vs. google.ie, for example. Without doing a deep dig yet, we might have thought that open-mag.com was more helpful for European searchers, for example. It looks like that site is based in Portugal.
tomrod
Kudos for keeping locational variety. Is there a flag/keyword we can enter into the search to ignore location?
furyg3
Such a flag would be reallllllly handy. I constantly switch between google's .com/.nl sites based on what language i'm searching in and what I'm looking for.
Sephr
Some people may recommend http://www.google.com/ncr, but have you tried actually asking for English when you request pages on Google? It sounds like you don't have your Accept-Language header set to English. If an Accept-Language: en header doesn't fix the problem, you should complain on HN (with a new thread) about it instead of using /ncr. Google really should support that.
bretthellman
Fantastic though I'd rather see google block sites like efreedom all together.
jjclarkson
I think that would be a dangerous precedent to set. Whose opinion should decide what sites add value to content that may not be completely original? I'm not saying efreedom is not "being evil", but I could foresee someone somewhere using another's original content and making it more accessible. For example someone could easily improve the accessibility of the content from experts-exchange.com.
benologist
Google already makes that decision about lots of search results, usage on their services etc.
RealGeek
Will this change effect sites like filestube.com and freshwap.net? FilesTube ranks for majority of the long tail keywords, even those not related to downloads/torrents/rapidshare.

I see filestube's auto-generated search listing pages ranking on Google all the time. Pages like: http://www.filestube.com/m/matt+cutts http://www.filestube.com/g/google+scraper

Same goes for freshwap: http://www.freshwap.net/387/dl/Google+Matt+Cutts

These sites will give out an auto-generated page for every keyword you enter into it. Apparently, Google loves to index them... there are 126 million pages of files tube indexed in Google. I thought indexing search listing pages of other search engine was against Google's policies.

xenophanes
There's some really annoying torrent sites like this. I mean, sites that pretend they have search results for whatever torrent you're searching for. Those show up on a google a lot and they're useless.
moultano
This is another class of problems we're working on. Expect some changes here in the next few months.
skinnymuch
Your account page doesn't say anything. I assume you work for Google?
moultano
Yep, in fact I wrote the change we're talking about in this thread. :)
skinnymuch
Hah. Good to know. I guess I'm your 'enemy' then since if there was a label for me it'd probably be blackhatter. Though my 'spam' tends to be of much greater quality than what most of the stuff BHW sort of people produce (I have some original content written and excluding my bottom of the barrel sites, the others have their automated parts like scraping edited/checked by a hired person).
rhizome
Though my 'spam' tends to be of much greater quality than what most of the stuff BHW sort of people produce

You threw a vitamin pill into a bucket of mud?

[deleted]
(comment deleted)
RealGeek
It is acceptable to rank these websites for keywords related to torrents and downloads. But, those auto-generated pages has been ranking even for keywords not related to torrents/downloads.
tristanperry
Sounds good, thanks for the update Matt. I must ask though: I wonder whether you've seen the "January 26 2011 Traffic Change - Back to 'Zombie Traffic' " discussion over at Webmaster World? A number of webmasters there (who own websites with fully unique content) are reporting that they've seen lower quality content sites and/or content scrapers rank above their established sites with good content, starting from around the 26th Jan.

There's no specific queries/websites talked about there (I might be wrong but I think Webmaster World has some rules preventing specific discussion of websites/queries), although I thought I'd flag it up since some webmasters have noticed adverse affects from a Jan 26th algo change; and it sounds like this might be the cause.

Anywhoo, that being said: it's great to see Google continuing to be on the ball and responding to the recent feedback from various blogs and other sources (e.g. here).

Matt_Cutts
I love Webmaster World, but one frustrating aspect of webmaster forums is that specific sites and queries are rarely given. It can be very hard to assess what's really going on since you don't know the specific site that's being discussed.
seanalltogether
Matt, I just went through my search history because i remembered a very specific instance of seeing this. Here's the query.

http://www.google.com/search?q=nstoolbar+bottom+bar&ie=u...

You'll notice that efreedom.com shows on the first page with content taken directly from stackoverflow. While stackoverflow does show in the results, the exact page that efreedom copies does not. Anyway, I'm glad you guys are taking this seriously.

For reference here is what I see right now - http://dl.dropbox.com/u/1437645/googlesearchresult.png

Matt_Cutts
It looks like we've got SO above efreedom for that query, but it's always nice to find a url that we didn't have that we'd like to be indexed. That lets us check whether we can improve our crawling/indexing. Thanks for the example!
chr15
This is why I love Hacker News.
seanalltogether
Yes SO is above efreedom in this instance, but the SO results are actually worse then the efreedom result based on the query.
chaosmachine
This is a situation I've seen many times in the past. Often the right site is on top, but it's showing the wrong result, meanwhile the scraper site surfaces the correct one.

It seems like just showing a few more results from the "real" site would solve the problem.

millerc
If you tell me that sites who chronically scrape content elsewhere get a "pull-down weight" attached to the whole of their content, you've made my day.
bergie
That wouldn't be nice. There is a big difference between copying content from Stack Overflow, to running a Planet news aggregator on a community site for your open source project. The first is an ad scam, the second is valuable service to the members of the community.
AndrewHampton
I'm seeing two SO URLs (5 and 6) before efreedom (8th) when I click that link.
seanalltogether
Yes, but 2 things.

1. The SO results shown don't address the query I submitted as well as the efreedom one does.

2. Why doesn't the stackoverflow article that is being duped show in the results?

[dead] phillijw
Maybe it's being downgraded because it's being seen as dupe content?
jfager
The two SO urls that rank above the efreedom url are for different questions. The identical SO link (http://stackoverflow.com/questions/3977343/nstoolbar-on-bott...) doesn't seem to show up in the results at all.

Two SO links on the first page above any efreedom urls is a positive thing, but even so, if content is apparently considered good enough to make it to the first page, it should come from the original source rather than a scraper.

bhavin
did you try with &pws=0 (personalized search disable) option with the query? That solved the problem for me. I do not see efreedom on top for this query.
[deleted]
(comment deleted)
gregable
For [nstoolbar bottom bar] I see (from an incognito window):

#2) stackoverflow.com/questions/3977343/nstoolbar-on-bottom-of-window

#3) stackoverflow.com/questions/tagged/nstoolbar

#9) efreedom.com/Question/1-3977343/NSToolbar-Bottom-Window

The #9 result is copied from stackoverflow.com/questions/3977343/nstoolbar-on-bottom-of-window, which is the same as the #2 result.

This actually is really hard to do as well as Google has given that the stackoverflow page doesn't have the word "bar" on it anywhere, while the efreedom page does.

Neither Bing nor DDG seem to have either the StackOverflow page nor the Efreedom page. I would give an uninformed guess that they probably can't match the StackOverflow page because it doesn't contain "bar" and are probably doing some sort of hand demotion/removal for efreedom. As a result, you get no pointers to the information you want instead of two.

Tiomaidh
For [nstoolbar bottom bar], searched from my logged-in window, a permalink to this comment is the top result.

Um...

Hovertruck
Clearly by observing the search results, we have changed them.
rick_2047
Ironically, now this post ranks at number 1 for the query. World works in a strange way it seems.
tejaswiy
So HN is in the first spot for this query now. And it redirects to this question. Make of that what you will.
shazow
Make that first two spots.
Confusion
It's strange how these results vary from person to person. When I click that link, I see an eFreedom link (with SO content) at #6 and a different SO link at #7.
jacquesm
Now that's good news. Thanks!
runjake
FYI:

Don't flag this because there's no link or "citation". Matt Cutts is the web spam guy at Google.

sjs382
Here's the accompanying link: http://www.mattcutts.com/blog/algorithm-change-launched/
Matt_Cutts
Yup, I actually posted over here at HN a little bit before I did a post on my personal blog.
jjcm
I'd like to mention that this is really an impressive way of working with the community Matt. Companies say all the time that they value their customer's opinions, but rarely do you see a grievance that's posted on a social news site being a.) initially responded to by the guy who's responsible for it, and b.) amended by that person and his team with a request for review. It's heightened my faith that maybe Google can actually keep that small-company-feel no matter how large they get. I really hope you guys continue in this fashion, and thanks for the search fix.
Matt_Cutts
HN has been a really high signal/noise site in discussing these issues, so it only seemed fair to give folks a heads-up here and see what other issues people were seeing. But thank you. :)
yuhong
Yea, I have recently posted a comment starting with "Any Google employees here" knowing that there is more than one employee that posts here.
miah_
This is awesome! Now if only I could completely remove certain sites from my search results ala the Google Wiki stuff. I'd love to drop swik, and expertsexchange and a few other annoying sites. Its possible this algorithm change will make these sites less annoying to me though.
SkyMarshal
Second that. WTB ability to customize my search results by blacklisting certain sites. Big change I know, but would be a hugely useful service Google could provide.
PaulJDrago
why not just use the - function?

Create something like:

http://www.google.com/search?q=google+-google.com&ie=utf...

I suspect you could hack together a quick greasemonkey script to automatically inject your negative site list when the URL is generated

Devilboy
This only works for a small number of blacklisted domains.
miah_
Because Chrome still doesn't sync your "search engine" settings. So maintaining a massive customized search engine across the multiple computers I use would be a pain. If this was a native feature it would work so long as I logged in to Google. Of course, if they fix chrome I'll be happy too.
gregable
Bonus trick. Experts Exchange actually often has some good stuff on it. The pages are designed to make you think you have to buy a subscription to see the answers, but if you simply scroll down beyond the ads for their service the answers (which aren't scraped from elsewhere) are visible near the bottom of the page. Yes annoying, I wish they didn't work this way, but there are times that I've found experts exchange to be the most useful result.
lukeinth
That may be the case sometimes, but as far as I'm concerned they are spam. I would like the option to blacklist sites from my results. Wouldn't it be great if we had choice.
X-Istence
This only seems to be the case when coming from Google, if you refresh the page or paste the link to a co-worker the results are gone.

As for the long list of crap between the question and the answer, use AdBlock to remove that various DIV's.

Nick_C
> which aren't scraped from elsewhere

They used to be scraped from usenet. They may well still be, but I don't have any current evidence. Are they still doing that?

WillyF
Matt, this is great news.

How about sites that rank well with no content, just navigation? Here's an example:

http://www.collegegrad.com/entryleveljob/entrylevelaccountin...

It's generally a high quality site, but that page has absolutely no relevance to the query except for a title tag and some internal anchor text. The search terms aren't even on the page.

If I remember correctly, it used to rank #1 for "accounting entry level jobs," and now it's down to #8. My question is why is it even ranking at all? It's not even low quality content. It's no content.

[deleted]
(comment deleted)
tivaceous
I notice that you run a competing site to collegegrad.com - namely onedayonejob.com. Obviously you are likely to have paid attention to some of the things your competitor has been doing. But are you sure you're not just using this as an opportunity to stick it to them? If so, that would strike me as a distasteful use of this forum, especially since Matt has been very gracious to give this opportunity to the HN community.
wizard_2
Even if they were direct competitors with the exact same market and product (Which they don't appear to be), that could mean that he's got a much better idea of how his competitors may be doing spammy things then we do. Since these are algorithmic changes not site specific changes any resulting fix would get applied to his own site as well.

The motives here don't offend me.

WillyF
I completely understand why you'd question my motives. I thought really hard about whether I should post this example or not. I tried to find an alternative example that would demonstrate the same problem, but this is the only one that I could come up with. I'm sure if I spent significantly more time looking I could have found something similar, but this example is very apparent to me because it's in SERPs that I watch closely.

I'm not trying to "out" this site. I don't think they've done anything wrong or manipulative—they just have some pages with no content that are ranking well. I think that this type of issue should be on Matt's radar. Most of the site is of extremely high quality, and it deserves the high rankings that it gets.

It's a page that is solely navigational structure, yet it ranks very well for a very specific keyword because of title tags and anchor text. We've already seen that Google has a problem (that they're working towards fixing) with low content from low and medium quality sites. What are they doing about low quality content (or non-content) from high quality sites?

spidaman
Nice to hear about this change Matt. "Fuckin' efreedom..." is an oft heard missive around our office, godspeed in ranking them and their ilk down.
Aaronontheweb
Glad to see that Matt went out of his way to help us geeks and the StackOverFlow community!

In the future is there going to be some way for webmasters to do something like rel="canonical" across domains so if I want to syndicate a piece of content across two properties I own I can indicate which one is the original source? My understanding is that rel="canonical" is only meant to be used between pages on the same root domain today but I could be mistaken.

moultano
Syndicating content across your own domains is a common practice and will not be effected by this.
brown9-2
Is this change restricted to programming-related queries only?

I noticed today that a search for "mubarrek london" returns a page of results where every result on the first page, besides the top one, is spam from www.88searchengines.com, www.30searchengines.com, www.70searchengines.com, etc.

I know this might not be related to topic of scraper sites directly, but not sure how else one can easily report these types of things.

treeface
I can't seem to replicate this:

http://i.imgur.com/kDwPd.png

Are you sure there isn't something else going on?

petervandijck
Google doesn't link to search engines in its results, that's their policy, so I would think those sites will be removed from their index at some point.
giberson
I imagine theres not much you can be specific about when talking about google's algorithm, but can you at least disclose if the identification of "original" content providers is "determined" (by some automatic process) or is it "specified" (manual intervention)?

Stack overflow is an obvious benefactor of this new change, I'm just wondering if smaller content providers might benefit as well?

eurohacker
may be here is a good place to ask - may be someone can answer - what does the term "original content" actually mean , original in what sense ?

if you have a blog about wines, then what do you need to write - in order to be "original" blog about wines - does it mean mostly:

1) express original opinions about wines, 2) original structure of sentence and original wording - but the same opinions that 10 other sites write about, 3) original brands of wine you talk about - original names mentioned, 4) original content in the sense that you have not copied the text 100% from someone else - or what 5) combination of the factors mentioned above 6) something else that makes your content original ..

socialmediaking
In order to be considered "original" from the search engine perspective, it needs to be around 30% new content. There are websites like http://www.copyscape.com/ that can tell you how original something appears.

Internet marketers use software to "spin" text and make it appear unique. Sometimes the text is annoying for humans to read, but search engines eat it up. You can take one article and turn it into 50 with the click of a couple of buttons.

lallysingh
I think this is a much simpler content, as in not being a literal copy of another's text.

To be an "original" wine blog, write your own blog entries. Don't copy someone else's.

[deleted]
(comment deleted)
hsmyers
While I applaud such an improvement, I still despair over Google's inability to handle context. For instance the search phrase 'tex decorative rules' is totally miss managed. First it overrides the search and changes tex to text. Humorously enough, when you counter override back to tex, the search is even worse. We won't even speak of what happens when you change tex to latex--- porno doth ever rise to the top I guess. While we may be in a new millennium, some things remain far behind. And yes I know that things are being worked on from the those who were academic and are now working at Google, but still...
Matt_Cutts
tex is a hard one. And [latex decorative rules] is even trickier--most people wouldn't expect technical documentation if they just saw that as a random query. That's more of an issue to pass on to the synonyms team and not in the scope of this change, but I'll pass this one on to the right folks
runningdogx
Would it be feasible to introduce a search customization setting: "lower the estimated probability that I mistyped"? I assume google uses something like two cut-offs: A < B, where if P(mistype)>A, it suggests but doesn't search for the correction, while if P(mistype)> B, it displays auto-corrected results. Simply an user option to nudge P(mistype) lower would probably suffice, or an option to increase B.

A major usability problem for me with auto-corrected results is the two lines with links: "Showing results for <link>" and "Search instead for <link>". Interpreting those two lines takes me out of the flow of analyzing search results. I haven't yet been conditioned to click on the second link. I never want to click on the first link, because google is already showing me those results.

I frequently search for unusual acronyms, terms, variables, made-up words, and gibberish that shows up in logs, among other things that google likes to "correct" for my benefit. I realize those use cases are not typical, but current auto-correction behavior can be extremely aggravating in those cases.

Auto-suggest a correction all you want, but auto-correcting and suggesting the original is going too far I think, unless there are zero hits, or some equivalently strong algorithmic determination is made that I couldn't possibly want the query the way I typed it.

Matt_Cutts
Personally, I think this would be a good idea. You could learn how tech/Google-savvy different users are and adjust the search results accordingly. Someone who has done a site: search is probably more likely to know their way around Google, for example.
MelissaLB
will there be any penalization for sites that use wikipedia content. Specifically a site that would NOT show up in the same search results as a wikipedia page but a commerce site that uses some unaltered wiki content in the descriptors of the products?

I've seen many sites use this method of adding text to an image heavy site and have watched one particular site that uses this method of adding text, drop dramatically out of the search results since October 28th and again around Dec 28th.

Would you advise to discontinue the use of wikipedia content even though the targeted keywords differ from those of the actual wikipedia page?

[dead] MelissaLB
will there be any penalization for sites that use wikipedia content. Specifically a site that would NOT show up in the same search results as a wikipedia page but a commerce site that uses some unaltered wiki content in the descriptors of the products?

I've seen many sites use this method of adding text to an image heavy site and have watched one particular site that uses this method of adding text, drop dramatically out of the search results since October 28th and again around Dec 28th.

Would you advise to discontinue the use of wikipedia content even though the targeted keywords differ from those of the actual wikipedia page?

Alex3917
Not exactly a scraper site, but if you do a search for "learn to hack" the top result is just a list of SEO keywords:

http://www.learn-to-hack.com/

Several of the other results are rather dubious as well. The reason I bring it up is because the Squidoo lens that comes up is something I made, and while certainly not perfect it's still a much better than many of the SEO spam sites and fake eBooks that rank above it. (And plus the ad revenue is going to charity rather than some shady organized crime ring.)

Anyway sorry if it's a faux pas to complain about my own stuff, but I feel like it's a legitimate problem with the way Google works.

moultano
>Anyway sorry if it's a faux pas to complain about my own stuff, but I feel like it's a legitimate problem with the way Google works.

Definitely not a faux pas. Thanks for the example. My biggest annoyance in threads like these is people who write essays about their site losing traffic but then aren't willing to provide an url for people to check out.

[dead] Enlightenment
The guilty don't want google to know what they are doing <grin>.

Keep up the good work squelching the trolls!

[deleted]
(comment deleted)