As someone who knows folks who got wrongly blacklisted.. Well, you pretty much never get back. No matter what you do.
You're lucky if you get a decent reply from customer support to begin with..
Your best bet is to change domains, fill in a redirect and try to build up backlinks again. It'll take a year or two and hopefully no competitor pops up in your space.
Or, you can try and get Andreesen-Horowitz interested in your business.
I have helped sites with over 2 million backlinks and got them out of manual penalty in under 3 months in very competitive verticals like travel & finance.
It entirely depends upon the scope of penalty. Unless you are completely deindexed from Google search results, you can always recover from a penalty.
Feel free to email me if you have any specific need.
They have the word 'lyrics' at the end of the slug for that post and a number of others which don't have any relevance to lyrics. Surely that's still bad SEO practice?
Very observant... I wonder what Google would say about this. I would almost find this more egregious than their original offense due to the posts having nothing to do with lyrics.
The fact that all our text and song page URLs end in "-lyrics" is a relic of a time when we hosted almost exclusively lyrics.
Making the changes required to host all kinds of annotated texts has been a big project and making the URLs make sense is something we haven't gotten around to.
The reason it's not a totally trivial change is that both users and texts share the same top-level namespaces, and so right now we use the "-lyrics" suffix to differentiate song and text pages from user profile pages.
We want to change the suffix to "-annotated" for non-music pages and will hopefully get around to in the next few weeks. To our knowledge the existing situation doesn't help with SEO (no one is searching "bartleby the scrivener lyrics"), and it's confusing, so we 100% agree that the current situation is bad.
Pretty interesting decision to include a lengthy "technical digression" in their SEO apology:
<< Technical Digression: How to scrape 178k URLs in Ruby in less than 15 minutes - Ok, so you have 178k URLs in a postgres database table. You need to scrape and analyze all of them and write the analysis back to the database. Then, once everything’s done, generate a CSV from the scraped data. >>
As I see it, the "technical digression" serves two purposes:
- first, they use it with Google to show how sincere they were about "cleaning up"
- second, and perhaps more important, it gives Google a public excuse to justify why they restored RG so quickly in rankings. As in, "Oh they did a very thorough job of complying by our standards that we had no option but to restore their status".
Unfortunately in this case Google's response speed comes across as a bad thing. Because it applies to only a very select group of sites that are either very rich (aka large advertisers) or very well connected.
My favorite is that they seemed to have spent a few hours (more than 3!) improving their code so that it would run in 15 minutes instead of three hours.
>Though Google is an extremely important part of helping people discover and navigate Rap Genius, we hope that this ordeal will make fans see that Rap Genius is more than a Google-access-only website. The only way to fully appreciate and benefit from Rap Genius is to sign up for an account and use Rap Genius – not as a substitute for Wikipedia or lyrics sites, but as a social network where curious and intelligent people gather to socialize and engage in close reading of text.
ie, you should really just become a member rather than relying on google...
But this truly saddens me as well. I cannot for the life of me understand how they were able to fix ten-of-thousands of backlinks and other shady practices and get back in the good graces of Google so darn quickly.
Frankly I do not know a single other legitimate(+with good traffic) site who was able to get back in the good graces of Google after algorithm updates without spending many many months researching any possible reasons to be penalized and fixing it themselves after repeatedly asking google for help to no avail.
Could it be that Rap Genius was special because it has Andreessen Horowitz?
Sure seems that way.
I am not going to stop using Google, but I am inclined to use them lesser now.
I would like to hear why, how rap genius was fixed this and what this means for other people who get penalized from Matt Cutts though!!
They're back on Google, but keep in mind that they're seemingly not in a great position. Googling "justin beiber lyrics" doesn't show Rapgenius. Neither does "lady gaga lyrics" or "tupac lyrics". In all three cases, azlyrics.com has the top spot.
> They're a startup, and this was a problem which was either going to kill them or be surmounted quickly. Naturally, they surmounted it quickly.
Google laid out a clear plan of action for them. Is it hard to believe they finished it quickly?
Yeah, most of the other sites I refer to were also start-ups by almost any definition.
They did not quite finish their action plan. They got away on reasonable doubt.
This is the first time I am seeing this defense considered by Google.
I willing to bet any other non-vc funded bootstrapped start-up would never be afforded this consideration by Google.
This rant is not a judgement on Rap Genius, I am actually glad they maneuvered around this.
I am judging Google.
They have killed other start-ups for reasons such as other sites farming and duplication their content more vehemently. Reasonable doubt has never been considered for anybody else and most likely will never be hence forth.
I just want to add that this attitude doesn't even reflect poorly solely on Google Search. Adsense is another product that if you get kicked out of, you have zero recourse.
What about that gmail fiasco a few months ago? Users got blocked out of G services for not providing their real names and had no recourse to get their data for several weeks/months.
Just goes to show that in several cases it's not what you know but who you know.
I have never seen Rap Genius on Google, even before all of this. I've seen Metro Lyrics, AZ Lyrics, Song Meaning, etc. I've gone through pages of Google results because every lyrics site sucks, still haven't seen them. Everyone says Rap Genius is excellent, but I've never heard of them outside of HN. I find that odd.
You have the causality wrong. Rap Genius has Andreessen Horowitz because Rap Genius is special.
In this case, the user experience for a Google user is worse when Rap Genius is removed. Google has some incentive to be more active in fixing the problem there, especially since Rap Genius was cooperative, technically competent, and aside from their "rapper swag persona", genuinely nice guys.
Can someone clarify for me, again, what is so special about Rap Genius? It's just a bunch of stupid rap lyrics. So what? In fact, how are they not being sued for copyright by the owners of all the lyrics they're posting, anyway?
I run across Rap Genius now and then and have always considered it to be spam, just like yahoo answers, ehow, and mahalo pages.
Can someone clarify for me, again, what is so special about Rap Genius?
Why? You've no doubt read 10x reasons people here have said so (in the other threads). Do you just want to argue? If you don't agree, fine, but clearly a large and vocal group does. So what? Move along - find other things to care about.
However, if their licensing deals are anything like what other sites have to pay for lyrics, it's going to significantly (entirely?) cut into their profit margin.
Sorry to say, but your assessment of the site is 100% incorrect. It STARTED as a Rap Lyrics website, but it quickly expanded to be an ANY lyrics website (and they will probably expand beyond that)
The main driving force behind it is that anyone can edit the lyrics, and even better, anyone can ANNOTATE the lyrics to explain what certain sections mean.
I looked up this song one day trying to understand more of the allusions made in the song; I know the title itself alludes to Plato's 'The Cave' and wanted to delve deeper. Because of this site, I can see that Verse 3:
> So come out of your cave walking on your hands
> And see the world hanging upside down
> You can understand dependence
> When you know the maker's hand
is an allusion to St. Francis.
Most of the time when Rap Genius ISN'T the top result for lyrics when I google, I facepalm and go directly to their site.
I should emphasize here that this overly positive post isn't intended to condone their shady, spammy blackhat SEO tactics, but damnit if they aren't the BEST lyrics site by a fair margin.
> even better, anyone can ANNOTATE the lyrics to explain what certain sections mean.
Maybe I looked at the wrong lyrics, but I saw precious few explanations, instead most lines had jokes and/or totally unrelated pictures attached to them. Often enough the jokes is to simply restate what was said in rapper slang in plain english, which is funny for exactly the first time you read it, then it's just spam.
All I know is that they somehow come up in my google results fairly often and usually unrelated to the context of my intended search and it always felt like it was SEO'd junk crapping up my results. I added them to my google filter to keep them from showing up in my searches anymore, along with all the inane commentary from their community discussion underneath each song.
I actually avoid using them because the design is hard on the eyes and I find the way they present annotations to be annoying. I don't think there's anything special about them at all, especially when you consider that the idea is completely unoriginal.
But I'm not trying to use any lyrics site. Rap Genius garbage just always popped up on my (to me)-unrelated searches. People can respond to me with how they're not spam or they're not as spammy as other lyrics sites, but the point is that they pop up in the same frequency and same context as every other link-farm/content-mining crap-fest on the internet, like ehow, cha-cha, yahoo answers, mahalo, paidcontent, and so on.
> Google has some incentive to be more active in fixing the problem there
That's exactly how I feel too. Though the incentive for google is very unclear, the lack of clarity is explored further below.
> You have the causality wrong. Rap Genius has Andreessen Horowitz because Rap Genius is special.
Not saying Rap Genius is not a special lyrics start-up. Not even saying that it would not have been a good candidate for any VC financing them.
Though, Definitely saying that the special case leniency shown by Google sure make it seem like having Andreessen Horowitz is the only thing different between Rap Genius and scores of sites before with this and similar situations.
> In this case, the user experience for a Google user is worse when Rap Genius is removed.
And its not when other good sources are removed because they were link bombed or their content was scraped more than their competitors?
> Rap Genius was cooperative
Find me one person in their situation who would not be. It was to either co-operate or die.
> Rap Genius was technically competent
How?
> Aside from their "rapper swag persona", genuinely nice guys.
I am judging Google.
I have nothing against Rap Genius founders, nor their "rapper swag persona" and I'll believe you when you say the are nice guys.
Having worked for large ftse100 publishers who had sites penalized in algo updates its stinks that such an avowedly black hat site gets of with a slap on the wrist.
On a similar topic, how do affiliate linking programmes even work anymore? Surely Google would classify all affiliate programmes as paid links. Even big ones such as Amazon Affiliates, Commission Junction, Affiliate Window are all being plugged by affiliates, and the vast majority of those affiliates are using templated sites or stuffing keywords in blog posts that are largely irrelevant to the product or service they are plugging for affiliate fees?
Aren't all affiliate programmes technically bad for the publisher after Penguin?
If the folks at Google can figure out how to detect this sort of gaming more reliably, that may end up in a future algorithm update, and no amount of pleading will matter for anyone affected by it - they'll have to fix the problem or continue to suffer the consequences.
> Frankly I do not know a single other legitimate(+with good traffic) site who was able to get back in the good graces of Google after algorithm updates without spending many many months researching any possible reasons to be penalized and fixing it themselves after repeatedly asking google for help to no avail.
Well there wasn't an algorithm update, there was a manual action. Google reversed the manual action after RapGenius stopped breaking the rules and cleaned up its mess. If this happened to you there would be a note about it in Webmaster Tools and you would be able to ask Google to take another look. I'm sure RapGenius had all hands on deck making sure they were up to snuff.
This post is complete bullshit. Of course they have relations. Maybe they cleaned their backlinks, but still I cannot believe how they got back so fast in SERP.
Don't be so spiteful. It might very well be that they have relations, don't make it sound like that's a bad thing. Besides that they also seem to just have cleaned up their act very thoroughly.
And about Franz Kafka, I think he'd turn in his grave if he heard about how petty people like yourself criticize a website that tries to make literary criticism of his work digestible and enjoyable for the masses.
> It might very well be that they have relations, don't make it sound like that's a bad thing.
It is a bad thing here: it shows that Google is apparently willing to act preferentially to favoured businesses, undermining the idea that search results are provided with neutrality.
No it doesn't. It means that Google employees can do stuff manually, when asked by a friend or colleague. It says nothing about Google as a business favouring Rap Genius as a business.
The fact that you can request a google employee in a position to help you out to do so displays enormous leverage.
Imagine a business in Germany, Vietnam, or even.. Texas having to do the same? It's pretty much impossible. This is what happens to everyone else out there.
Which is what the parent is referring to when he says that google results aren't as neutral as they claim to be. They clearly favored someone with leverage in this instance.
But this is the only instance that we know of. So far.
Although I understand your concerns, this is just how the world works. If you know the president of the United States personally, there's a good chance he might spend a little of his good will smoothing out some bureaucratic processes for you. Nothing illegal or abusive, just the president asking official this-and-that why process so-and-so is taking so long for your company.
It's not a question of having to do the same, it just means that knowing a Google employee with some magic power is a valuable asset.
But yes, if you're a company that doesn't have this asset and you make a stupid mistake like Rap Genius did your site might be off Google for a month or two, possibly destroying your business.
I still don't follow how that makes Google not neutral in any significant way.
Okay, I see why you were not able to follow the problem.
>> stupid mistake like Rap Genius did your site might be off Google for a month or two, possibly destroying your business.
Sometimes businesses get wrongly blacklisted. Yes, it happens. They do get destroyed because of this, people do lose homes and jobs over this.
But this isn't because Google takes a month or two to sort this kind of thing out (a month or two is also not really acceptable - too long - given the impact Google has on a business). It's because experience shows that once you are on the wrong side of the big G, there's pretty much nothing you can do to get on the right side. It's no longer a question of a month or two, it's about registering a new domain, and starting from scratch. Sometimes you have to do this even when you didn't do anything wrong. This applies to almost every Google service whether it's Search, Adsense, Gmail, or anything else.
But Rap Genius definitely did something wrong. It was obvious for everyone to see. Yet, they managed to follow the 'recommendations' and get back in the game pretty quick. We know that this is probably because of connections because every business out there that gets blacklisted by Google will make sure that they are 'regulation-compliant' ASAP. Yet, many of them never manage to make it back and those that do take years at times.
Yes, knowing the US President helps, but that's provided you can get your work done without knowing him as well, in a reasonable time frame. Especially when it may make the difference between running a thriving business and being homeless.
Probably the story is missing the help of a highly skilled SEO consultant/company but it's impressive anyway. To get it done so quickly they had to work 24/7. Kudos!
I've followed SEO on and off for years and I wouldn't have thought that their strategy of allowing people to generate and embed links back to the site was shady at all. I even recommended something similar a few years ago in good faith.
They're useful, relevant, and it just seems like good marketing. These things will spread organically and drive traffic even if you remove the SEO benefits.
Could you say that Scribd or Slideshare are doing something similar by allowing people to embed documents in their websites along with a no follow link back to their site?
Allowing people to create links to your website isn't shady.
Paying people to link back to your website on high-value keywords like 'justin beiber lyrics' from blog posts on other sites to artificially inflate your pagerank is 100% blackhat SEO. The payment in this case was a Twitter post about the link provider's website.
The tech part is obvsiouly here to calm down people from Hacker News and make them say "Oh this part was good, after all they are not so bad".
It's just a shitty scraper but they talk about it like it was a revolutionnary script...
There is no way they could get back so fast and so easily on google.
I do not understand why people are offended that they came back quickly AFTER doing a bit of penance.
Should their investors not have made calls? However, it should be noted that they did their part.
One of my favorite quotes is this: "To save a drowning man, he must first give you his hand"
If the Rap Genius guys were incompetent and did not do their part in removing the links so quickly (see main story for technical details) there would be little or nothing their investors would have been able to do about that.
BTW, it also helped that people really found Rap Genius useful. I'm sure many searched had rapgenius appended to it. So the lesson is, make your app so good that when Google delists you, it will make them look bad.
Thankfully?, there is also a precedence for other offenders to use. if you can detail you have atoned for your sins and Google insists your you must do a certain time, you know where to turn to.
BTW, this is Hackernews, and I look forward to the discussion of the (de)merits of how they scrapped and analyzed 177,000 links and not espousing of anger that they survived the punishment.
I'm sure Mahbod (my favorite) and co have learned that you do not go 'daaawging' when you are in a hole. I wish them success.
Just to hijack on one of your points: "it also helped that people really found Rap Genius useful."
This is painfully true, and I know it has been discussed in prior discussions. Rap Genius is without a doubt THE BEST lyrics site available, hands down. Even if the other sites weren't ad-ridden scammy buggy shitty websites, if they got their acts together and actually TRIED, Rap Genius would still be top dog.
I don't care for the attitudes or personalities of the people in charge, and I am absolutely disgusted that they felt the need to scam their Google rank, but I am willing to see that some infractions can be rectified through penance, and am happy to know that they will not be forced to shut down because of this fiasco.
Because it is a double standard. There are thousands of sites which get blacklisted by Google. I'm sure all of them fix their errors really quickly once they realize they are on the black list. And none of them get back on Google nearly as quickly as rap genius did.
If Google applied these rules consistently with all websites, I would have no problem with it, but they don't. This breaks the myth that Silicon Valley is a meritocracy - as it shows that the connections you have are as important as raw merit, if not more
The only problem with this is that there's no real course of communication with google. They make it a matter of policy to not provide customer service.
"in 2003 when Denise Griffin, the person in charge of Google’s small customer-support team, asked Page for a larger staff. Instead, he told her that the whole idea of customer support was ridiculous. Rather than assuming the unscalable task of answering users one by one, Page said, Google should enable users to answer one another’s questions."
http://www.wired.com/magazine/2011/03/mf_larrypage/3/
Fairness does not exist in real life. Screaming "unfair" will not help you achieve anything. It's such a ridiculous situation for me. All these guys are sitting on the line, raging at real life because popular culture brain washed them.
Exactly. Everything about how Google handled this is wrong, in my opinion. I couldn't care less about RapGenius - they're just a symptom of the underlying problem which is we don't have any real competition in the search engine market, and we're paying for it with unfair antics like this. Longer thoughts here: http://peebs.org/2014/01/04/we-need-viable-search-engine-com...
Very much so. Google's decisions are starting to have impacts on other companies' bottom lines which easily run into the millions of dollars.
Just as a thought experiment ... how tempting do you think it would be for Google to do the following:
1) Run a query to identify all companies that cut their AdWords spending over 50% over the last year.
2) Look at which of these companies are now getting significant numbers of click from organic results.
3) Find out what is causing them to rank high organically.
4) Penalize them using a generic message and refuse on principle to answer any inquiries as to why.
Apart from (anec-)data that seems to be pointing towards this already happening; should we be OK as a market with this approach? Especially when considering that the rules are so vague that almost every site is guaranteed to break some of Google's guidelines?
>Apart from evidence that seems to be pointing towards this already happening;
Please provide this evidence. I am getting a bit tired of this misinformation. Organic search and paid search are silo'd. To say that Google favors advertisers in the organic results or the other way round, is simply not true. It would destroy Google's credibility.
>how tempting do you think it would be for Google
It approaches conspiracy thinking. I love that, but I think HN is not the place. You are basically accusing a company of a very evil act, without proper evidence. Do you realize that many Googlers frequent this site? It would be a shame if all they get to read are conspiracy theories and baseless accusations.
>Especially when considering that the rules are so vague that almost every site is guaranteed to break some
Maybe if you want to spam or manipulate, then these rules are vague. Maybe when you don't read the guidelines then they are vague. I bet that if you can point to a rule that almost every site breaks, without them knowing, then Google will adjust that rule to be more clear.
You can not please them all.
>Penalize them using a generic message and refuse on principle to answer any inquiries as to why.
You can certainly not please the spammers that were caught with their hands in the cookie jar and then take to the internet to say that Google is unfair, and that the big brands get away with anything.
If I think of a lot of things my users would search for and try to create valuable information for them concerning those queries... is that valuable content or are those doorway pages?
If I give when someone few months of my service for free when they write a nice article with a link to my site... is that good customer service or a link scheme?
If I create a press release on PR Web and link to my service with an appropriate keyword ... is that participating in a link scheme?
None of these cases are all that clear cut and I for one am not comfortable with Google being the judge, jury and executioner.
> It approaches conspiracy thinking.
That's not the point. The point is that Google is a massive company wielding massive power.
When it comes to fighting a penalization their position is that they don't need to defend their decisions, implying that they can be trusted not to abuse their position and hence deserve to have the final say, legally, over these decisions.
I don't buy that and think it's about time that contesting Google penalties in court becomes a regular thing.
> If I think of a a lot of things my users would search for and try to create valuable information for them concerning those queries... is that great content or are those doorway pages?
Create a useful, information-rich site, and write pages
that clearly and accurately describe your content.
Think about the words users would type to find your pages,
and make sure that your site actually includes those words
within it.
Doorway pages are typically large sets of poor-quality
pages where each page is optimized for a specific keyword
or phrase.
Some examples of doorways include:
Having multiple domain names targeted at specific regions
or cities that funnel users to one page
Templated pages made solely for affiliate linking
Multiple pages on your site with similar content designed
to rank for specific queries like city or state names
>If when someone writes a nice article with a link to my site I give them a a few months of my service for free... is that good customer service or a link scheme?
Avoid tricks intended to improve search engine rankings.
A good rule of thumb is whether you'd feel comfortable
explaining what you've done to a website that competes
with you, or to a Google employee. Another useful test is
to ask, "Does this help my users? Would I do this if
search engines didn't exist?"
Any links intended to manipulate PageRank or a site's
ranking in Google search results may be considered part of
a link scheme and a violation of Google’s Webmaster
Guidelines. This includes any behavior that manipulates
links to your site or outgoing links from your site.
Large-scale article marketing or guest posting campaigns
with keyword-rich anchor text links can negatively impact
a site's ranking in search results.
The best way to get other sites to create high-quality,
relevant links to yours is to create unique, relevant
content that can naturally gain popularity in the Internet
community.
Effectively promoting your new content will lead to
faster discovery by those who are interested in the same
subject. Avoid: attempting to promote each new, small
piece of content you create; go for big, interesting
items. Avoid: involving your site in schemes where your
content is artificially promoted to the top of these
services. Avoid: spamming link requests out to all sites
related to your topic area. Avoid: purchasing links from
another site with the aim of getting PageRank instead of
traffic.
> If I create a press release on PR Web and link to my service with an appropriate keyword ... is that participating in a link scheme?
Links with optimized anchor text in articles or press
releases distributed on other sites can negatively impact
a site's ranking in search results.
Note: I wouldn't expect links from press release web
sites to benefit your rankings, however.
If there is evidence that Google gives special treatment to Adwords customers in organic search then this could be verified as a lie. As long as this evidence remains nothing more than accusations and suspicions, then I choose to believe Google that there is no teacup orbiting Venus. It seems unlikely, but not impossible. I am willing to sway my view when I get evidence. Till then, I will treat it as a common newbie myth.
Every single startup I've worked for, one of the objectives was to befriend Matt Cutts so you would have a quick line to getting stuff like this fixed. Seems like a reasonable amount of power to give one person/company...
You will be surprised by the amount of time people take to work seriously on fixing their penalty.
I have helped over 40 websites recover from Google penalty in the last 7-8 months. 80% of the websites seek help only after they have screwed up atleast one reconsideration request without taking the necessary steps.
Google typically responds to penalty reconsideration requests within a week, often less than a week. So if you have taken the right action and have submitted a really solid reconsideration request, its totally possible to get out of a penalty in under 2 weeks. Also, the backlink count was under 200K which is a reasonably manageable.
Rap Genius could have saved some time using tools like scrapebox to perform some of the scraping activities. But thats another story.
I want to give both Rap Genius and Google the benefit of the doubt.
I like to believe that Google checked the link profile again and found a large amount of low-quality links removed. Google would not remove a penalty if the spammy links were still abundant. So that they are out of a manual penalty is justified.
Rap Genius showed good intentions to comply with the guidelines. They went out of their way to clean up their act. These were no doorway pages or elaborate linking schemes with their own servers. It was a Linking Bieber Scheme that is considered greyhat in the industry, and after this, will probably be less popular.
This was a large and visible PR drama. Of course both Rap Genius and Google were on top of things. Google never stated that this was to be a month-long ban. We don´t know how those thousands of other sites were penalized. Maybe the large majority is so spammy/crappy that they simply won´t rank near the top with their spammy links removed, on the merits of their content and audience alone. For sites that are entirely build on manipulating SERPs, then yeah, a Google penalty can seem permanent. To me, that is an ok thing.
About the connections. Sure they help. I don´t see what is wrong though with leveraging your connections when there are millions on the line. Maybe a connection humbly asked: Hey, we effed up, how can we restore trust? This thread makes it sound like the connections pressured Google into removing the penalty. That seems unlikely if we all give the benefit of the doubt.
This was not greyhat. This was blackhat. Paying money for inbound links on keywords that trasnfer page rank (don't include NOFOLLOW) has been blackhat for a LONG time.
It was Panama hat. Rap Genius did not pay money, they offered a tweet. In a crude and obvious enough way to be a little scheme-y. Which unfortunately for RG was picked up in the media.
"even I was scratching my head to figure out if this was actually a Google violation or not. Rap Genius’s apology post had the company deciding itself that maybe it violated guidelines that links should be “editorially placed." - Danny Sullivan.
Paying with a Tweet from an account with a large number of followers (which definitely has value) instead of money is still paying. At best, it's basically a link exchange, which is also not allowed.
Is it so inherently wrong to prioritize a high-profile case of a popular site?
Merit is not a synonym for equality. From a user perspective that decision might surely have a lot of merit.
BTW, it also helped that people really found Rap Genius useful.
Er, what? You suggesting that Google execs are benchmarking the productivity of google searches? There are a million other "substitutes"...hence the shady SEO...
Google is constantly improving the search results. If factors, such as a low bounce rate and high dwell time, show that a site is popular and useful, why not rank it higher?
People tend to think in binary: A site is either spam or no spam. But spam is much more complex. A site can be 90% good and 10% bad. A site can turn spammy without their intention: comment spam, hacked sites. A site can pull out all the stops to hide their link buying. A site can be useful to have in index, regardless of some spammy behavior.
In order to judge if something is better, it presumes absolute reference points. In otherwords, google ranks the "productivity", generally. And then only later, refines the relative merits within a sub-class. (And if this was true, it would have nothing to do with Rap Genius.)
Avoiding the (questionable) debate on the productive capacity of planet-earth...the "model" of what customers' "really need" is questionable. Do they need (1) information that is factually correct?; and/or (2) do they need the actually meaning or interpretation of (1). And if its true a subset need (2) what is the ratio of (2/1)?
I have mentioned this before in this thread, I'll do it again. I AM really happy for rap genius.
I also view this post from RapGenius as a kind of a PSA.
They used the reasonable doubt argument, and won, they could have kept quiet about it. But, they let people know that others could appeal on reasonable doubt too.
Which is commendable!
Unfortunately, the underlying message some people miss when they feel this is about bashing RapGenius is that it is not.
The outrage is against Google. More specifically about 3-4 things:
1. Google has penalized sites for way lesser, and refused to reconsider, even when reasonable doubt was jarringly obvious (ex. some sites affected by panda/penguin algo updates targeted towards content/blog farms were penalized for as less as having similar posts type of intra-site linking as all content farms were using it)
2. Google will never apply this reasonable doubt argument for anyone in the future, unless they have friends like the investors at Andreessen Horowitz.
3. Google has given no indication on how they modified the algorithm for this special circumstance, the question that people are thinking is more in regards with if this was a team decision, or an individual's special favor.
4. It's a slap on the face for all white hat SEO professionals, webmasters and founders who have been burned by Google's no bullshit policy. Again its not about the policy existing, that's just a fact everyone has to work with. It's because now enforcement is clearly optional dependent on connections.
2] How do you know this is the case? Matt Cutts said they've relisted them quickly because AH slipped Google some cash? Or maybe RapGenius followed their cleanup policy correctly and were relisted naturally after submitting like many other companies a reconsideration request.
3] How do you know this was a special case? Thousands of sites I'm sure re de/relisted every day that you don't hear about (because they aren't being talked about on HN and no one outside here cares), are you sure none of those get relisted as quickly as RapGenius?
I don't. I also don't know if this is not the case. Do you? This uncertainty IS the reason for the current loss of faith.
> Matt Cutts said they've relisted them quickly because AH slipped Google some cash?
I never said cash exchanged hands. I doubt it did. Can you with a 100% surety say that a special favor of any kind did not occur?
> Maybe RapGenius followed their cleanup policy correctly
What clean up policy. If there is such a policy why isn't it public, why doesn't anyone know of it? [0]
> (RapGenius was) relisted naturally
A manual modification of the algorithm is not what has been established as natural over the last decade+.
> Thousands of sites I'm sure re de/relisted every day that you don't hear about (because they aren't being talked about on HN and no one outside here cares)
Ah, but know tens of these sites that were penalized, as in know 'em in and out, know the whole team, know of every SEO practice they implemented, these only include those that were penalized for having site structures similar to content farms (themes, permalinks and similar post type cross linking between posts) that were not even replied too let alone getting a chance. I also know a lot of SEO professionals who don't know any site that was afforded this (I emailed 'em, if I hear of one I'll update the post).
These sites were all with 300K+ uniques a month && 1MM+ pageviews/month.
HN is not my only source of information and news.
> are you sure none of those get relisted as quickly as RapGenius
Yes.
edit: [0] the only tool that I know Google allows webmasters to use to reduce potential penalty is the disavow links tool[1]. In the past I have seen this only useful for people who were targeted by link bombing etc. at times it was because a SEO practioner did in fact use some shady link sources.
Kudos to RapGenius for talking about it in their blog post, nonetheless, the results were still too quick (not judging rapgenius or their connections... only targeting Google).
At the point where you start suggesting that Matt Cutts was bribed [or accepted a favor], you know you've left the land of rationality and entered into wholesale speculation. Your argument is on the face ridiculous, but just to be clear: 1) Matt Cutts is very wealthy and doesn't need any favors. 2) Matt Cutts once penalized Google's own Chome site for improper links. If there were ever any conflict of interest that ever existed, that would be it, yet Matt went ahead with the penalty.
> At the point where you start suggesting that Matt Cutts was bribed
When did I do that?
I have not and am not suggesting that Matt Cutts or anyone at Google took a bribe of any kind whatsoever.
I have on the other hand said that there is some uncertainty if favors may have been given/used to expedite the process of re-listing.
Which if true, in itself looks really bad, simply because of the position of authority Google sits at.
Pray tell, when did a favor for a friend, acquaintance or anyone for that matter, regardless of their alleged wealth or social standing come to be defined as bribery/monetary value exchanging hands?
Or quite literally if I go by a dictionary: an act of kindness beyond what is due or usual. Example: "I've come to ask you a favor". Synonyms: service, good turn, good deed, kindness, act of kindness, courtesy.
To further elaborate to address your failure to understand a semantic line of argument:
1) The terms Favor and Bribery are not interchangeable.
2) I have only used the word favor.
3) I have insinuated[1] that there is possibility[2] of an alleged[3] favor having occurred in expediting[4] the re-listing of RapGenius, and that this may have happened due to good connections from the VCs.
4) If true, this is not in any way illegal, by the law of the land (as opposed to bribery[6], which is illegal). And this would simply be a moral wrong doing to others who cannot use this speedy service based. This is due to the position of power Google has on web start-ups, websites and businesses.
5) In addressing your previous comment I stated that none of my statements can or should be construed[5] as referring to bribes or any thing of monetary value, this includes gifts.
Meanings and Definitions:
[0] Favor - an act of kindness beyond what is due or usual
[1] Insinuate - suggest or hint (something bad or reprehensible) in an indirect and unpleasant way
[2] Possibility - a thing that may happen or be the case
[3] Allege - claim or assert that someone has done something illegal or wrong, typically without proof that this is the case
[4] Expedite - make (an action or process) happen sooner or be accomplished more quickly
[5] Construe - interpret (a word or action) in a particular way
[6] Bribe - a sum of money or other inducement offered or given in this way
You missed my point again. The semantic line of argument is pointless. Favor and bribery are interchangeable because my argument holds for either case.
So you're saying they were given a Blueprint to fix their mishaps but got off based on Reasonable Doubt before they went thru with the recommended plan of action?
I suppose it's just a Hard Knock Life for companies that don't have prominent investors backing them. It's almost as if the investors pulled an American Gangster type situation on Google.
Well I suppose at the end of the day this is a reminder to Blackhat SEO users to Watch The Throne.
> "To save a drowning man, he must first give you his hand"
is very wrong from anything I ever read about the topic. Drowning leads to involuntary reactions that cannot be controlled, like trying to climb on the person trying to save you, endangering them as well. That's why approaching and grabbing a drowning person from behind is suggested, and maybe even knocking them out if need be. (unless you can throw them a lifesaver or something else they can grab safely, of course)
This really shows that even in Silicon Valley, your success is driven as much by the connections you have and the people you know, rather than pure talent and merit.
Paul Graham sold out and is now just like every other capitalist. Exploiting privilege to get richer. ycombinator's biggest successes are marketing successes--their success is defined by PR and Cronyism.
This isn't Steve Wozniak--the tech genius--creating the Apple II. This is fat headed fratboys using PR stunts and cronies to rake in money by squeezing out the competition.
The rich will get richer. The children of the rich will get richer. PR darlings and frat douchebags will get richer. Hackers will be ignored by YCombinator--Hackers aren't as profitable.
This really is the thing that struck me. All this talk about companies being destroyed and jobs lost and hardly a mention of worry about the power Google yields.
We need more competition in this space. Sites need a search engine with real market share to compete against Google.
I have to note, you have to be big to make comeback like this. Google provided disavow tool, but from my experience it simply not working. So it is manual action for sure.
That they blatantly employed blackhat SEO techniques and are back on Google within a matter of days is sickening.
Think of all the sites that are penalized in various ways for similarly shady SEO practices and have to spend _months_ to regain their previous position.
Hopefully Bing et al can capture some marketshare and bring some degree of neutrality to the search engine landscape (i.e. not have every website owner on the planet beholden to the beast).
The "technical digression" with Github code on how to do a mass scrape is cool, though not surprising (er...how else would you accomplish such a big data task...though I'm glad I'm not the only one who uses Typhoeus, a great but underused gem).
But what's surprising to me is that a company that knows how to automate things and work with data...why the f--- were they manually soliciting bloggers to post links? That seems seriously inefficient and comically backwards...as if Sergey Brin spent his early Google days modifying SERPs by editing and uploading Excel spreadsheets to the server.
It seems seriously inefficient to me because at this point, Rap Genius has a lot of traffic and a lot of linkage already. So did the few backlinks, per day, that RapGenius got, through lame blogmasters who would participate in such a thing...really have an effect on their SEO rank, i.e. is it so easy to still game Google?
Or, was RG just doing it because they thought it would have a worthwhile effect (keep in mind that this is one of their co-founders who was individually emailing people), i.e. RG was being naively optimistic?
Actually, it's understandable that Google set this on High Priority...
#1: it's a high traffic website, that is a startup and the whole deal got a lot of attention of us (HN Readers) + Matt even saw this here. So it was good to respond fast (no choice)
#2: RapGenius is dependant on Google (although they try to let you see it as a social network, but it's not), they would have gone bankrupt ( i suppose) without Google.
#3: Without a quick response and/or measure to fix their shit on their end, investors wouldn't invest any more money in RapGenius and Google would probably hurt a lot of future investments (how is this startup dependent on Google, ...)
BMW once got a penalty from Google, they didn't got easily off. But their business wasn't dependent on Google.
Either way, there are both pro and cons to each action. But overall, i think they handled if fairly well.
Infinite scroll (or some type of weird JS) won't let me get to the bottom of the page... Ain't got ish to do with this but I just thought that I should mention.
One thing I know is that RapGenius is good about making a big stink, being in the news, and then having things work out in their favor, even if they were the ones doing a dumb thing in the first place.
They freaked out at Heroku with decent reason because it was costing them money, but if they ran their own infrastructure, that would have been a total non issue and at the scale they are running, why ARE they using Heroku?
They did some incredibly dumb SEO things because they are greedy and are willing to cheat to win. That is on RapGenius, but once they got banned it is somehow Google's fault and they should be reincluded? I don't get it.
RapGenius is great at one thing - creating controversy and getting press for it. I guess that works for them, but it seems like a very selfish way to grow a business. Instead of focusing on building value themselves, they are willing to tear down their business partners in public to get what they want.
I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate.
They freaked out at Heroku with decent reason because it was costing them money, but if they ran their own infrastructure, that would have been a total non issue and at the scale they are running, why ARE they using Heroku?
Are you arguing that what Heroku was doing wasn't wrong?
I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate.
Comments on every single RapGenius post on HN have been filled with vitriol toward the founders, their attitude, and the culture they foster.
202 comments
[ 2.8 ms ] story [ 301 ms ] threadYou're lucky if you get a decent reply from customer support to begin with..
Your best bet is to change domains, fill in a redirect and try to build up backlinks again. It'll take a year or two and hopefully no competitor pops up in your space.
Or, you can try and get Andreesen-Horowitz interested in your business.
It entirely depends upon the scope of penalty. Unless you are completely deindexed from Google search results, you can always recover from a penalty.
Feel free to email me if you have any specific need.
Moreover, this article feels pretty neutral/humble and techy, a far cry from the haughty attitude their founders usually display in interviews.
http://poetry.rapgenius.com/Franz-kafka-le-nouvel-avocat-lyr...
More shady SEO practices from Rap Genius, disappointed but not surprised.
And some of those are not even poems:
http://poetry.rapgenius.com/Herman-melville-bartleby-the-scr...
This one is particularly weird / egregious:
http://poetry.rapgenius.com/Engineering-genius-understanding...
The fact that all our text and song page URLs end in "-lyrics" is a relic of a time when we hosted almost exclusively lyrics.
Making the changes required to host all kinds of annotated texts has been a big project and making the URLs make sense is something we haven't gotten around to.
The reason it's not a totally trivial change is that both users and texts share the same top-level namespaces, and so right now we use the "-lyrics" suffix to differentiate song and text pages from user profile pages.
We want to change the suffix to "-annotated" for non-music pages and will hopefully get around to in the next few weeks. To our knowledge the existing situation doesn't help with SEO (no one is searching "bartleby the scrivener lyrics"), and it's confusing, so we 100% agree that the current situation is bad.
For more info on the "Worse is Better" philosophy that caused the current jank situation to persist for so long, see this talk: https://www.youtube.com/watch?v=X45YY97FmL4
<< Technical Digression: How to scrape 178k URLs in Ruby in less than 15 minutes - Ok, so you have 178k URLs in a postgres database table. You need to scrape and analyze all of them and write the analysis back to the database. Then, once everything’s done, generate a CSV from the scraped data. >>
- first, they use it with Google to show how sincere they were about "cleaning up"
- second, and perhaps more important, it gives Google a public excuse to justify why they restored RG so quickly in rankings. As in, "Oh they did a very thorough job of complying by our standards that we had no option but to restore their status".
Unfortunately in this case Google's response speed comes across as a bad thing. Because it applies to only a very select group of sites that are either very rich (aka large advertisers) or very well connected.
[Edit: clarified imagined comment]
>Though Google is an extremely important part of helping people discover and navigate Rap Genius, we hope that this ordeal will make fans see that Rap Genius is more than a Google-access-only website. The only way to fully appreciate and benefit from Rap Genius is to sign up for an account and use Rap Genius – not as a substitute for Wikipedia or lyrics sites, but as a social network where curious and intelligent people gather to socialize and engage in close reading of text.
ie, you should really just become a member rather than relying on google...
Also, where was bing in all of this?!? ;)
But this truly saddens me as well. I cannot for the life of me understand how they were able to fix ten-of-thousands of backlinks and other shady practices and get back in the good graces of Google so darn quickly.
Frankly I do not know a single other legitimate(+with good traffic) site who was able to get back in the good graces of Google after algorithm updates without spending many many months researching any possible reasons to be penalized and fixing it themselves after repeatedly asking google for help to no avail.
Could it be that Rap Genius was special because it has Andreessen Horowitz?
Sure seems that way.
I am not going to stop using Google, but I am inclined to use them lesser now.
I would like to hear why, how rap genius was fixed this and what this means for other people who get penalized from Matt Cutts though!!
They did not quite finish their action plan. They got away on reasonable doubt.
This is the first time I am seeing this defense considered by Google.
I willing to bet any other non-vc funded bootstrapped start-up would never be afforded this consideration by Google.
This rant is not a judgement on Rap Genius, I am actually glad they maneuvered around this.
I am judging Google.
They have killed other start-ups for reasons such as other sites farming and duplication their content more vehemently. Reasonable doubt has never been considered for anybody else and most likely will never be hence forth.
What about that gmail fiasco a few months ago? Users got blocked out of G services for not providing their real names and had no recourse to get their data for several weeks/months.
Just goes to show that in several cases it's not what you know but who you know.
In this case, the user experience for a Google user is worse when Rap Genius is removed. Google has some incentive to be more active in fixing the problem there, especially since Rap Genius was cooperative, technically competent, and aside from their "rapper swag persona", genuinely nice guys.
I run across Rap Genius now and then and have always considered it to be spam, just like yahoo answers, ehow, and mahalo pages.
Why? You've no doubt read 10x reasons people here have said so (in the other threads). Do you just want to argue? If you don't agree, fine, but clearly a large and vocal group does. So what? Move along - find other things to care about.
They have had some legal troubles, but are trying to sign licensing deals http://www.nytimes.com/2013/11/15/business/media/rap-genius-...
However, if their licensing deals are anything like what other sites have to pay for lyrics, it's going to significantly (entirely?) cut into their profit margin.
The main driving force behind it is that anyone can edit the lyrics, and even better, anyone can ANNOTATE the lyrics to explain what certain sections mean.
Let's take for example the lyrics for Mumford & Sons' "The Cave": http://rock.rapgenius.com/Mumford-and-sons-the-cave-lyrics
I looked up this song one day trying to understand more of the allusions made in the song; I know the title itself alludes to Plato's 'The Cave' and wanted to delve deeper. Because of this site, I can see that Verse 3:
> So come out of your cave walking on your hands
> And see the world hanging upside down
> You can understand dependence
> When you know the maker's hand
is an allusion to St. Francis.
Most of the time when Rap Genius ISN'T the top result for lyrics when I google, I facepalm and go directly to their site.
I should emphasize here that this overly positive post isn't intended to condone their shady, spammy blackhat SEO tactics, but damnit if they aren't the BEST lyrics site by a fair margin.
Maybe I looked at the wrong lyrics, but I saw precious few explanations, instead most lines had jokes and/or totally unrelated pictures attached to them. Often enough the jokes is to simply restate what was said in rapper slang in plain english, which is funny for exactly the first time you read it, then it's just spam.
Is anyone else getting more and more suspicious of the scorn that people have been gratuitously heaping on RG? Like, did they run over everyone's dog?
That's exactly how I feel too. Though the incentive for google is very unclear, the lack of clarity is explored further below.
> You have the causality wrong. Rap Genius has Andreessen Horowitz because Rap Genius is special.
Not saying Rap Genius is not a special lyrics start-up. Not even saying that it would not have been a good candidate for any VC financing them.
Though, Definitely saying that the special case leniency shown by Google sure make it seem like having Andreessen Horowitz is the only thing different between Rap Genius and scores of sites before with this and similar situations.
> In this case, the user experience for a Google user is worse when Rap Genius is removed.
And its not when other good sources are removed because they were link bombed or their content was scraped more than their competitors?
> Rap Genius was cooperative
Find me one person in their situation who would not be. It was to either co-operate or die.
> Rap Genius was technically competent
How?
> Aside from their "rapper swag persona", genuinely nice guys.
I am judging Google.
I have nothing against Rap Genius founders, nor their "rapper swag persona" and I'll believe you when you say the are nice guys.
Aren't all affiliate programmes technically bad for the publisher after Penguin?
If the folks at Google can figure out how to detect this sort of gaming more reliably, that may end up in a future algorithm update, and no amount of pleading will matter for anyone affected by it - they'll have to fix the problem or continue to suffer the consequences.
Well there wasn't an algorithm update, there was a manual action. Google reversed the manual action after RapGenius stopped breaking the rules and cleaned up its mess. If this happened to you there would be a note about it in Webmaster Tools and you would be able to ask Google to take another look. I'm sure RapGenius had all hands on deck making sure they were up to snuff.
Anyway, Franz Kafka would die again if he saw his text on this site (http://poetry.rapgenius.com/Franz-kafka-a-dream-lyrics), so let's just forget about this site and don't use it.
And about Franz Kafka, I think he'd turn in his grave if he heard about how petty people like yourself criticize a website that tries to make literary criticism of his work digestible and enjoyable for the masses.
It is a bad thing here: it shows that Google is apparently willing to act preferentially to favoured businesses, undermining the idea that search results are provided with neutrality.
Imagine a business in Germany, Vietnam, or even.. Texas having to do the same? It's pretty much impossible. This is what happens to everyone else out there.
Which is what the parent is referring to when he says that google results aren't as neutral as they claim to be. They clearly favored someone with leverage in this instance.
But this is the only instance that we know of. So far.
It's not a question of having to do the same, it just means that knowing a Google employee with some magic power is a valuable asset.
But yes, if you're a company that doesn't have this asset and you make a stupid mistake like Rap Genius did your site might be off Google for a month or two, possibly destroying your business.
I still don't follow how that makes Google not neutral in any significant way.
>> stupid mistake like Rap Genius did your site might be off Google for a month or two, possibly destroying your business.
Sometimes businesses get wrongly blacklisted. Yes, it happens. They do get destroyed because of this, people do lose homes and jobs over this.
But this isn't because Google takes a month or two to sort this kind of thing out (a month or two is also not really acceptable - too long - given the impact Google has on a business). It's because experience shows that once you are on the wrong side of the big G, there's pretty much nothing you can do to get on the right side. It's no longer a question of a month or two, it's about registering a new domain, and starting from scratch. Sometimes you have to do this even when you didn't do anything wrong. This applies to almost every Google service whether it's Search, Adsense, Gmail, or anything else.
But Rap Genius definitely did something wrong. It was obvious for everyone to see. Yet, they managed to follow the 'recommendations' and get back in the game pretty quick. We know that this is probably because of connections because every business out there that gets blacklisted by Google will make sure that they are 'regulation-compliant' ASAP. Yet, many of them never manage to make it back and those that do take years at times.
Yes, knowing the US President helps, but that's provided you can get your work done without knowing him as well, in a reasonable time frame. Especially when it may make the difference between running a thriving business and being homeless.
QED: Monopolies are bad.
They're useful, relevant, and it just seems like good marketing. These things will spread organically and drive traffic even if you remove the SEO benefits.
Could you say that Scribd or Slideshare are doing something similar by allowing people to embed documents in their websites along with a no follow link back to their site?
Paying people to link back to your website on high-value keywords like 'justin beiber lyrics' from blog posts on other sites to artificially inflate your pagerank is 100% blackhat SEO. The payment in this case was a Twitter post about the link provider's website.
There is no way they could get back so fast and so easily on google.
Should their investors not have made calls? However, it should be noted that they did their part.
One of my favorite quotes is this: "To save a drowning man, he must first give you his hand"
If the Rap Genius guys were incompetent and did not do their part in removing the links so quickly (see main story for technical details) there would be little or nothing their investors would have been able to do about that.
BTW, it also helped that people really found Rap Genius useful. I'm sure many searched had rapgenius appended to it. So the lesson is, make your app so good that when Google delists you, it will make them look bad.
Thankfully?, there is also a precedence for other offenders to use. if you can detail you have atoned for your sins and Google insists your you must do a certain time, you know where to turn to.
BTW, this is Hackernews, and I look forward to the discussion of the (de)merits of how they scrapped and analyzed 177,000 links and not espousing of anger that they survived the punishment.
I'm sure Mahbod (my favorite) and co have learned that you do not go 'daaawging' when you are in a hole. I wish them success.
This is painfully true, and I know it has been discussed in prior discussions. Rap Genius is without a doubt THE BEST lyrics site available, hands down. Even if the other sites weren't ad-ridden scammy buggy shitty websites, if they got their acts together and actually TRIED, Rap Genius would still be top dog.
I don't care for the attitudes or personalities of the people in charge, and I am absolutely disgusted that they felt the need to scam their Google rank, but I am willing to see that some infractions can be rectified through penance, and am happy to know that they will not be forced to shut down because of this fiasco.
If Google applied these rules consistently with all websites, I would have no problem with it, but they don't. This breaks the myth that Silicon Valley is a meritocracy - as it shows that the connections you have are as important as raw merit, if not more
This is the PERFECT time for anyone who has done penance and is still being blacklisted by Google to document it all and go to the press.
There are A LOT of people/journalists waiting to jump on a story that shows Google has double standards.
But first they must be willing to do their part by documenting these facts and reaching out the journos. It will not come to them.
Like I quoted "to save a drowning man.....
Fairness does not exist in real life. Screaming "unfair" will not help you achieve anything. It's such a ridiculous situation for me. All these guys are sitting on the line, raging at real life because popular culture brain washed them.
I can't believe I'm saying this, but
What a bunch of stupid nerds!
RapGenious had no excuse.
Still, I'm glad they're out of penalty now.
Just as a thought experiment ... how tempting do you think it would be for Google to do the following:
1) Run a query to identify all companies that cut their AdWords spending over 50% over the last year.
2) Look at which of these companies are now getting significant numbers of click from organic results.
3) Find out what is causing them to rank high organically.
4) Penalize them using a generic message and refuse on principle to answer any inquiries as to why.
Apart from (anec-)data that seems to be pointing towards this already happening; should we be OK as a market with this approach? Especially when considering that the rules are so vague that almost every site is guaranteed to break some of Google's guidelines?
Please provide this evidence. I am getting a bit tired of this misinformation. Organic search and paid search are silo'd. To say that Google favors advertisers in the organic results or the other way round, is simply not true. It would destroy Google's credibility.
>how tempting do you think it would be for Google
It approaches conspiracy thinking. I love that, but I think HN is not the place. You are basically accusing a company of a very evil act, without proper evidence. Do you realize that many Googlers frequent this site? It would be a shame if all they get to read are conspiracy theories and baseless accusations.
>Especially when considering that the rules are so vague that almost every site is guaranteed to break some
Hogwash. Their rules are very clear and succinct.
https://support.google.com/webmasters/answer/35769?hl=en
Maybe if you want to spam or manipulate, then these rules are vague. Maybe when you don't read the guidelines then they are vague. I bet that if you can point to a rule that almost every site breaks, without them knowing, then Google will adjust that rule to be more clear.
You can not please them all.
>Penalize them using a generic message and refuse on principle to answer any inquiries as to why.
You can certainly not please the spammers that were caught with their hands in the cookie jar and then take to the internet to say that Google is unfair, and that the big brands get away with anything.
If I think of a lot of things my users would search for and try to create valuable information for them concerning those queries... is that valuable content or are those doorway pages?
If I give when someone few months of my service for free when they write a nice article with a link to my site... is that good customer service or a link scheme?
If I create a press release on PR Web and link to my service with an appropriate keyword ... is that participating in a link scheme?
None of these cases are all that clear cut and I for one am not comfortable with Google being the judge, jury and executioner.
> It approaches conspiracy thinking.
That's not the point. The point is that Google is a massive company wielding massive power.
When it comes to fighting a penalization their position is that they don't need to defend their decisions, implying that they can be trusted not to abuse their position and hence deserve to have the final say, legally, over these decisions.
I don't buy that and think it's about time that contesting Google penalties in court becomes a regular thing.
So you can't figure out exactly how you can game the system... boo hoo.
According to who? Google? Is there any transparency or any way for an outsider to verify that?
Matt Cutts answers: Do AdWords customers get special treatment in organic search results?
http://www.youtube.com/watch?v=7aV5DmL_eog
If there is evidence that Google gives special treatment to Adwords customers in organic search then this could be verified as a lie. As long as this evidence remains nothing more than accusations and suspicions, then I choose to believe Google that there is no teacup orbiting Venus. It seems unlikely, but not impossible. I am willing to sway my view when I get evidence. Till then, I will treat it as a common newbie myth.
I have helped over 40 websites recover from Google penalty in the last 7-8 months. 80% of the websites seek help only after they have screwed up atleast one reconsideration request without taking the necessary steps.
Google typically responds to penalty reconsideration requests within a week, often less than a week. So if you have taken the right action and have submitted a really solid reconsideration request, its totally possible to get out of a penalty in under 2 weeks. Also, the backlink count was under 200K which is a reasonably manageable.
Rap Genius could have saved some time using tools like scrapebox to perform some of the scraping activities. But thats another story.
I like to believe that Google checked the link profile again and found a large amount of low-quality links removed. Google would not remove a penalty if the spammy links were still abundant. So that they are out of a manual penalty is justified.
Rap Genius showed good intentions to comply with the guidelines. They went out of their way to clean up their act. These were no doorway pages or elaborate linking schemes with their own servers. It was a Linking Bieber Scheme that is considered greyhat in the industry, and after this, will probably be less popular.
This was a large and visible PR drama. Of course both Rap Genius and Google were on top of things. Google never stated that this was to be a month-long ban. We don´t know how those thousands of other sites were penalized. Maybe the large majority is so spammy/crappy that they simply won´t rank near the top with their spammy links removed, on the merits of their content and audience alone. For sites that are entirely build on manipulating SERPs, then yeah, a Google penalty can seem permanent. To me, that is an ok thing.
About the connections. Sure they help. I don´t see what is wrong though with leveraging your connections when there are millions on the line. Maybe a connection humbly asked: Hey, we effed up, how can we restore trust? This thread makes it sound like the connections pressured Google into removing the penalty. That seems unlikely if we all give the benefit of the doubt.
"even I was scratching my head to figure out if this was actually a Google violation or not. Rap Genius’s apology post had the company deciding itself that maybe it violated guidelines that links should be “editorially placed." - Danny Sullivan.
At least now we know. This is blackhat.
Er, what? You suggesting that Google execs are benchmarking the productivity of google searches? There are a million other "substitutes"...hence the shady SEO...
People tend to think in binary: A site is either spam or no spam. But spam is much more complex. A site can be 90% good and 10% bad. A site can turn spammy without their intention: comment spam, hacked sites. A site can pull out all the stops to hide their link buying. A site can be useful to have in index, regardless of some spammy behavior.
Avoiding the (questionable) debate on the productive capacity of planet-earth...the "model" of what customers' "really need" is questionable. Do they need (1) information that is factually correct?; and/or (2) do they need the actually meaning or interpretation of (1). And if its true a subset need (2) what is the ratio of (2/1)?
And only then does this framework make sense.
I also view this post from RapGenius as a kind of a PSA.
They used the reasonable doubt argument, and won, they could have kept quiet about it. But, they let people know that others could appeal on reasonable doubt too.
Which is commendable!
Unfortunately, the underlying message some people miss when they feel this is about bashing RapGenius is that it is not.
The outrage is against Google. More specifically about 3-4 things:
1. Google has penalized sites for way lesser, and refused to reconsider, even when reasonable doubt was jarringly obvious (ex. some sites affected by panda/penguin algo updates targeted towards content/blog farms were penalized for as less as having similar posts type of intra-site linking as all content farms were using it)
2. Google will never apply this reasonable doubt argument for anyone in the future, unless they have friends like the investors at Andreessen Horowitz.
3. Google has given no indication on how they modified the algorithm for this special circumstance, the question that people are thinking is more in regards with if this was a team decision, or an individual's special favor.
4. It's a slap on the face for all white hat SEO professionals, webmasters and founders who have been burned by Google's no bullshit policy. Again its not about the policy existing, that's just a fact everyone has to work with. It's because now enforcement is clearly optional dependent on connections.
3] How do you know this was a special case? Thousands of sites I'm sure re de/relisted every day that you don't hear about (because they aren't being talked about on HN and no one outside here cares), are you sure none of those get relisted as quickly as RapGenius?
I don't. I also don't know if this is not the case. Do you? This uncertainty IS the reason for the current loss of faith.
> Matt Cutts said they've relisted them quickly because AH slipped Google some cash?
I never said cash exchanged hands. I doubt it did. Can you with a 100% surety say that a special favor of any kind did not occur?
> Maybe RapGenius followed their cleanup policy correctly
What clean up policy. If there is such a policy why isn't it public, why doesn't anyone know of it? [0]
> (RapGenius was) relisted naturally
A manual modification of the algorithm is not what has been established as natural over the last decade+.
> Thousands of sites I'm sure re de/relisted every day that you don't hear about (because they aren't being talked about on HN and no one outside here cares)
Ah, but know tens of these sites that were penalized, as in know 'em in and out, know the whole team, know of every SEO practice they implemented, these only include those that were penalized for having site structures similar to content farms (themes, permalinks and similar post type cross linking between posts) that were not even replied too let alone getting a chance. I also know a lot of SEO professionals who don't know any site that was afforded this (I emailed 'em, if I hear of one I'll update the post).
These sites were all with 300K+ uniques a month && 1MM+ pageviews/month.
HN is not my only source of information and news.
> are you sure none of those get relisted as quickly as RapGenius
Yes.
edit: [0] the only tool that I know Google allows webmasters to use to reduce potential penalty is the disavow links tool[1]. In the past I have seen this only useful for people who were targeted by link bombing etc. at times it was because a SEO practioner did in fact use some shady link sources.
Kudos to RapGenius for talking about it in their blog post, nonetheless, the results were still too quick (not judging rapgenius or their connections... only targeting Google).
[1] https://support.google.com/webmasters/answer/2648487?hl=en
EDIT: added []
When did I do that?
I have not and am not suggesting that Matt Cutts or anyone at Google took a bribe of any kind whatsoever.
I have on the other hand said that there is some uncertainty if favors may have been given/used to expedite the process of re-listing.
Which if true, in itself looks really bad, simply because of the position of authority Google sits at.
Pray tell, when did a favor for a friend, acquaintance or anyone for that matter, regardless of their alleged wealth or social standing come to be defined as bribery/monetary value exchanging hands?
I used both interchangeably (see point #2), and I've edited for clarification.
If you want to keep going on the overly semantic line of argument, bribery = gifts, not just money.
Or quite literally if I go by a dictionary: an act of kindness beyond what is due or usual. Example: "I've come to ask you a favor". Synonyms: service, good turn, good deed, kindness, act of kindness, courtesy.
To further elaborate to address your failure to understand a semantic line of argument:
1) The terms Favor and Bribery are not interchangeable.
2) I have only used the word favor.
3) I have insinuated[1] that there is possibility[2] of an alleged[3] favor having occurred in expediting[4] the re-listing of RapGenius, and that this may have happened due to good connections from the VCs.
4) If true, this is not in any way illegal, by the law of the land (as opposed to bribery[6], which is illegal). And this would simply be a moral wrong doing to others who cannot use this speedy service based. This is due to the position of power Google has on web start-ups, websites and businesses.
5) In addressing your previous comment I stated that none of my statements can or should be construed[5] as referring to bribes or any thing of monetary value, this includes gifts.
Meanings and Definitions:
[0] Favor - an act of kindness beyond what is due or usual
[1] Insinuate - suggest or hint (something bad or reprehensible) in an indirect and unpleasant way
[2] Possibility - a thing that may happen or be the case
[3] Allege - claim or assert that someone has done something illegal or wrong, typically without proof that this is the case
[4] Expedite - make (an action or process) happen sooner or be accomplished more quickly
[5] Construe - interpret (a word or action) in a particular way
[6] Bribe - a sum of money or other inducement offered or given in this way
Your insinuation is still baseless.
I suppose it's just a Hard Knock Life for companies that don't have prominent investors backing them. It's almost as if the investors pulled an American Gangster type situation on Google.
Well I suppose at the end of the day this is a reminder to Blackhat SEO users to Watch The Throne.
> "To save a drowning man, he must first give you his hand"
is very wrong from anything I ever read about the topic. Drowning leads to involuntary reactions that cannot be controlled, like trying to climb on the person trying to save you, endangering them as well. That's why approaching and grabbing a drowning person from behind is suggested, and maybe even knocking them out if need be. (unless you can throw them a lifesaver or something else they can grab safely, of course)
This is rapgenius. If it had to cross the line why on something so materially insignificant.
Successful startups, like mainstream corporations, are determined by
1. Public relations stunts, getting press, marketing bullshit
2. Connections, Privilege, Cronyism, Nepotism, Harvard, Yale, Stanford
Paul Graham sold out and is now just like every other capitalist. Exploiting privilege to get richer. ycombinator's biggest successes are marketing successes--their success is defined by PR and Cronyism.
This isn't Steve Wozniak--the tech genius--creating the Apple II. This is fat headed fratboys using PR stunts and cronies to rake in money by squeezing out the competition.
The rich will get richer. The children of the rich will get richer. PR darlings and frat douchebags will get richer. Hackers will be ignored by YCombinator--Hackers aren't as profitable.
We need more competition in this space. Sites need a search engine with real market share to compete against Google.
https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ss...
https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ss...
That they blatantly employed blackhat SEO techniques and are back on Google within a matter of days is sickening.
Think of all the sites that are penalized in various ways for similarly shady SEO practices and have to spend _months_ to regain their previous position.
Hopefully Bing et al can capture some marketshare and bring some degree of neutrality to the search engine landscape (i.e. not have every website owner on the planet beholden to the beast).
But what's surprising to me is that a company that knows how to automate things and work with data...why the f--- were they manually soliciting bloggers to post links? That seems seriously inefficient and comically backwards...as if Sergey Brin spent his early Google days modifying SERPs by editing and uploading Excel spreadsheets to the server.
It seems seriously inefficient to me because at this point, Rap Genius has a lot of traffic and a lot of linkage already. So did the few backlinks, per day, that RapGenius got, through lame blogmasters who would participate in such a thing...really have an effect on their SEO rank, i.e. is it so easy to still game Google?
Or, was RG just doing it because they thought it would have a worthwhile effect (keep in mind that this is one of their co-founders who was individually emailing people), i.e. RG was being naively optimistic?
#1: it's a high traffic website, that is a startup and the whole deal got a lot of attention of us (HN Readers) + Matt even saw this here. So it was good to respond fast (no choice)
#2: RapGenius is dependant on Google (although they try to let you see it as a social network, but it's not), they would have gone bankrupt ( i suppose) without Google.
#3: Without a quick response and/or measure to fix their shit on their end, investors wouldn't invest any more money in RapGenius and Google would probably hurt a lot of future investments (how is this startup dependent on Google, ...)
BMW once got a penalty from Google, they didn't got easily off. But their business wasn't dependent on Google.
Either way, there are both pro and cons to each action. But overall, i think they handled if fairly well.
They freaked out at Heroku with decent reason because it was costing them money, but if they ran their own infrastructure, that would have been a total non issue and at the scale they are running, why ARE they using Heroku?
They did some incredibly dumb SEO things because they are greedy and are willing to cheat to win. That is on RapGenius, but once they got banned it is somehow Google's fault and they should be reincluded? I don't get it.
RapGenius is great at one thing - creating controversy and getting press for it. I guess that works for them, but it seems like a very selfish way to grow a business. Instead of focusing on building value themselves, they are willing to tear down their business partners in public to get what they want.
I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate.
Are you arguing that what Heroku was doing wasn't wrong?
I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate.
Comments on every single RapGenius post on HN have been filled with vitriol toward the founders, their attitude, and the culture they foster.