Depends on how you look at it, The wage fits in around the rough ave pay for a Wendys "crew memeber", however you are responsable for your own costs while working (computer, internet bill, electricity bill, heating) but on the flip side, its part-time, no comute, no long hours standing, no dress code.
Acually ... no, but pinterest provides results that are relevant and satisfying to the general search (and the SQR - the Search Quality Rating, see link in discussion). Hence Pinterest stays in good standing even if most people like me (you, too?) hate it for its login/content walling tactics.
> but pinterest provides results that are relevant and satisfying to the general search
More like it matches some keywords like run-of-the-mill WordPress malware generates. I've seen plenty of both. Not to mention the similarity to the masses of biz/buzz/xyz spam websites.
It's not like you need a PhD to confirm if a search result sucks. Results are supposed to be good enough for the average user, so you need common sense and some on-job training to do a good job.
And having 10 common-sense guys churning through user feedback @$11/h is better than 2 geniuses @$55/h.
As one comment says, it probably beats fliping burgers.
Where is the limit? Why not 20 guys in a low-cost country @$5.50/h?
They cannot rate all results anyway, so why not get the best possible evaluations?
Shouldn't quality data beat quantity if you extrapolate from the evaluated samples to the entire index?
Spammers target the lower part of the bell curve. If you let that part evaluate the quality of the results, do you think that they will recognize spam?
It can be suprisingly difficult. A lot of the difficulty is in understanding the user intent behind a search query (since you're usually rating other people's queries, not your own).
For example, here are some queries in my history - would the average person understand what I'm looking for and what a good search result would be?
Good examples. Raters have - afaik- to assume multiple possible intentions on two word searches (and general searches on one). This results in better ratings for results (top 10 generally) that partially match different search intentions. so for "pandas databases" it would match the panda python libraries, but also databases literally of pandas ... and maybe a company that is called pandas and has databases. And this is the reason why we can never have "pure" results (in most cases) for a specific search, because the algorithm and the search rating is aiming for result-topic-diversity most of the time (if not all of the time).
I think everyone in SQR is based on the national average/context. In germany - even 15 years ago - SQRs were paid quite okay (around 16€/hour back then) including health insurance and social security pay.
Yes, Google has tricked a massive army of people into doing free work for one of the most profitable companies in the world. Things like "Google Local Guides" give people a gold star (and maybe a pair of branded socks) for doing unpaid labor. It's kinda hilarious, but also sad.
I imagine there are Google executives in a room somewhere laughing their full heads off every time they think about how there are people who do free work for Google who think they are making the world a better place or something.
When they tried to lure me into a "Local Guide" I was offered a free flight, hotel and food at Menlo Park. Once, but still good enough for no work at all.
>laughing their full heads off every time they think about how there are people who do free work for Google who think they are making the world a better place or something.
So when you go to a store that Google Maps says is open, and find it's closed, you refuse to correct the hours and go "MUHAHAHAHA...I refuse to help Google, take that, corporate scum!"
Why would I bother checking Google Maps? But basically, yeah. If Google wants my expertise and data collection ability, they can pay me for it. As it stands now they just cold call businesses and ask them with an AI thing.
Okay! I just opened OsmAnd and looked up my local supermarket. Sadly, no work hours. And this is a very popular supermarket right next to a train station in central Brooklyn, NY.
The thing is that business owners do update the info on Google Maps, and have an incentive to do so, but only volunteers update OSM.
OSM shines where businesses don't care but volunteers do: trails in a park, non-commercial POIs, etc.
At a company my wife worked for, the social media department spent half its time fixing store hours, addresses, and map locations in Google Maps and Apple Maps.
I guess it's nice that ordinary people try to do these updates for Big G and Apple. But they're often wrong, or they presume a one-day change is a permanent change, or that it applies to every location on the planet.
I'm sure you can figure it out. I believe in you. (And that response is because I'm 100% treating your question in good faith and not "I know why I would, but also I know why I wouldn't and prefer that, so I'll pretend I don't know why I would".)
I can see how the gamification aspect of points and levels might be more motivating for some people, but mostly I see Local Guides as not too different from people posting pictures and reviews to various social networks, except that it is more discoverable by people using Maps. I submit contents to Maps for mostly the same reasons why I use social networks and expect nothing in return.
This is probably one of the more embarrassing and shameful parts of Google's labor ecosystem.
These "Google $title Experts" work as uncompensated customer support (a legal requirement that they treat as such, if the quality is any indicator) on various social media outlets like Reddit and Twitter. They don't just provide free labor, they also go to bat for Google's reputation, acting as marketing and PR liaisons to prevent backlash from users who've been failed by their official support processes.
One 'expert' I spoke with justified working for free by the "connections and access" they received. When I asked if Google would be flying them out to the events they have access to to meet with the people they have connections with, they said they were told they would have to cover their own travel and lodging. I suppose if Google doesn't pay them, they never have to worry about any Experts connecting and starting a union.
We're talking about a multi trillion dollar company. Absolutely ridiculous.
> They don't just provide free labor, they also go to bat for Google's reputation, acting as marketing and PR liaisons to prevent backlash from users who've been failed by their official support processes.
This is so true. When I try to find a fix for some broken behavior in a major Google app the most common place I end up is a support forum with zero solution, no comment from any Google employee, and a volunteer support person saying how the defect isn't actually a problem but that you should submit feedback within the app.
These volunteers can't be helping at this point. Google's disdain for users is pretty widely recognized by now and seeing a whole bunch of frustrated users being blown off yet again doesn't help.
90% of the time when I run into one of these "experts" on their support forum, the solution is just that the customer is wrong in some way and such feature isn't needed or something like that. Do these experts even have any access to help solve problems? I'm not sure what makes them so compelled to do this for free for one of the biggest companies ever.
Google pays for support at different tiers from various locations, and the Experts usually have access to second tier support which is usually hosted in the US. They're supposedly ITIL, so they claim to use CSI to solve recurring problems, which is even worse, because now they're not compensating people for skilled non-entry level work. It's mostly nativity and being able to tell their families that they're "working with Google".
First tier support is hosted primarily in India where they don't know who or how many are doing the work, and it's nearly impossible to get escalated as a customer to the correct support tier outside of their script. It's frustrating and mostly useless, but they can handle volume changes quickly at sub-human costs.
This is 100% a race to the bottom. Years ago the Leapforce (now Appen as mentioned in the article) and Lionsbridge "contractors" could average over $18/hr. They would only work on Google tasks and adhere to Google's policies. The work seemed inconsistent (people would have 5 hours of work some days, none the next) and about what you would expect for AI training. No worker protections, benefits, etc.-- all while doing work, albeit opaquely, for one of the richest companies of all time. Why can't companies at least try to take care of their workers?
I worked for Leapforce from the UK in 2015-16 and I definitely didn't reach 18$/hr and the availability of work wasnt too great. Probably not bad if you have another gig you can do in the mean time though.
Former longtime Leapforce/RaterLabs preferred/senior rater here. There was always some exploitation going on, but it got way worse under Ruth Porat's tenure.
Standard Leapforce US English raters were paid $13.50/hr. If you were lucky enough to get promoted to "Preferred Agent" (PA), you made $17.40/hr. The criteria for PA promotion was basically to be in the top 5% or so on their quality metrics, plus various subjective quality requirements. Promotion rounds were irregular and random.
Leapforce's US English pay rates never changed over time. There were no raises, other than the aforementioned rare and seemingly luck-based promotion opportunity. I was one of the fortunate ones.
Leapforce raters were misclassified as 1099 contractors. There were no benefits. Training time was unpaid. Originally, raters could work as many hours as they wanted, subject to work availability. Around the end of 2013, they added a 40 hour/week limit because they were afraid a court or regulator might rule the "contractors" were employees and force them to pay retroactive overtime.
If a state dared to go after Leapforce or Google for misclassification, they would leave that state and fire the workers there. No severance, of course. There were numerous states they did not hire in, including their home state of California.
Task availability was usually very good for US English raters, but there were definitely periods (often around holidays) where that was not the case. There was never any communication about projected task availability. We certainly would have appreciated a quick heads-up of "hey, many of our engineers will be on vacation between mid-December and mid-January, so expect task availability to be light. Tasks will be added daily at 8AM PST, but probably won't last more than two or three hours."
I understand other locales were not as fortunate as we were for task availability.
Work availability for a particular rater was not necessarily as consistent. If you scored poorly on a monthly quality review or random quality audit, you could be abruptly suspended, fired, or limited to an hour or two a day until the next monthly review. There would be no prior warning of this. There were automated tripwires (which we called the "bot") that would flag and suspend you if your ratings were considered outliers, or if your task comments were too similar (even when there was a batch of tasks that were very similar), or if you worked too fast, or if you worked too slowly, or a million other reasons. If you were flagged by the bot, you had no work until the Leapforce quality people got around to reviewing you. There was no compensation for missed hours even if it turned out you hadn't done anything wrong.
Even more insultingly, Google loved to use quality reviews to informally implement guideline changes without actually changing the guidelines. If you weren't good at guessing their future direction, you could easily have your hours cut for the next month.
Around 2017, Google realized they (not just Leapforce or Lionbridge) could be sued for this misclassification nonsense, so they forced Leapforce and Lionbridge to convert their misclassified contractors to employees. Leapforce morphed into RaterLabs in what appeared to be some kind of liability-dodging scam. As employees, they cut us to 26 hours/week so they wouldn't have to pay for health insurance. There were still no benefits other than the minimum required by state law, if you were lucky enough to be in a state that required any. Nominal pay rates were cut in what seemed like a random and inconsistent (not even location based) way; depending on the cut, you got a few cents raise after accounting for the self-employment tax you no longer had to pay (which RaterLabs was quick to point out), or a pay cut.
There was a rater revolt around that time, but nothing seemed to come of it.
Appen bought Leapforce and RaterLabs later in 2017. Leapforce founders Daren Jackson and Caine La...
Thank you for all of this information! I've roughly followed it as a marketer (YMYL ranking factors from the guidelines type stuff) and as an observer-- it is interesting that the shifts lined up with the Porat era.
The two people I knew who did Leapforce also did human powered answers for ChaCha(?) where they were paid per answer versus an hourly rate, and then migrated to Leapforce at a lower date. They were both in US and I would remember them complaining on AIM over task availability-- sometimes they'd have work, others they wouldn't. They would take breaks from playing Counterstrike to see if new tasks were available and disappear if there was work available. With Leapforce, were there different rigid time allotments per task? E.g. 30 seconds to classify an image vs 2 minutes for ranking a webpage to be shown as a search result? If you were flagged as a low performer, was there any recourse or remediation, or just getting disabled and hoping for a review?
There were different time allotments per task, yes. In the Raterhub era, each task had an "Average Estimated Time" (AET) set by Google. A typical 5x5 side-by-side search result comparison task would have been 9 minutes, for example. I may be forgetting something but I don't think we ever saw the individual URL rating tasks (which I think I recall were 1.5 mins? maybe it was 2. It's been so long) after the Raterhub switch.
AETs were usually reasonable until the Ruth Porat era. After Porat, they seemed to be getting pretty aggressive about reducing them as far as possible to the point of absurdity. We could release tasks for insufficient AET. I suspect they added automated tooling in the Porat era to "optimize" AET in their favor. With only a few exceptions I can remember, AETs only went down, not up.
If you were flagged by the bot, there was nothing you could do except wait for Leapforce to review you. That could take anywhere from a few hours to several days depending on how busy they were. I wouldn't say there was any meaningful appeal route; I mean, you could email their quality team, and maybe you could plead for leniency, but I don't know anyone had any real success with that. (I don't have personal experience with that; as I said, I was a PA/senior, which was a quality-based designation, so it follows that I tended to be pretty good at the job! But I always felt bad for other raters in chat who had to deal with that, seeing their livelihoods put at risk in such an arbitrary and robotic manner. If someone isn't a good fit for the job, then yes, it's not going to work out, but I do believe they should be given feedback and guidance on how to improve and a fair chance to do so.)
I'll add that even though the pay rates were nominally quoted per hour, under Leapforce, it was really task-based because they wouldn't pay more than AET. I know raters had their invoices rejected if their productivity was materially under 100%.
I assume they pay for all time worked under RaterLabs, due to employee wage and hour laws. Of course, they can still fire a rater for not meeting productivity requirements.
I thought the management theory was to in-source your core-competencies and out-source everything else. Surely quality search results is one of Google's core competencies!
Totaly oversimplification. If google wanted you only to click on ads, it would be a type of yellow pages. What it is (as it was with newspapers and magazines a time ago): A place where you can place paid ads based on context. If you remove the context, it becomes worthless. So while everyone is complaining about search quality, its an uphill battle for any search company. Like PR tries to sneak in Ad-Material into reguluar journalism, companies try to manipulate search results with SEO ranging form keyword-stuffing to bait and switch search pages. (which are one reason why google needs humans and not ai to verify).
Another prime example of how ML and AI aren't mature. With all the data google collects, they have to rely on minimum wagers to "find" the best results.
In the sense that ML and AI aren't yet AGI, yes, but this doesn't mean that Google's ML and AI aren't very useful and benefit from having another source of training and validation data.
Actually, ML/AI would not know by the rules of Google for "Quality" how to rate content, since content is ever changing and diverse as it gets. If you spend some time reading the guide (https://static.googleusercontent.com/media/guidelines.raterh...) you will realize that its not about the "best" results but on classificaion / verificationin the YMYL (Your Money, Your Life) areas that should not be left to machines in the first place. Money is mainly low because there are so many people that can do the job and so many that are needed to redundantly rate the content.
If Google can't use it's vast array of user data to tell the difference between SEO bots and non-bot users, what's even the point of it collecting all that fucking data in the first place?
I would assume that if Google is paying 10$/hour for humans to rank content, someone else is paying 11$/hour to fool it with a human generating content, clicks, etc.
Additionally, people writing bots learn just like Google learns, and start generating what appears to be legitimate interactions. Same algorithms that fit to critique between 'bot' and 'non-bot' class can be used to create bots that fall into the 'non-bot' category.
It's a losing battle - but what we get is better than we would get if they collected less data.
That’s what’s confusing in the articles that have come out about quality raters.
Quality Raters don’t actually rank websites.
Quality Raters look at websites and answer questions about them. Google then uses that data as a small part of their algorithm to adjust some factors about those sites.
Quality Raters don’t actually rank pages on google.
civilized wrote about enhancing the ranking for the person who casts the vote. Using these votes to do changes in the Google ranking for the general public can of course easily be abused.
2. After clicking on that link, do you go back to the search or not?
3. How long do you stay there before coming back?
Adding an explicit up/downvote would capture some additional information, but it would overlap with the above, and most users would ignore the button. I’d guess Google has at some point experimented with this and found it wasn’t worth keeping.
I'm sure they have all sorts of fancy stuff but at the end of the day, there's a lot of SEO garbage at the top of the rankings that I would really love to downvote.
> Adding an explicit up/downvote would capture some additional information, but it would overlap with the above, and most users would ignore the button. I’d guess Google has at some point experimented with this and found it wasn’t worth keeping.
I love stackoverflow content, but hate the surface, e.g. how it loads jquery from google (???) and if I disable it, then the content breaks, because obviously the only way to implement a toggleable div is when you send your users browsing history to google directly..
Note that it's often difficult to tell if these signals are good or bad. For example:
1. Clicking on a link is often a negative signal. If you're searching for "when was barack obama born?", hopefully you get the answer on the search results page itself, so you never have to click. Or, the caption for each search result should have a snippet containing the answer.
2. If you stay a long time on the search result you click on, is it because the information is hard to find or because the page is deep and useful?
I used to work on Search measurement at YouTube and Facebook, and run a human evaluation platform now. Here's an example what they can look like when you use high-quality raters: https://demos.surgehq.ai/google-vs-bing/
Not exactly true. Most people would infrequently use the buttons, but when they do, you can be sure they want it that way.
I would suggest that the big reason not to go down the avenue of social buttons is the inevitable gamification of it. Already we have have ratings farms for apps and amazon. Of course we will have similar if search implemented this.
We know they probably use CTR as a ranking factor. However, a site could just literally add JS code on the page and disable the back button. So I don’t think that would work out that well for Google.
Since a few years there’s been plenty of cheap tricks that would’ve earned you a manual ban/downrank back in the day that Google now turns a blind eye to.
Google used to do this, but they rolled it back because the hard honest truth is most people don't like taking surveys. Behavior is a better and more available signal.
How are you supposed to “engage” with Google search once you’ve downvoted all the spammy ad-ridden garbage and actually get useful results on the first non-ad link?
Surprisingly badly paid. I wonder how strongly quality would track pay? Probably they act as a CAPTCHA mutually cross checking and so QA is by volume, and not coincidentally demands they work in isolation.
Quality control is often indeed by volume. When I was at Google / Facebook, search evaluations would typically have 5 raters each. One of the difficulties, though, is that search evaluation can be quite subjective. And honestly, most of the search engine raters we used were low quality and did a bad job (so the aggregate answer was often nonsense).
This. In the early days of my SQR rating time, the rating was not sliced into time per rating. And you had to write a short blurb about your reasons, often using the acronyms so it read like "NR, LQ / SPAM". Later you would see the other SQR notes and would wonder if both of you had seen the same page.
When those reports come out that “The average Google employee makes $220k a year” or whatever, it’s totally misleading since it doesn’t account for all the outsourced labor at $10 an hour that is critical to tuning their algorithms.
I did voice training tasks for Google through a 3rd party I can't remember the name of. I was recruited via /r/beermoney or something. I enjoyed it, and it was easy work that fit around my schedule. It also gave a feeling of improving the lives of users, which was cool.
Its better paid than Amazon Mechanical Turk afaik (not saying it should not be paid better). The tuning is done literally by thousands world wide and since every rating task is done by multiple quality raters (afaik side by side is done by at least 5), it seems to be a cost vs. scale factor.
Amazon Turk is more of a platform for small business to hire other workers for micro-tasks. Raterlabs is used only by Google to improve their core product.
I don't think that's misleading at all. Why would I assume that non-Google employee compensation would go into the calculation for what the average Google employee makes?
As someone else pointed out, this pays substantially more than AWS Mechanical Turk. To be honest, it's difficult for me to imagine an easier job than this one. I mean, most people who would do this job could easily now make more at McDonald's or Walmart or whatever. But I think jobs at McDonald's or Walmart are considerably harder, so I don't see why this job should pay the same.
Google has publicly described these people as it’s “extended workforce.”
Employee is generally used as the legal definition, but if you’re only job is to work for company X under their explicit micromanaged directions you fit the colloquial definition of employee even if your paycheck is signed by a different company.
No, not generally, because it depends. With mandatory paid social security and cheaper rent, it is somehow lower (for example) germany - but not for each every job, since some are in high demand (ml/ai specialists for example) . Google employees in switzerland are paid higher than in the US for similar tasks due the cost of living higher. (Same with other companies like facebook, twitter, etc.)
Ex Google Search quality Rater here, based in Germany. From 2004 to 2006 I worked on SQR and "special projects", with a much better pay (maybe because of german circumstances). I've been tracking the SQR-guide since then (public version here https://static.googleusercontent.com/media/guidelines.raterh...), so I have still some current understanding. If you have questions, I'd be happy to try to answer them since my NDA has expired ;-).
Its an interesting boring job in its core. At least when I did it. You agree to see theoretically everything that people search for and try to rate it according to the linked manual. In my time - really long ago - britney spears was one of the most common searches (I remember it probably wrong, but the test I had to do was half rating questions on spears and tom hanks). But I also got to rate "donkey s*x" and things like "divorce help" which makes you think about the impact search results can have.
The boring part was that it is kind of repetetive to rate and you seem to learn how simplistic people search most of the time (but then again, only common/frequent searches are tested, at least most of the time). And it was enlightning how search spammers adapted to each no measure to keep them out. I still know the names of the people who were infamous at that time for setting up fake, keyword stuffed "blogs" to get people to their shop.
Good question, disapointing answer: Only time will tell. I have a hunch however, from stylometry work. Its that for short texts "generated" text will be virtually indestinguishable and this will grow as ml/ai systems advance to longer and longer texts. This will probably lead to different techniques to rate content (like more emphasis on the Your life/Your money factor).
“Your ratings will not directly affect how a particular webpage, website, or result appears in Google Search, nor will they cause specific webpages, websites, or results to move up or down on the search results page. Instead, your ratings will be used to measure how well search engine algorithms are performing for a broad range of searches.”
Is this accurate? Or is the guide bending the truth to discourage malicious raters from trying to influence site rankings? Google only uses the rater data to evaluate its search algorithm and doesn’t use the rater data to teach the algorithm? Seems like a missed opportunity.
Mostly five raters indenpendently rate the same task and are randomly chosen from a pool. This should reduce SQR conspiracies to near zero. Also, afaik, google engineers use the SQR Feedback to enhance algorithms (example: spam detection) Second: In my active time - long ago - white text was still a thing to make keyword stuffing "invisible". As a result of the SQRs work (and maybe a lot of complaints about bad search results ;-D), the next revision of search ranking "killed" that method - at least for some time - for all pages and not only the rated ones (which were only a sample of all in existance).
The quote matches my own understanding of the system from when I worked on Google Search circa 2005. Human ratings were used to facilitate an automated test that could tell you if your new algorithm actually made search results better or not. That let search quality engineers iterate on ideas quickly. But the human ratings were not directly used to produce search results, since that wouldn't scale (only a relatively small number of queries actually had rated results).
Obviously 2005 was a long time ago though and everything could have changed since then.
If there is a web site with a LOT of content, does Google get SQR for each page or do they get SQR for "a couple of pages" and apply that rating for the entire site?
When you rate "accruacy" of information or "authorativeness" of a site, how do you know the site is "an authority"? are you guessing?
SQR is used to test if google search results are "right" across most searches, not if a website is "right" in general. Rating of a site from the point of google relies on other data points like speed, support of https, links to the site (aka page rank) and a lot more. Have a look at the SQR Guide (PDF Link in my comment above) for examples how Google classifies information results. (Especially the YMYL - Your money or your life classification and Expertise, Authoritativeness, and Trustworthiness (E-A-T))
Google absolutely employs more raters than software engineers. This is the big fraud of Google search. It’s not PageRank, which hasn’t been used for years. It’s exploited human labor. That’s the real lie behind Google.
First sentence is true. Second is not, as is the third. I believe you don't know what you are talking about. A software engineer for example writes code that scales across thousands of systems and might be (human) language independet. A SQR tests samples of searches if they match google quality standards. As this cannot scale across all websites it is done with a subset of the web, like a one in a million ratio. It has to be done by 5 people to get an average so it is independent from one rating = one result. And it has to be done in context to countries / cultures.
"The raters Yahoo Finance spoke to work from home, but they have no clue who their direct boss is, or if they have one boss or many bosses. They don’t even know their boss’ (or bosses’) name."
"Across the board, the raters Yahoo Finance spoke to worry about waking up to a pink-slip-email, and two said they know of instances where people haven’t received an email at all — the worker’s account was just deactivated. "
Is there a way to sign up to tell them Gmail's spam detection doesn't work anymore? Seems like anyone can get through by putting "invoice" in the subject.
116 comments
[ 2.0 ms ] story [ 198 ms ] threadMore like it matches some keywords like run-of-the-mill WordPress malware generates. I've seen plenty of both. Not to mention the similarity to the masses of biz/buzz/xyz spam websites.
It's not like you need a PhD to confirm if a search result sucks. Results are supposed to be good enough for the average user, so you need common sense and some on-job training to do a good job.
And having 10 common-sense guys churning through user feedback @$11/h is better than 2 geniuses @$55/h.
As one comment says, it probably beats fliping burgers.
They cannot rate all results anyway, so why not get the best possible evaluations?
Shouldn't quality data beat quantity if you extrapolate from the evaluated samples to the entire index?
Spammers target the lower part of the bell curve. If you let that part evaluate the quality of the results, do you think that they will recognize spam?
For example, here are some queries in my history - would the average person understand what I'm looking for and what a good search result would be?
"neovim worktrees"
"django models through"
"pandas econdb"
[1] https://developers.google.com/community/experts/
I imagine there are Google executives in a room somewhere laughing their full heads off every time they think about how there are people who do free work for Google who think they are making the world a better place or something.
So when you go to a store that Google Maps says is open, and find it's closed, you refuse to correct the hours and go "MUHAHAHAHA...I refuse to help Google, take that, corporate scum!"
The thing is that business owners do update the info on Google Maps, and have an incentive to do so, but only volunteers update OSM.
OSM shines where businesses don't care but volunteers do: trails in a park, non-commercial POIs, etc.
I guess it's nice that ordinary people try to do these updates for Big G and Apple. But they're often wrong, or they presume a one-day change is a permanent change, or that it applies to every location on the planet.
Back when it was new and interesting, TripAdvisor used to send people hats if they reviewed a new or unusual hotel.
I wish I hadn't lost my hat before I had a chance to wear it while checking into a hotel.
These "Google $title Experts" work as uncompensated customer support (a legal requirement that they treat as such, if the quality is any indicator) on various social media outlets like Reddit and Twitter. They don't just provide free labor, they also go to bat for Google's reputation, acting as marketing and PR liaisons to prevent backlash from users who've been failed by their official support processes.
One 'expert' I spoke with justified working for free by the "connections and access" they received. When I asked if Google would be flying them out to the events they have access to to meet with the people they have connections with, they said they were told they would have to cover their own travel and lodging. I suppose if Google doesn't pay them, they never have to worry about any Experts connecting and starting a union.
We're talking about a multi trillion dollar company. Absolutely ridiculous.
This is so true. When I try to find a fix for some broken behavior in a major Google app the most common place I end up is a support forum with zero solution, no comment from any Google employee, and a volunteer support person saying how the defect isn't actually a problem but that you should submit feedback within the app.
These volunteers can't be helping at this point. Google's disdain for users is pretty widely recognized by now and seeing a whole bunch of frustrated users being blown off yet again doesn't help.
[1] https://en.wikipedia.org/wiki/AOL_Community_Leader_Program
First tier support is hosted primarily in India where they don't know who or how many are doing the work, and it's nearly impossible to get escalated as a customer to the correct support tier outside of their script. It's frustrating and mostly useless, but they can handle volume changes quickly at sub-human costs.
It doesn't work that way. Contrariwise.
Standard Leapforce US English raters were paid $13.50/hr. If you were lucky enough to get promoted to "Preferred Agent" (PA), you made $17.40/hr. The criteria for PA promotion was basically to be in the top 5% or so on their quality metrics, plus various subjective quality requirements. Promotion rounds were irregular and random.
Leapforce's US English pay rates never changed over time. There were no raises, other than the aforementioned rare and seemingly luck-based promotion opportunity. I was one of the fortunate ones.
Leapforce raters were misclassified as 1099 contractors. There were no benefits. Training time was unpaid. Originally, raters could work as many hours as they wanted, subject to work availability. Around the end of 2013, they added a 40 hour/week limit because they were afraid a court or regulator might rule the "contractors" were employees and force them to pay retroactive overtime.
If a state dared to go after Leapforce or Google for misclassification, they would leave that state and fire the workers there. No severance, of course. There were numerous states they did not hire in, including their home state of California.
Task availability was usually very good for US English raters, but there were definitely periods (often around holidays) where that was not the case. There was never any communication about projected task availability. We certainly would have appreciated a quick heads-up of "hey, many of our engineers will be on vacation between mid-December and mid-January, so expect task availability to be light. Tasks will be added daily at 8AM PST, but probably won't last more than two or three hours."
I understand other locales were not as fortunate as we were for task availability.
Work availability for a particular rater was not necessarily as consistent. If you scored poorly on a monthly quality review or random quality audit, you could be abruptly suspended, fired, or limited to an hour or two a day until the next monthly review. There would be no prior warning of this. There were automated tripwires (which we called the "bot") that would flag and suspend you if your ratings were considered outliers, or if your task comments were too similar (even when there was a batch of tasks that were very similar), or if you worked too fast, or if you worked too slowly, or a million other reasons. If you were flagged by the bot, you had no work until the Leapforce quality people got around to reviewing you. There was no compensation for missed hours even if it turned out you hadn't done anything wrong.
Even more insultingly, Google loved to use quality reviews to informally implement guideline changes without actually changing the guidelines. If you weren't good at guessing their future direction, you could easily have your hours cut for the next month.
Around 2017, Google realized they (not just Leapforce or Lionbridge) could be sued for this misclassification nonsense, so they forced Leapforce and Lionbridge to convert their misclassified contractors to employees. Leapforce morphed into RaterLabs in what appeared to be some kind of liability-dodging scam. As employees, they cut us to 26 hours/week so they wouldn't have to pay for health insurance. There were still no benefits other than the minimum required by state law, if you were lucky enough to be in a state that required any. Nominal pay rates were cut in what seemed like a random and inconsistent (not even location based) way; depending on the cut, you got a few cents raise after accounting for the self-employment tax you no longer had to pay (which RaterLabs was quick to point out), or a pay cut.
There was a rater revolt around that time, but nothing seemed to come of it.
Appen bought Leapforce and RaterLabs later in 2017. Leapforce founders Daren Jackson and Caine La...
The two people I knew who did Leapforce also did human powered answers for ChaCha(?) where they were paid per answer versus an hourly rate, and then migrated to Leapforce at a lower date. They were both in US and I would remember them complaining on AIM over task availability-- sometimes they'd have work, others they wouldn't. They would take breaks from playing Counterstrike to see if new tasks were available and disappear if there was work available. With Leapforce, were there different rigid time allotments per task? E.g. 30 seconds to classify an image vs 2 minutes for ranking a webpage to be shown as a search result? If you were flagged as a low performer, was there any recourse or remediation, or just getting disabled and hoping for a review?
AETs were usually reasonable until the Ruth Porat era. After Porat, they seemed to be getting pretty aggressive about reducing them as far as possible to the point of absurdity. We could release tasks for insufficient AET. I suspect they added automated tooling in the Porat era to "optimize" AET in their favor. With only a few exceptions I can remember, AETs only went down, not up.
If you were flagged by the bot, there was nothing you could do except wait for Leapforce to review you. That could take anywhere from a few hours to several days depending on how busy they were. I wouldn't say there was any meaningful appeal route; I mean, you could email their quality team, and maybe you could plead for leniency, but I don't know anyone had any real success with that. (I don't have personal experience with that; as I said, I was a PA/senior, which was a quality-based designation, so it follows that I tended to be pretty good at the job! But I always felt bad for other raters in chat who had to deal with that, seeing their livelihoods put at risk in such an arbitrary and robotic manner. If someone isn't a good fit for the job, then yes, it's not going to work out, but I do believe they should be given feedback and guidance on how to improve and a fair chance to do so.)
I assume they pay for all time worked under RaterLabs, due to employee wage and hour laws. Of course, they can still fire a rater for not meeting productivity requirements.
There is nothing wrong with semi-supervised learning.
Maybe that's why the search "X reddit" is so popular - it's got a bit of that social validation.
Correct, SEO bots.
Additionally, people writing bots learn just like Google learns, and start generating what appears to be legitimate interactions. Same algorithms that fit to critique between 'bot' and 'non-bot' class can be used to create bots that fall into the 'non-bot' category.
It's a losing battle - but what we get is better than we would get if they collected less data.
Quality Raters don’t actually rank websites.
Quality Raters look at websites and answer questions about them. Google then uses that data as a small part of their algorithm to adjust some factors about those sites.
Quality Raters don’t actually rank pages on google.
> Quality Raters look at websites and answer questions about them.
Wrong. I was never asked any questions beyond to rank to site quality for a query
> Quality Raters don’t actually rank website
Wrong again. This is what we did. We assessed website rankings for specific queries.
> Quality Raters look at websites and answer questions about them.
Wrong. At least in my case. I was never asked any questions beyond to rank to site quality for a query
1. Do you click on the link in the results page?
2. After clicking on that link, do you go back to the search or not?
3. How long do you stay there before coming back?
Adding an explicit up/downvote would capture some additional information, but it would overlap with the above, and most users would ignore the button. I’d guess Google has at some point experimented with this and found it wasn’t worth keeping.
Sounds pretty ripe for abuse too.
civilized wrote (emphasis by me): "and Google enhances its ranking for me based on that".
I love stackoverflow content, but hate the surface, e.g. how it loads jquery from google (???) and if I disable it, then the content breaks, because obviously the only way to implement a toggleable div is when you send your users browsing history to google directly..
1. Clicking on a link is often a negative signal. If you're searching for "when was barack obama born?", hopefully you get the answer on the search results page itself, so you never have to click. Or, the caption for each search result should have a snippet containing the answer.
2. If you stay a long time on the search result you click on, is it because the information is hard to find or because the page is deep and useful?
I used to work on Search measurement at YouTube and Facebook, and run a human evaluation platform now. Here's an example what they can look like when you use high-quality raters: https://demos.surgehq.ai/google-vs-bing/
I would suggest that the big reason not to go down the avenue of social buttons is the inevitable gamification of it. Already we have have ratings farms for apps and amazon. Of course we will have similar if search implemented this.
I heard it didn’t work very well for them. The results weren’t very good or helpful.
Sad story.
As someone else pointed out, this pays substantially more than AWS Mechanical Turk. To be honest, it's difficult for me to imagine an easier job than this one. I mean, most people who would do this job could easily now make more at McDonald's or Walmart or whatever. But I think jobs at McDonald's or Walmart are considerably harder, so I don't see why this job should pay the same.
Employee is generally used as the legal definition, but if you’re only job is to work for company X under their explicit micromanaged directions you fit the colloquial definition of employee even if your paycheck is signed by a different company.
Did you work from an office or remotely? Did you interact with other raters?
Was there an interview process? How about performance reviews? How did Google ensure quality?
“Your ratings will not directly affect how a particular webpage, website, or result appears in Google Search, nor will they cause specific webpages, websites, or results to move up or down on the search results page. Instead, your ratings will be used to measure how well search engine algorithms are performing for a broad range of searches.”
Is this accurate? Or is the guide bending the truth to discourage malicious raters from trying to influence site rankings? Google only uses the rater data to evaluate its search algorithm and doesn’t use the rater data to teach the algorithm? Seems like a missed opportunity.
https://static.googleusercontent.com/media/guidelines.raterh...
Obviously 2005 was a long time ago though and everything could have changed since then.
It was OK at first and then it started having me watch increasingly extreme porn. I think I made it about 3 months
When you rate "accruacy" of information or "authorativeness" of a site, how do you know the site is "an authority"? are you guessing?
This is some Kafka-esque shit. Unreal.
[0] https://webcache.googleusercontent.com/search?q=cache:3HkKSZ...