This is a serious issue. On the one hand, our current system makes a one-star review far, far more consequential than it ought to be. If EVERYONE leaves positive reviews, than a single negative is very powerful.
On the other hand, leaving positive reviews out of politeness does a disservice to others. I hosted a couchsurfer once (a woman) who told me about her experience with another host. This guy had 10-20 positive reviews. And he was a total creep. Hit on her the entire time, asked her to sleep with him, and was intimidating. She left as soon as she could.
But she didn't leave a negative review! And clearly, other women must have had her experience, yet they said nothing either. This guy was left free to continue his horrible behavior.
Trusted reviews. A user should be able to trust others, and to filter reviews based on that. Integration with social networking would allow for friends of friends trust as well. It would be far from perfect but it would better replicate the experience pre-internet of asking people around you, which works pretty well.
Kind of like eBay buyer reviews. They are completely useless as the only person that will likely ever give a bad review is a small seller that has been completely jerked around and doesn't at all rely on ebay for income.
The solution seems pretty clear: set a high bar before giving a one-star review, but be ruthless about giving it for behavior that's beyond the pale, like in your story.
Another way to phrase the heuristic might be: be lenient toward incompetence (including the bad turn in OP), but strict toward malice or unethical behavior.
Yet another phrasing: if the guy was CEO of a large company, say mozilla, should his behavior get him fired? If so, give him a one-star review. I find such thought experiments changing relative status to be quite useful.
You could perhaps give a two, three or a four star review for various degrees of incompetence.
It depends on the review system. The article was addressed at five stars systems, where one star reviews do tend to have too low a bar for some, whereas others never leave negative reviews of any sort.
Other system are good/bad and there it's a harder choice whether to leave criticism. There's incompetence, and there's something that wasn't good but doesn't necessarily signify systemic incompetence. Leaving a review marked "bad" can't distinguish between those two, at least in the eyes of those that perceive it.
Give accurate reviews, and provide context in a text format. If a system will fail because it has to meet an arbitrary quota (eg Uber's 4.6 stars), then that's a problem for the system to work out, not the person providing the review. The solution is simply to give dispassionate, honest reviews.
The author projects his own hypersensitivity to the well-being of the Lyft driver on the general population of Lyft customers. Or I'm just an asshole :)
I'm pretty sure it's the latter. Most decent people, when faced with their own emotions versus financially ruining others will err on the "keeping their mouth shut" side.
While some might feel a sense of duty and obligation to the rest of the customer base to leave honest reviews... I think most selfless and considerate people will weigh that against the impact it has on the person they are reviewing and wonder if its really worth being an asshole.
It's 2014. As cruel as it sounds, if you haven't figured out that you need to underpromise and overdeliver to your customers, your days are numbered.
There will be growing pains, people will suffer greatly, but a wonderful new system will hopefully arise from the ashes.
I do think that there is a better way, and that most users know how to discount a really bad review. It's kinda hard to hide bias when you are angry. So, the system will eventually balance out.
Bottom line, I don't know how to fix this system, but I do know how to survive in it. (hint: underpromise, overdeliver)
How do you underpromise and overdeliver for a ride service? Sure he shouldn't have taken the detour, but should he lose his job because of the mistake? The answer is unknown to us.
One thing that can be said is a customer who becomes sick with power is hard to deal with. If someone gets in the car and makes unreasonable demands with the threat of a bad review what do you do?
Should he lose his job? That's not really a decision for the customer to make. If he loses his job because of one justifiably bad review, that's due to the ride service's policy, not the client.
You are skirting my main point about unjustifiable reviews. The example in the article is certainly fire worthy, I will not contest that. (I was attempting to not claim that the writer was incorrect in his actions or that the actions of the driver were okay)
One possible partial solution is to express the quality of the source of a one-star review. A one-star review from someone who has posted many, many 4 and 5-star reviews should carry some weight, while a one-time reviewer or someone with a history of giving many 1-star reviews should not. Perhaps color coding such reviews, or preferentially ranking reviews from the reliable at the top. This might lead to some legitimate 1-star reviews being ignored and it is at odds with the anonymity most people prefer on the internet, but it's better than presenting all 1-star reviews equally. Companies that make employment decisions based on these reviews might also choose to weight reviews based on the reliability of the source as well.
> A one-star review from someone who has posted many, many 4 and 5-star reviews should carry some weight, while a one-time reviewer or someone with a history of giving many 1-star reviews should not.
Shouldn't the reverse be true as well? A 5-star review from someone who gives lots of a 1- and 2-star reviews should mean a lot more than a 5-star review from someone with little history or a history of giving many 5-star reviews.
Really, a big problem here is cultural norms vary and subjective ordinal ratings don't mean the same thing to different raters -- ISTR studies showing that White Americans tend to rate things near the top of a scale if they are acceptable, toward the middle but still above the center even when they are unacceptable, and below the center only rarely -- whereas different nationalities and ethnicities had measurably different patterns of ratings compared to their actual descriptions of their responses to products. If that's true, a one-sided selective discounting of reviewers that give lots of low-star reviews may be effectively saying "I want reviews from White Americans to count the most."
That got me thinking. If you averaged the users' total reviews and then centered their score on a slightly larger scale then you could accomplish something like that:
So if I rate a bunch of things and my average review score is 4, then things I rate as a "1" would have an adjusted value of "2", and things I rate as a "5" have an adjusted value of "6". That would "find everyone's center" and maybe score things more evenly.
Is that because they're using the same scale as for grading school performance? So, three stars out of five is seen as 60% which is only barely passing, when in fact it should be for average service.
Weighting reviews based on the variance in the history of that reviewer seems like a great idea to me, because it gives preference to reviews that probably carry more information in them. A review of five stars from someone who always hands out five star reviews, is meaningless and doesn't inform anyone else of anything. Likewise for one star reviews. In fact, intuitively I think maybe even a five star review from someone who usually hands out one star reviews, or vice versa, isn't very informative either? Reason being that these people clearly don't put a lot of thought into their reviews in the first place.
The experience was pretty bad, but who are we to take away someone else’s livelihood?
Someone who drives so badly that they endanger the lives of their passengers in a paid service? Whether or not he's a nice person, you should rate the action appropriately. If you're the only bad review, the driver's rating should stay high.
Did the driver learn from the bad mistake? How would you know? Someone can be nice and still clueless. Just rate the driver appropriately.
Personally, I find the bad reviews to be much more informative than the good reviews. Star ratings are largely meaningless (unless the thing is at 1 star or less). Read the bad reviews - if they're just frothing trolls of the kind that complain about everything, the item is probably fine (and you can check out more reviews at that point). If the bad reviews are raising valid complaints - and 'drove so badly we nearly died to his manoeuvres' is a valid complaint. If several people are making that complaint, then there's likely to be substance to the claim.
Well, I wouldn't say selfishly, since the reviewer is doing it out of kindness to another, but yes, you could argue that in this particular case, it's endangering.
1) Evaluation inflation/simplification. Happens with grades, happens with reviews. 4 stars becomes the new 'adequate'. If you're under 4 stars, you're bad. Everyone subconsciously knows this, hence the thinking 'if I don't give them a 5 star review they might get fired.'
A more meaningful, diverse group of metrics is likely the answer to this. Get rid of 'overall' ratings. Rate drivers on their friendliness, safety, timeliness, etc. Uber can decide on some algorithm for determining when to let drivers go, and users feel less bad about negative reviews: 'Well, I gave him a 2 on safety but he was really friendly so at least I gave him a 5 on that."
2) Not enough separation between reviews and individuals. If I leave a review on amazon, I don't want that being tied to my account. At all. I don't want to see the seller's replies to my reviews. If someone finds out my amazon account name, they should not be able to tie individual reviews back to me. Batch reviews up in groups of 5, randomize them, and post them all at once. Don't let there be any possibility of tying reviews back to individuals. The fear of someone seeing a mean review I wrote and forever thinking I'm an asshole is a serious barrier to negative feedback (barring extraordinary circumstances).
The other problem with the one-star reviews is that since the livelihood of the person (or company) is directly affected by it - they fight tooth and nail to hide it or somehow drown it by putting fake 5 star reviews on top of it. Something which they won't necessarily resort to.
This is of course based on my own personal experience but at least on tripadvisor.com I have seen this happen a lot. Every time there is a 1-star review of a hotel you can can bet that there will be 3-4, 5 star reviews on top of it by reviewers who have left 1 review in total (suggesting they are fake or outsourced) just so that the 1-star review can be bumped to page 2 or drowned.
I just left a one-star review for a book. But I think it deserved it and I spelled out why. People need to be honest about their thoughts and experiences, and sometimes that means giving the worst possible rating.
I mean, you could review one star for things that are, like, actually poisonous or dangerous, but that doesn't really work either. As beloch suggests below, a weighting system is a good answer, and Amazon does a pretty good job (I think) of letting you look through positive and critical reviews. Kinda different when you add the human element, but still.
If people have the attitude that a high review is mandatory, then there's effectively no point to review systems.
Sorry, but not every book should be read. Not every restaurant should stay in business. Not every Uber driver should be driving. Giving accurate reviews allows the system to work, both for customers and (eventually) for providers (by steering them out of work they're inept at).
I think it is the five star reviews that are the problem, because we are all taught that, for services, leaving anything less than five star reviews will make trouble for them.
The fact is the average Uber ride is going to be average. The average person shipping stuff on Ebay is average, the average person selling on the AMZN marketplace is average, and so forth.
If you're supposed to leave 5 stars to say service was adequate, then there is no way to reward those who go above and beyond.
if you take a cab, and the experience its terrible, what do you do? if you take an "online ride share service"? and the experience its terrible, what do you do? are the answers different? are the same?
On freelancing sites, if you give someone a 4 star review they come back and dispute it. 4 star is actually a bad rating on these sites. A better way to judge someone is based on how many clients/hours they have done on that particular site.
Online ratings are so depersonalized that they are near meaningless without a large sampling body to back them up. Even then, most online ratings systems appear to be somewhat gameable, which renders them worse than useless.
Even though I consume online resources, I rarely leave ratings, since I can't really tell that it's worth the effort.
An approach used in psychology tests for questions people are embarrassed to answer: add a random number to each rating.
The aggregate effect is the same (if a random number from 1-5 is added to each reviewer's review, on average, you just need to subtract 3 from the average review score), but reviewers may feel more-free to be honest. More reviews are required to get an accurate review, but the reviews may be more accurate.
The larger the random spread is compared to the rating's dynamic range, the greater the obfuscation, and the more reviews you need to learn something new.
Perhaps the problem lies with having too many options without any agreed upon standards. What's the difference between a 1 and a 2 star ride? Anecdotally it would seem like it's always going to be 5, 4, or 1. Seems like the 1-5 system is antiquated and needs to go.
I can easily think of what would make a ride differ between 1 2 3 4 and 5. I'm disturbed by the trend of ratings being reduced so that the reviewer doesn't have to put any thought into it. It's encouraging intellectual laziness. I felt the same about YouTube. If you must then maybe put suggestions next to the 1 through 5 ratings like NewGrounds does. I don't want to lose the ability to say something is average but not great or terrible, or between average and terrible, or between average and great.
1 star (and 5 star) are always going to be the biggest buckets. Even if you try to be very fair and strict, what does a driver that doesn't show up get? 1 star. What about a driver that stabs you? 1 star. Crashes? 1 star. Gets lost and makes you late? 1 star. Smells bad and a bit rude, but gets you there on time? Maybe this is finally 2 stars. These are all bad, but getting stabbed is a lot worse than not showing up or getting lost.
Likewise for 5 stars. Friendly, safe, on time, gives you a cold bottle of water, have a nice conversation. 4 or 5 stars. Chases you down with the bag you forgot in his car? 5 stars. You fall in love and get married? 5 stars.
Yeah, you have people giving a 5 for anything from a 2 to a 5 experience. Then you have people giving a 4 for anything but the most amazing ride they have ever been on.
I know that Uber does background checks on their drivers, but do they also provide training to their drivers? Are drivers evaluated first by an Uber employee?
There has to be room in any rating system for the lowest rating, for those who deliver the worst possible service - obviously.
The problem I see is that the perception of what warrants a given rating varies pretty widely, and most sites do a terrible job of trying to help their users calibrate their ratings. I monitor Yelp regularly for my girlfriend, who owns a wine bar. Her ratings are generally excellent - over 4 star avg., with most ratings at 5 for the past two years. However, every once in a while we see an anomalous rating - for instance:
- one current 3-star says "I have to be honest, I didn't have anything to drink, I was with friends waiting for the midnight showing at the theatre next door. However, it was packed and happening and EVERYBODY was having a good time."
So this person, who wasn't a patron, but who had nothing but positive impressions, still left what ultimately will be considered a sub-standard rating. This is actually most common for us with 3-stars - they are generally very positive reviews, with nothing negative at all, but with what most Yelpers would consider a sub-standard rating.
Mostly when I look at Yelp ratings, it follows the same pattern - 80% of the ratings seem to correlate closely with the reviews and the typical Yelp ratings standards. 10% seem to be using a different rating standard. And 10% are just weird.
Places like Yelp who have many reviews from the same person should be able to curve the reviews onto a common standard. At least they should be able to say that a 3-star review is, e.g., +1 sigma for this particular person.
Any data set will have outliers that need to be filtered out, and I'd hope that those interpreting the data would know that. In particular, there's the selection bias that people who are pissed off are much more likely to actually give a review in the first place.
Firing someone for a single one-star review is asinine.
This is one hell of an impediment to the success of these services. Failing at quality control is going to drive away most of their current client base. Forget luxury services... Give me an old school car service any day.
An "n/n" review for me means basically "I can't possibly imagine this service having been any better." That's a pretty hard standard to meet, and hence I almost never give them. An n/2 rating means "adequate, no specific complaints, no specific praise". If you don't calibrate the rating like that, you lack dynamic range on either side.
It seems to me that human actions based on reviews require human judgement. Taking away someone's livelihood because someone gets a lot of 4 star but positive reviews strikes me as dangerously wrong. Taking away someone's livelihood because he or she callously puts others in harms way is a different story.
There's a huge difference between:
1 star: I waited for an hour for my pickup and this caused me to miss my very important appointment. Yes, traffic was horrible today, and so maybe it isn't the driver's fault but this was a very bad experience I don't want to repeat. You folks need to do a better job of setting expectations!
and
1 star: my driver narrowly missed rear-ending the car in front of us because he was distracted. He didn't even apologize!
In the second case we might blame the driver. In the first case, who gets the blame depends on the system.
Paraphrasing Tolstoy, "All 5 star reviews are alike; each 1 star review is unhappy in its own way."
Five star reviews tend to be fluff, one star reviews tend to be specific. I read them first, I look for dealbreakers. No amount of "I love it" can outvote "claims X but can't really X properly".
Exactly. On Amazon (the review system I'm the most familiar with), if I find several one-star reviews that say some device was dead on arrival or a technical book was full of factual errors and typos, I'll definitely think twice before buying it, even if 90% of the reviews are five-star. I'm always grateful for the people who took the time to write honest and informative negative reviews.
56 comments
[ 3.1 ms ] story [ 137 ms ] threadOn the other hand, leaving positive reviews out of politeness does a disservice to others. I hosted a couchsurfer once (a woman) who told me about her experience with another host. This guy had 10-20 positive reviews. And he was a total creep. Hit on her the entire time, asked her to sleep with him, and was intimidating. She left as soon as she could.
But she didn't leave a negative review! And clearly, other women must have had her experience, yet they said nothing either. This guy was left free to continue his horrible behavior.
I don't know the solution.
Trusted reviews. A user should be able to trust others, and to filter reviews based on that. Integration with social networking would allow for friends of friends trust as well. It would be far from perfect but it would better replicate the experience pre-internet of asking people around you, which works pretty well.
Another way to phrase the heuristic might be: be lenient toward incompetence (including the bad turn in OP), but strict toward malice or unethical behavior.
Yet another phrasing: if the guy was CEO of a large company, say mozilla, should his behavior get him fired? If so, give him a one-star review. I find such thought experiments changing relative status to be quite useful.
It depends on the review system. The article was addressed at five stars systems, where one star reviews do tend to have too low a bar for some, whereas others never leave negative reviews of any sort.
Other system are good/bad and there it's a harder choice whether to leave criticism. There's incompetence, and there's something that wasn't good but doesn't necessarily signify systemic incompetence. Leaving a review marked "bad" can't distinguish between those two, at least in the eyes of those that perceive it.
Give accurate reviews, and provide context in a text format. If a system will fail because it has to meet an arbitrary quota (eg Uber's 4.6 stars), then that's a problem for the system to work out, not the person providing the review. The solution is simply to give dispassionate, honest reviews.
While some might feel a sense of duty and obligation to the rest of the customer base to leave honest reviews... I think most selfless and considerate people will weigh that against the impact it has on the person they are reviewing and wonder if its really worth being an asshole.
There will be growing pains, people will suffer greatly, but a wonderful new system will hopefully arise from the ashes.
I do think that there is a better way, and that most users know how to discount a really bad review. It's kinda hard to hide bias when you are angry. So, the system will eventually balance out.
Bottom line, I don't know how to fix this system, but I do know how to survive in it. (hint: underpromise, overdeliver)
One thing that can be said is a customer who becomes sick with power is hard to deal with. If someone gets in the car and makes unreasonable demands with the threat of a bad review what do you do?
Shouldn't the reverse be true as well? A 5-star review from someone who gives lots of a 1- and 2-star reviews should mean a lot more than a 5-star review from someone with little history or a history of giving many 5-star reviews.
Really, a big problem here is cultural norms vary and subjective ordinal ratings don't mean the same thing to different raters -- ISTR studies showing that White Americans tend to rate things near the top of a scale if they are acceptable, toward the middle but still above the center even when they are unacceptable, and below the center only rarely -- whereas different nationalities and ethnicities had measurably different patterns of ratings compared to their actual descriptions of their responses to products. If that's true, a one-sided selective discounting of reviewers that give lots of low-star reviews may be effectively saying "I want reviews from White Americans to count the most."
Weighting reviews based on the variance in the history of that reviewer seems like a great idea to me, because it gives preference to reviews that probably carry more information in them. A review of five stars from someone who always hands out five star reviews, is meaningless and doesn't inform anyone else of anything. Likewise for one star reviews. In fact, intuitively I think maybe even a five star review from someone who usually hands out one star reviews, or vice versa, isn't very informative either? Reason being that these people clearly don't put a lot of thought into their reviews in the first place.
Someone who drives so badly that they endanger the lives of their passengers in a paid service? Whether or not he's a nice person, you should rate the action appropriately. If you're the only bad review, the driver's rating should stay high.
Did the driver learn from the bad mistake? How would you know? Someone can be nice and still clueless. Just rate the driver appropriately.
Personally, I find the bad reviews to be much more informative than the good reviews. Star ratings are largely meaningless (unless the thing is at 1 star or less). Read the bad reviews - if they're just frothing trolls of the kind that complain about everything, the item is probably fine (and you can check out more reviews at that point). If the bad reviews are raising valid complaints - and 'drove so badly we nearly died to his manoeuvres' is a valid complaint. If several people are making that complaint, then there's likely to be substance to the claim.
1) Evaluation inflation/simplification. Happens with grades, happens with reviews. 4 stars becomes the new 'adequate'. If you're under 4 stars, you're bad. Everyone subconsciously knows this, hence the thinking 'if I don't give them a 5 star review they might get fired.'
A more meaningful, diverse group of metrics is likely the answer to this. Get rid of 'overall' ratings. Rate drivers on their friendliness, safety, timeliness, etc. Uber can decide on some algorithm for determining when to let drivers go, and users feel less bad about negative reviews: 'Well, I gave him a 2 on safety but he was really friendly so at least I gave him a 5 on that."
2) Not enough separation between reviews and individuals. If I leave a review on amazon, I don't want that being tied to my account. At all. I don't want to see the seller's replies to my reviews. If someone finds out my amazon account name, they should not be able to tie individual reviews back to me. Batch reviews up in groups of 5, randomize them, and post them all at once. Don't let there be any possibility of tying reviews back to individuals. The fear of someone seeing a mean review I wrote and forever thinking I'm an asshole is a serious barrier to negative feedback (barring extraordinary circumstances).
This is of course based on my own personal experience but at least on tripadvisor.com I have seen this happen a lot. Every time there is a 1-star review of a hotel you can can bet that there will be 3-4, 5 star reviews on top of it by reviewers who have left 1 review in total (suggesting they are fake or outsourced) just so that the 1-star review can be bumped to page 2 or drowned.
I mean, you could review one star for things that are, like, actually poisonous or dangerous, but that doesn't really work either. As beloch suggests below, a weighting system is a good answer, and Amazon does a pretty good job (I think) of letting you look through positive and critical reviews. Kinda different when you add the human element, but still.
After averaging reviews across reviewers, you can see a positive correlation between the two metrics: http://i.imgur.com/QXIU3qb.png
Sorry, but not every book should be read. Not every restaurant should stay in business. Not every Uber driver should be driving. Giving accurate reviews allows the system to work, both for customers and (eventually) for providers (by steering them out of work they're inept at).
Or maybe I'm just an asshole...
The fact is the average Uber ride is going to be average. The average person shipping stuff on Ebay is average, the average person selling on the AMZN marketplace is average, and so forth.
If you're supposed to leave 5 stars to say service was adequate, then there is no way to reward those who go above and beyond.
"Always tip 20%. It's how they make their money!"
vs
"I'm not tipping anything for bad service."
Even though I consume online resources, I rarely leave ratings, since I can't really tell that it's worth the effort.
The aggregate effect is the same (if a random number from 1-5 is added to each reviewer's review, on average, you just need to subtract 3 from the average review score), but reviewers may feel more-free to be honest. More reviews are required to get an accurate review, but the reviews may be more accurate.
The larger the random spread is compared to the rating's dynamic range, the greater the obfuscation, and the more reviews you need to learn something new.
Likewise for 5 stars. Friendly, safe, on time, gives you a cold bottle of water, have a nice conversation. 4 or 5 stars. Chases you down with the bag you forgot in his car? 5 stars. You fall in love and get married? 5 stars.
The problem I see is that the perception of what warrants a given rating varies pretty widely, and most sites do a terrible job of trying to help their users calibrate their ratings. I monitor Yelp regularly for my girlfriend, who owns a wine bar. Her ratings are generally excellent - over 4 star avg., with most ratings at 5 for the past two years. However, every once in a while we see an anomalous rating - for instance:
- one current 3-star says "I have to be honest, I didn't have anything to drink, I was with friends waiting for the midnight showing at the theatre next door. However, it was packed and happening and EVERYBODY was having a good time."
So this person, who wasn't a patron, but who had nothing but positive impressions, still left what ultimately will be considered a sub-standard rating. This is actually most common for us with 3-stars - they are generally very positive reviews, with nothing negative at all, but with what most Yelpers would consider a sub-standard rating.
Mostly when I look at Yelp ratings, it follows the same pattern - 80% of the ratings seem to correlate closely with the reviews and the typical Yelp ratings standards. 10% seem to be using a different rating standard. And 10% are just weird.
Firing someone for a single one-star review is asinine.
But I've gathered that I'm the exception...
There's a huge difference between:
1 star: I waited for an hour for my pickup and this caused me to miss my very important appointment. Yes, traffic was horrible today, and so maybe it isn't the driver's fault but this was a very bad experience I don't want to repeat. You folks need to do a better job of setting expectations!
and
1 star: my driver narrowly missed rear-ending the car in front of us because he was distracted. He didn't even apologize!
In the second case we might blame the driver. In the first case, who gets the blame depends on the system.
Five star reviews tend to be fluff, one star reviews tend to be specific. I read them first, I look for dealbreakers. No amount of "I love it" can outvote "claims X but can't really X properly".