Terrible article. Calling histograms awful, based on nothing more than an opinion.
Then trying to conclude that some convoluted scatter plot system makes more sense is laughable.
Not to mention, this system is still just a star rating system. This would be no different than having two histograms side by side.. assuming, of course, that you'd even want to rate different aspects of the same thing.
I can't even imagine scatter plots on amazon, or trying to convince the general public that "it makes more sense"
My point was you can't create a scatter plot from a single dimension of data, and that histograms are awful because they don't allow you to see patterns. But of course you're right, it is just my opinion.
But histograms do allow you to easily see one of the most important patterns to me, at least for technical books, which is that the book is excellent but too "hard" for some who aren't willing to put in the effort. Those books tend to have a lot of 5 star and 1 star ratings, with few in between. Check out the SICP reviews on Amazon for example. When augmented with a set of comments, it's usually easy to predict which of the two groups I'd be in.
This would be no different than having two histograms side by side..
Yes it would, and the article shows why and how. Scatter plots are easy to read (for comp./math. educated people). Two histograms side by side are easy to read (and find correlations) for nobody.
Though a scatterplot conveys less information (e.g. the correlations between the two axes), I think it takes longer to process. It also takes more screen real estate than a pair of histograms.
A scatter plot definitely shows more information than a pair of histograms. Who knows if that additional information will be useful.
It would be nice if the article actually compared different visualizations of the same data, rather than showing histograms of 2 separate data sets and scatter plots of a 3rd data set.
The point is that scatter plots are harder to read than histograms. It's not two histograms vs. one scatter plot, it's replacing the current single histogram system with a single scatter plot.
Also, side-by-side histograms aren't the only way to display two parameters in a single histogram. What about stacked histograms? They scale up to an arbitrary number of parameters and everybody who can read a histogram can read a stacked histogram. Scatter plots seem like thermonuclear overkill for a problem which most movie sites seem to consider to be "solved".
I think it's a problem of brevity. You write something, you refine, you cut out the pet witty comment. You cut your message to the bone. Now you're done.
Read the text of the article describing each movie. The correlations are absolutely meaningless. He doesn't hit on them at all. In fact, with comments like "almost nobody" he's specifically looking at averages. Then coming to a conclusion effectively based on the average of Score 1 and the average of Score 2 to determine what type of movie it is.
So on this Quality/Rewatchability grading system:
Starship Troopers: 3:4
The Fifth Element: 4:4
Blade Runner: 4.5:3.8
Drop that in place of the graphs and the conclusions would sound as seemingly valid.
It's true it's harder to find correlations, but this particular correlation is not likely to be meaningful.
As far as reading the distribution, two separate histograms are definitely more readable and understandable in order to understand the distribution. Adding the complexity of a scatter plot because it also shows a correlation that people are not actually interested in makes things less understandable.
There's a pretty clear correlation between the histograms and the scatterplots, too, if you look.
If you gave someone those 3 histograms and those 3 scatterplots, I bet they could match them up correctly.
Aside- The scatterplots are an awful user interface because of the cognitive effort to interpret them, but perhaps there's a way to present the same information in a usable way.
It's different from having 2 histograms because it lets you distinguish people who thought it was a good movie but not re-watchable, from people who thought it was a good movie and fairly re-watchable, from people who thought it was an OK movie and not re-watchable. With a bunch of histograms you can't correlate the variables.
Why would I care about rewatchability? The torrents are full with thousands of lives on video I'll never watch, what's the point of rewatching anything but the dearest, youth-defining films?
The comment I was replying to said This would be no different than having two histograms side by side. And I pointed out that it is not.
Oh well I'll answer your question anyway. Because 90% of everything is crap (probably more than that on P2P sites). Re-watching a movie can be like walking in the same park more than once, looking at pleasant things with a sense of recognition. You could put a movie in to suit your mood. Sometimes you'd rather watch a good "original" movie again than watch yet another crappy remake or rip-off.
rewatchability should indicate a light-weight movie.
If you want that, you look at rewatchability, if you want something more complex, you use quality.
I think this is the underlying assumption.
Because the rewatchability rating doesn't just tell you how rewatchable a movie is, but also how good and "timeless" it is, i.e mainly how deeply it resonates with people.
This is not about picking a movie to watch again after you have already seen it once: this is about picking a movie you haven't seen but that is so good that other people like to watch it again and again.
Speaking as a film buff; this is actually quite a good guide to the sort of movie it is (when combined with quality). If lots of people mark it good quality, but wouldn't watch it again, that implies that you have to be in the right mood for it.
If people mark rewatchability high, even if the quality rating varies, you know it is much more easy going film.
You are not trying to gauge if you would want to re-watch the film. If other would re-watch the film, maybe, just maybe, you would enjoy watching it the first time.
Because "rewatchability" is not just useful to judge how much people like to watch a movie again -- it's also useful to judge how much people like the movie, find it deep, and connect with it (over pure mindless fun).
So rewatchability roughly translates to "emotional connection + quality".
Now compare your point of view with the one expressed by mhellmic[1] in this very discussion. 'Rewatchability' sounded to me as something even more vague than 'quality' while reading the article. Also, my pattern for rewatching films differs greatly from both your description for rewatchability and mhellmic's, and from my own pattern for rereading books. I don't see how it can be applied to rating systems with just one additional parameter (the quality thing).
You're telling me its stupid to choose the 100% chance of some known amount of fun rewatching an old series than to bet compared to watching that latest Jennifer Aniston flick?
Terrible comment. I found the article interesting. As someone who has played around with many different types of rating systems, I applaud their effort at trying something different. Sounds like you area little too emotionally invested in histograms. I'm not even going to ask why.
I'll meet you halfway. It's a terrible comment about a terrible article. I agree that it's laudable to try to invent an improved version of online ratings, but this article wasn't effective in convicing me that they've succeeded.
Their argument that historgrams are just.awful. (I didn't care for the extra periods) seems to have two components:
1. asserting that historgrams are bad
2. showing us 3 histograms and saying they tell you the same thing about all 3 movies, when in fact there is a very clear and important difference between the histograms.
Its completely obvious from inspection ("people are really good at seeing patterns") that Starship Troopers has a much lower percentage of 5 star ratings than the other two, and a much higher fraction of 0 star ratings. It also appears to me that the Fifth Element has a higher fraction of 4 or 5 star ratings, and is probably the most apprciated of the 3 films, although Blade Runner is fairly close.
If you are going to cherry a set of 3 specific films to make your point, you should be sure to at least pick 3 films that support your point instead of refuting it.
We then learn about their hypothesis that while 5-star rating system sucks, a system that relies on two correlated 5-star ratings is great. They demonstrate this by using the two question system to draw the exact same conclusion I drew from the histograms of the 1 question ratings.
I would've liked some sort of objective attempt to compare the two rating systems. Perhaps it would be possible to measure how frequently the two question system leads people to make a better choice than the one question system, or at least some sort of statistical wonkery that would purport to show me that the two question system in practice draws more distinctions than the one question system. Unfortunatley we only get this one rather uninspired example ("watch this if you’re in the mood for something really good").
They also didn't address why "would you re-watch this film" is a better choice than any other second question. There are attempts to justify it being a good question, but no real evidence that other questions were tried and didn't perform.
Finally, the thing that really irked me was that this proposed system doesn't seem to do anything to address most of the actual problems with the regular 5 star system, namely that people who feel really strongly about something are more likely to rate so that most ratings tend towards the extremes, and that without context we have no idea why someone rated something a 5 instead of a 1. Those problems now exist along two dimensions instead of one.
I see this less as an article and more as an advertisement hitching a ride on an xkcd comic.
I agree, the good idea was to add more than one dimension to the rating. I suspect that it would be even more interesting if we had a couple more ratings to separate that one number into more precise groups.
It seems to me like "rewatchability" isn't really a useful metric when I am looking for a new-to-me movie to watch.
And when I am looking at user rankings for a movie, almost by definition I am only concerned with rankings for movies I haven't yet seen, since I already have an internal self-ranking for a movie I've seen already.
Histograms are extremely useful for knowing the spread of rankings, which the scatter plot also illuminates.
It's not a terrible article at all. It's a suboptimal solution but the article is pointing out what's wrong with 5-star rating systems. It's labelled as a "response", not as a solution to all our earthly rating problems.
From a statistician's standpoint, ratings systems suck. But, from a consumer standpoint, they are super easy to understand. A scatter plot system makes sense to me, but I would never put it in front of a user.
In my opinion, current ratings systems are 80% UX and 20% data.
For example, Newegg uses a pretty intuitive system of allowing you to sort a product page by Best Reviews and Most Reviews. In my opinion, this allows the user to make a more educated decision if they seek the information out.
The simple answer there relates back to UX, just don't show the stars when there isn't enough data. Set a minimum number of reviews as a baseline so that you don't get the result you mentioned.
If there is a written review component, make a note of the review but don't quantify the value of said review until the minimum threshold is reached.
We actually use this whenever we do any ranking within the site. On the film page though, we think presenting the raw data as a scatter plot is better than a single number.
I think the two-axis argument is stronger than the anti-histogram argument. The problem isn't that the one-axis rating systems are aggregating their ratings into histogram bars rather than displaying them in some other way, but that the ratings curve just doesn't carry a lot of information.
The videos of the lectures but the book authors (Gerald Sussman and Hal Abelson) are also available for free.
Pick the MPEG1 versions. They are much heavier than the MPEG4 versions, but the text on the projected computer screen is at readable. IIRC, the MPEG4 are re-encoded versions of the MPEG1, which themselves were ripped from VHS.
If I'm deciding whether to watch a film for the first time, I really don't understand how whether or not other people would want to watch it a second time helps me make my decision.
"We rate movies on two criteria - ‘quality’ and ‘rewatchability’, so you can admit to your guilty pleasures and properly capture the feeling you get when a film leaves you exhausted."
You are using rewatchability to infer some potentially helpful labels ("guilty fun", "exhausting but worthy"). But there's no guarantee that those inferences are safe/generalisable across viewers/films/genres, or that people who see a rewatchable axis will know to interpret it like you do...
The article mentions an important point about 5 star systems - people tend to only use 5 stars or 1 star. This is sort of shown with the histograms.
A 2 axis system seems like a good idea. But I'd like to see it with 3 options per access - [UP] [INDIFFERENT] [DOWN].
I'm also interested to know how the system will cope with "controversial" films ( life of brian, for example) where some people are going to downvote whether they've seen it or not. And they'll campaign and ask all their friends to downvote too.
When I'm looking for people's opinion I want to know at least a little about the people and have more things in common with them, so that we have similar tastes. IMDB is almost useless for me, people like complete crap imo (is there any movie without at least one 10 star rating? That should be the single best movie ever). If you wouldn't fit in the community, the community's opinion on things is largely irrelevant and whether you look at the opinion through a histogram or a scatter plot is irrelevant.
If you actually have friends, why don't you ask them for recommendation in person. If your friend is really into arty movies and recommends you an arty movie as being very arty (and well done) you can consider it. Collapsing it into a single number doesn't make sense </rant>
EDIT: that's not to say that the scatter plot isn't an interesting idea, it's just not going to help much because people's background is important for rating
You've absolutely hit the nail on the head. Knowing the rating of a friend is worth so much more than a bunch of strangers. That's what Goodfilms is all about - it puts your friends' opinions ahead.
Why would someone who is looking for a film care about rewatchability? Presumably they'll be watching it for the first time and then can decide for themselves whether to ever watch it again.
Or did I miss the point and rewatchability is just a placeholder for something more useful?
there’s a lot of disagreement over whether it’s high quality of not, but generally this scores high-rewatchability. So, maybe not the most intelligent movie, but good fun.
What I intuitively deduced from this example is that rewatchability is metric of enjoyability.
On a single 5 stars rating, some people will give 5 stars because they really really enjoyed the movie, some others will give 5 stars because they thought the film was perfect on a cinematrographic-quality (i.e. scenario, cinematography, acting, casting, etc. insert here some academy-award-technical-category) point of view.
Actually, I find histograms pretty useful, primarily because if there is a secondary bump toward 1, it indicates there are a significant number of people who had bad experiences with the product -- more investigation required.
Having a two-dimensional graph might have more information, if the dimensions really matter. I'm doubtful that "stars" and "rewatchable" are really independent, and I'm unsure why I would care about it when I haven't seen the film. (If I have seen the film, I'll have my own opinion and not need the graph.)
I'm all for looking for improvements to the ratings game, though. What seems to work best for me is to actually read the reviews, but that's obviously time-intensive.
Often I'm presented with a choice of films at the cinema (some of which I've seen already) and only want to decide which one to see next. Perhaps an ordering by score would be more appropriate?
I'll say histograms are useful for certain things.
If you are looking at electronic devices or camera lenses there's the issue that a certain fraction of people get lemons. Some bad reviews are because of that.
Other people have unrealistic expectations of the product and give a bad review.
A histogram gives some immediate insight into this problem, and then looking at stratified samples of the reviews helps there on out.
Now, I will say the star ratings on Ebay are weak because of the fact that a less-than-perfect ranking gets people in trouble. Although "acceptable" performance on Ebay goes a considerable range (It's certainly a worse experience to have a long confused exchange with somebody with poor english -- this person shouldn't be punished, but they shouldn't be rewarded either.)
Edited to add: "If you're going to get any useful info, you have to read the reviews," is so incredibly true. I'm surprised it's not getting more mentions in these comments. The scatter plot is kind of cool, but I'd so much rather have a histogram and actual reviews to check so I can find out why the product got those ratings.
The problem with ratings are that they are not from a large enough random sample. Rating scores tell me what people that like to rate products think. I don't rate products and in general I suspect that people that are happy with a product don't really care to take the time to go out of their way to tell people about it online (that may change or be changing, I don't know). but one thing is for sure, people that dislike a product WILL go out of their way to tell everyone, thus further shifting the data set to being reviewed by people that are unhappy.
Take a random sample of the true population of the data set (everyone that has seen a movie) and not just the people that logon to rate it.
If you're interested in a more rigorous analysis of the problem, I highly recommend reading the paper "How to Count Thumb-Ups and Thumb-Downs: User-Rating based Ranking of Items from an Axiomatic Perspective":
Scatter plots are definitely more informative, once one gives them a couple of minutes to get used to them. However, I think you're shooting at the wrong target and your solution would exacerbate the root problem: bias.
The first time I really noticed the problem was when I published my own Flash game on Kongregate and started paying closer attention to the ratings. That led me to examine my own rating habits and I conjectured that is probably what happens to everyone else.
The bias I'm talking about is caused by the fact that most people can't be bothered to rate something. Most people only rate something when there's a powerful impulse to do so, so most of the votes will be 5 stars or 1 star. The 4-star ratings come from people who liked something enough to be moved to rate it, but not enough to gush about it; note that the group of people who makes that distinction is already substantially smaller than the 5- and 1-star reviewers. The rest comes from a very small minority, most of whom are people who didn't have anything better to do at that moment and decided to spend some time rating, but don't do it on the regular basis.
By the way, I realize that this is just a conjecture, but from what I've seen so far, it seems to be pretty accurate.
I think that introducing an additional axis will only exacerbate this, by raising the bar for rating. If the act of rating starts demanding more effort, you'll get a distribution that is even more skewed than now.
The two improvements I would like to see are:
1. a system that infers ratings from users' actions
2. better mechanisms for gauging the relevance of someone's review/rating based on my preferences/tastes
The first would help reduce the bias and the second would help me extract more useful information from the biased dataset.
Obviously, it doesn't explain everything from your example, but Kongregate rewards users for voting for games (1 point per rating in their levelling scheme). This will have some impact on how people vote.
I've noticed the same problem, where people tend to rate a 5 or a 1.
As a benefit, I've noticed I can usually find the best reviews on Amazon by looking for 3 star ratings, and to a lesser extend 4 and 2 stars. People who rate a 3 have looked at pros and cons of the product, and generally compare to similar goods. 3 star reviews usually provide FAR more information than glowing or glowering 4 and 1's.
I definitely go by the 4.5 stars == very good, <4 stars == crap heuristic, but to argue this is no good is ridiculous. It's actually very, very helpful.
E.g. when I go to Amazon I don't buy some random product with a 4.5 star review -- I search for a specific product or a specific kind of product and then reject candidates which are lousy. How is that not INCREDIBLY useful? Similarly, who goes to a movie simply based on whether it's good or not.
In general, if you create any point rating system people who like a thing will tend to rate it towards the top of the scale, e.g. 4/5 or 9/10.
I actually did an informal experiment -- I used to run role-playing tournaments, and do exit surveys on participants. For the first few years we asked players to rate us on a 5-point scale and scored slightly over 4/5 on average. Then we switched to a 10-point scale and scored slightly over 9/10. Not scientific -- but I don't think we suddenly got better.
This finding is backed up by serious research (which is why when a psychologist creates a scale, the numerical ranges need to stay constant in follow-up studies or the results are not statistically comparable).
Netflix, which tries to give users customized ratings, actually subtracts value (in my opinion) from its scores because it tries to make ratings mean "how much will you enjoy this?" BZZZT. I pick stuff for me, my wife, my au pair, and my kids. We don't all like the same stuff, and we don't want to track ratings individually. My kids want good kid stuff. I want good me stuff. Don't try to guess what I like based on our collective tastes.
The problem that XKCD gets at is simply translating/scaling the results; the article is solving a different problem.
Early on in the Netflix Challenge, I was able to get myself (very briefly) a leaderboard score with nothing more than analyzing every user's ratings; re-centering them by their mean, and re-scaling them according to their standard deviation. The by remembering their translations and scales, I could put a globally-predicted score back into their own language.
So just some very basic statistics is sufficient to erase much of the bias toward higher numbers, as well as halo effects and the like.
(I was pretty surprised that Netflix's own algorithm apparently wasn't doing anything this simple)
I was pretty surprised that Netflix's own algorithm apparently wasn't doing anything this simple
Netflix does have really interesting blind spots. They claim to take ratings seriously, to the point of offering a million dollars for the best rating algorithm. Then, as the GP says, they implement the rating algorithm in a way that renders it completely worthless to any household with more than one viewer.
Netflix does offer us a good demonstration of the failings of absolute technocracy, but it leaves the question of how best to rate movies wide open.
It's a nice idea, and the readership of HN would probably prefer the recommended way in this article, but I'm not sure that mass market consumers will find it that valuable.
might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well informed opinions. But how do you ever know that when you see an online rating of something that you haven't personally experienced?
Amazon has had star ratings for a long time. I largely ignore them. I read the reviews. For mathematics books (the thing I shop for the most on Amazon), I look for people writing reviews who have read other good mathematics books and who compare the book I don't know to books I do know. If an undergraduate student whines, "This book is really hard, and does a poor job of explaining the subject" while a mathematics professor says, "This book is more rigorous than most other treatments of the subject," I am likely to conclude that the book is a good book, ESPECIALLY if I can find comments about it being a good treatment of the subject on websites that review several titles at once, as for example websites that advise self-learners on how to study mathematics.
The problem with any commercial website with ratings (Amazon, Yelp, etc., etc.) is that there is HUGE incentive to game the ratings. Authors post bad ratings for books by other authors. The mother and sister and cousins of a restaurant owner post great ratings for their relative's restaurant, and lousy ratings for competing restaurants. I usually have no idea what bias enters into an online rating. So I try to look for the written descriptions of the good or service being sold, and I try to look for signals that the rater isn't just making things up and really knows what the competing offerings are like. When I am shopping for something, I ask my friends (via Facebook, often enough) for their personal recommendations of whatever I am shopping for. Online ratings are hopelessly broken, because of lack of authentication of the basis of knowledge of the raters, so minor details of dimensions of rating or of data display are of little consequence for improving online ratings.
Many reviewers are biased and judgemental. I prefer to look at the distribution of ratings that amazon shows. Good items have a distribution with one peak, even if it isn't at 5 stars. When ratings create a saddle the product is probably a fluke.
The cute XKCD comic aside, the distribution is also useful because, for certain types of things, a lot of fairly to very negative comments are illuminating even if the average rating is still pretty high. It's not just about polarizing material. If you look, for example, at genre fiction you'll get a lot of fans who give 5s no matter what sort of crap the current book is. But if there are also a notable number of 1s and 2s, that's often a good red flag.
In the context of a website like Netflix, where your recommendations and ratings for movies are based off of your history of ratings, aren't you the one rating the raters?
However, the type of rating mentioned in the OP and the type of rating on Netflix only seem to work in specific niches. I can't imagine how a website like Amazon would implement anything even close to what Goodfilms is doing.
IMO the perfect solution would be to rate only by saying "I like" or "I dislike". And then writing a review (to show how much you liked/disliked the product/movie etc...)
When you first get on the website, you're asked to rate a number of products. The more you rate, the more accurate the solution becomes.
When it can couple your test with users having the same test, you know see these users ratings in priority for new products. You can even ask them why they disliked/liked something if they didn't write a review. Because their opinion matters to you now.
> The problem with any commercial website with ratings (Amazon, Yelp, etc., etc.) is that there is HUGE incentive to game the ratings.
While I agree that this is a problem, I think a bigger problem is a simple matter of scale:
Amazon is huge, and many people buy things, but they don't split reviews and ratings by what kind of person is rating them. If they wanted to make an improvement, why not show me only ratings and reviews by people who are similar to me? They have tons of data about me and other people who use the service, so it should be possible for them to say "people like you rated this on average a 4, but everyone in the world rates it an average of 2.5."
That's much easier than having to read all the reviews and decide if the person is in my demographic or whether I agree with their review.
I'd just like to point out that bias feeds into written reviews as well. I may think good service is someone not refilling my water every 5 seconds where as you might think it's that your food didn't get out in under 10 minutes.
While words do express the point better, this rating system is a step in the right direction.
Relative ratings are more useful. Everyone uses their own scale, but their ratings are relative to the constant movie. I want to see how people rated a movie relative to other movies I've watched.
N people rated this better than X movie, but less than Y movie.
Ranking movies can be easy. Show 5 movie posters instead of 5 stars or have an auto-complete field for this movie is up there with:.
Anyone interested in relative ratings should look at Dan Areily's[1] Predictably Irrational. He is an economist who writes about behavioral economics and decision making.
I've often thought about some start-up ideas around relative ratings, and this book was the reason
I think Amazon's "Was this rating helpful (Yes/No)?" provides a good filter for ratings. A lot of mindlessly negative reviews get filtered out by the users who come along afterwards and rate the rating in their own self-interest.
That can be easily gamed as well. If you want to boost a rating for a book, mark all of the negative comments as not helpful... In fact, I see that happen on Amazon and Newegg a lot.
Netflix does. By cross referencing your likes and dislikes against those of your fellow Netflix members, the company is able to create a meta rating system, in which the score you see for a movie is your own. You see that score because that's how much Netflix thinks you'll like it, based on how similar people liked it.
This is the only good way of going about this method. The trick is, it's easy to do this with movies, but much more difficult with product ratings and the like. Maybe this is an opportunity for someone to build something on top of Facebook or Amazon.
Is Netflix really the only site which does this nearly-braindead machine learning approach?
Once you realize that people have different tastes and you know someone's preferences that is the obvious solution. Or is the process of crawling through that much statistical data that expensive that it can only be offered to paying subscribers?
rateyourmusic.com sort of has this, but it's not in the default view. You have to go to an album and click a "View my suggested rating" button, then it whirs for a few seconds before giving you an average of ratings from users with similar taste. It would be much more useful to browse the whole site with those ratings showing, but I get the feeling it's a computationally expensive feature.
The more accurate you want to get, the more computationally expensive. Netflix actually did a contest with a million dollar prize to the team that could come up with the most accurate rating prediction algorithm. In the end, the million dollar algorithm was too expensive to implement, so they never ended up using it.
Pandora does something similar. They only have "like" and "dislike" rating. If you like a song they look at other users who liked that song and try to find more songs/bands from the people with your taste. And the other way around for dislike i guess.
It works exceptionally well. You just listen to the stream of incoming songs, you never pick songs yourself. After a good song you click like, after a mediocre song you keep listening, during a bad song you click skip. After a few days you will only get good songs! (with a few exceptions of course) It's like magic, i can't even count how many new bands i found through pandora without any effort.
Regarding the unhelpfulness of online reviews, my company has problems with manufacturers/sellers writing 5-star reviews of their own product listings (ASIN's) on Amazon. We've begun (manually) data mining 5-star reviews to identify whether each 5-star-reviewer has any other reviews (or wish list, to indicate the possibility of a real user account), then calculating the % of reviews written by no-history user accounts. Of the ASIN's we've assessed, the gut-level-doesn't-seem-like-heavy-review-fraud listings can be in the 6% range, whereas the looks-like-review-fraud ASIN's are above 20%. We're working with Amazon to identify and penalize these manufacturers/sellers, but internally at Amazon the Seller Performance team is separate from their Community (user review) team, so it presents a challenge. Also hard for them to separate valid complaints from sour grapes complaints.
"but internally at Amazon the Seller Performance team is separate from their Community (user review)"
Indeed, my suspicion is that organizational politics have more to do with the lack of a better rating system than any technical limitation.
The approach I use is to read 3 star ratings first before biasing myself with the more extreme ratings. I also check to see what else the reviewer has rated and if there's nothing there then I immediately dismiss the review.
Hmm. "rewatchability" to me is loaded with multiple meanings. That's a problem.
I take rewatchability as possibly meaning complex or rich, in that there is something to be gained by watching it again. For example, Prometheus is rewatchable for me. I can also see it as meaning "a vacuous thrill ride I want to try again". Transformers is rewatchable, but only because it appeals to my childhood daydreams of bads, giant robots. It is certainly not deep.
Second, are there movies in the database high in rewatchability but low in quality? If not, then the variable are essentially dependent and the scatter plot useless.
Hmmm. Whatever the scientific or theoretical improvement such an approach may offer having to educate users on how your ratings system works is going to add a huge amount of friction to user engagement.
And frankly who has ever mentally rated a film in terms of "re-watchableness".? People just think in terms of of "good" or "bad" and current ratings systems a la Amazon leverage that. It's simple, fast and given the histogram presentation tells me everything I need to know about the number and distribution of votes in a flash. Plus whether I want to rewatch a film or re-read a book is largely down to my mood at the time. But my opinion on whether it's "good" or "bad" is pretty static.
Maybe Amazon's system is not statistically bullet-proof, but who cares? We're talking movies here: a cheap, casual and discretionary purcahse.
164 comments
[ 3.2 ms ] story [ 214 ms ] threadThen trying to conclude that some convoluted scatter plot system makes more sense is laughable.
Not to mention, this system is still just a star rating system. This would be no different than having two histograms side by side.. assuming, of course, that you'd even want to rate different aspects of the same thing.
I can't even imagine scatter plots on amazon, or trying to convince the general public that "it makes more sense"
It looks like they would still give a good picture of the data, as the trends are mostly linear.
Yes it would, and the article shows why and how. Scatter plots are easy to read (for comp./math. educated people). Two histograms side by side are easy to read (and find correlations) for nobody.
A scatterplot conveys objectively more information.
It would be nice if the article actually compared different visualizations of the same data, rather than showing histograms of 2 separate data sets and scatter plots of a 3rd data set.
Actually it is fundamentally different as the histograms show aggregate data and scatterplots show individual data points.
Also, side-by-side histograms aren't the only way to display two parameters in a single histogram. What about stacked histograms? They scale up to an arbitrary number of parameters and everybody who can read a histogram can read a stacked histogram. Scatter plots seem like thermonuclear overkill for a problem which most movie sites seem to consider to be "solved".
Read the text of the article describing each movie. The correlations are absolutely meaningless. He doesn't hit on them at all. In fact, with comments like "almost nobody" he's specifically looking at averages. Then coming to a conclusion effectively based on the average of Score 1 and the average of Score 2 to determine what type of movie it is.
So on this Quality/Rewatchability grading system:
Starship Troopers: 3:4 The Fifth Element: 4:4 Blade Runner: 4.5:3.8
Drop that in place of the graphs and the conclusions would sound as seemingly valid.
As far as reading the distribution, two separate histograms are definitely more readable and understandable in order to understand the distribution. Adding the complexity of a scatter plot because it also shows a correlation that people are not actually interested in makes things less understandable.
If you gave someone those 3 histograms and those 3 scatterplots, I bet they could match them up correctly.
Aside- The scatterplots are an awful user interface because of the cognitive effort to interpret them, but perhaps there's a way to present the same information in a usable way.
Oh well I'll answer your question anyway. Because 90% of everything is crap (probably more than that on P2P sites). Re-watching a movie can be like walking in the same park more than once, looking at pleasant things with a sense of recognition. You could put a movie in to suit your mood. Sometimes you'd rather watch a good "original" movie again than watch yet another crappy remake or rip-off.
This is not about picking a movie to watch again after you have already seen it once: this is about picking a movie you haven't seen but that is so good that other people like to watch it again and again.
Speaking as a film buff; this is actually quite a good guide to the sort of movie it is (when combined with quality). If lots of people mark it good quality, but wouldn't watch it again, that implies that you have to be in the right mood for it.
If people mark rewatchability high, even if the quality rating varies, you know it is much more easy going film.
And so on. Combining data points is good :)
So rewatchability roughly translates to "emotional connection + quality".
[1]http://news.ycombinator.com/item?id=4417184
If you've got kids, you know they'll just watch the same flashy animated movies over and over again, even if their opinion of the content is "meh".
What would probably happen is that family films, especially animated ones, would have skewed results.
Plus, they do not normally use online rating systems...
Their argument that historgrams are just.awful. (I didn't care for the extra periods) seems to have two components: 1. asserting that historgrams are bad 2. showing us 3 histograms and saying they tell you the same thing about all 3 movies, when in fact there is a very clear and important difference between the histograms.
Its completely obvious from inspection ("people are really good at seeing patterns") that Starship Troopers has a much lower percentage of 5 star ratings than the other two, and a much higher fraction of 0 star ratings. It also appears to me that the Fifth Element has a higher fraction of 4 or 5 star ratings, and is probably the most apprciated of the 3 films, although Blade Runner is fairly close.
If you are going to cherry a set of 3 specific films to make your point, you should be sure to at least pick 3 films that support your point instead of refuting it.
We then learn about their hypothesis that while 5-star rating system sucks, a system that relies on two correlated 5-star ratings is great. They demonstrate this by using the two question system to draw the exact same conclusion I drew from the histograms of the 1 question ratings.
I would've liked some sort of objective attempt to compare the two rating systems. Perhaps it would be possible to measure how frequently the two question system leads people to make a better choice than the one question system, or at least some sort of statistical wonkery that would purport to show me that the two question system in practice draws more distinctions than the one question system. Unfortunatley we only get this one rather uninspired example ("watch this if you’re in the mood for something really good").
They also didn't address why "would you re-watch this film" is a better choice than any other second question. There are attempts to justify it being a good question, but no real evidence that other questions were tried and didn't perform.
Finally, the thing that really irked me was that this proposed system doesn't seem to do anything to address most of the actual problems with the regular 5 star system, namely that people who feel really strongly about something are more likely to rate so that most ratings tend towards the extremes, and that without context we have no idea why someone rated something a 5 instead of a 1. Those problems now exist along two dimensions instead of one.
I see this less as an article and more as an advertisement hitching a ride on an xkcd comic.
And when I am looking at user rankings for a movie, almost by definition I am only concerned with rankings for movies I haven't yet seen, since I already have an internal self-ranking for a movie I've seen already.
Histograms are extremely useful for knowing the spread of rankings, which the scatter plot also illuminates.
In my opinion, current ratings systems are 80% UX and 20% data.
For example, Newegg uses a pretty intuitive system of allowing you to sort a product page by Best Reviews and Most Reviews. In my opinion, this allows the user to make a more educated decision if they seek the information out.
If there is a written review component, make a note of the review but don't quantify the value of said review until the minimum threshold is reached.
Probably not very scientific though.
http://www.evanmiller.org/how-not-to-sort-by-average-rating....
The canonical example is the SICP ratings on Amazon: 3.5 average; 177 ratings, 96 five stars, 53 one stars.
http://www.amazon.com/Structure-Interpretation-Computer-Prog...
Pick the MPEG1 versions. They are much heavier than the MPEG4 versions, but the text on the projected computer screen is at readable. IIRC, the MPEG4 are re-encoded versions of the MPEG1, which themselves were ripped from VHS.
http://archive.org/details/mit_ocw_sicp
"We rate movies on two criteria - ‘quality’ and ‘rewatchability’, so you can admit to your guilty pleasures and properly capture the feeling you get when a film leaves you exhausted."
You are using rewatchability to infer some potentially helpful labels ("guilty fun", "exhausting but worthy"). But there's no guarantee that those inferences are safe/generalisable across viewers/films/genres, or that people who see a rewatchable axis will know to interpret it like you do...
A 2 axis system seems like a good idea. But I'd like to see it with 3 options per access - [UP] [INDIFFERENT] [DOWN].
I'm also interested to know how the system will cope with "controversial" films ( life of brian, for example) where some people are going to downvote whether they've seen it or not. And they'll campaign and ask all their friends to downvote too.
In their example, blade runner has sligtly more "meh" votes and starship troopers is mostly "meh"
If you actually have friends, why don't you ask them for recommendation in person. If your friend is really into arty movies and recommends you an arty movie as being very arty (and well done) you can consider it. Collapsing it into a single number doesn't make sense </rant>
EDIT: that's not to say that the scatter plot isn't an interesting idea, it's just not going to help much because people's background is important for rating
Or did I miss the point and rewatchability is just a placeholder for something more useful?
there’s a lot of disagreement over whether it’s high quality of not, but generally this scores high-rewatchability. So, maybe not the most intelligent movie, but good fun.
What I intuitively deduced from this example is that rewatchability is metric of enjoyability.
On a single 5 stars rating, some people will give 5 stars because they really really enjoyed the movie, some others will give 5 stars because they thought the film was perfect on a cinematrographic-quality (i.e. scenario, cinematography, acting, casting, etc. insert here some academy-award-technical-category) point of view.
Having a two-dimensional graph might have more information, if the dimensions really matter. I'm doubtful that "stars" and "rewatchable" are really independent, and I'm unsure why I would care about it when I haven't seen the film. (If I have seen the film, I'll have my own opinion and not need the graph.)
I'm all for looking for improvements to the ratings game, though. What seems to work best for me is to actually read the reviews, but that's obviously time-intensive.
If you are looking at electronic devices or camera lenses there's the issue that a certain fraction of people get lemons. Some bad reviews are because of that.
Other people have unrealistic expectations of the product and give a bad review.
A histogram gives some immediate insight into this problem, and then looking at stratified samples of the reviews helps there on out.
Now, I will say the star ratings on Ebay are weak because of the fact that a less-than-perfect ranking gets people in trouble. Although "acceptable" performance on Ebay goes a considerable range (It's certainly a worse experience to have a long confused exchange with somebody with poor english -- this person shouldn't be punished, but they shouldn't be rewarded either.)
Edited to add: "If you're going to get any useful info, you have to read the reviews," is so incredibly true. I'm surprised it's not getting more mentions in these comments. The scatter plot is kind of cool, but I'd so much rather have a histogram and actual reviews to check so I can find out why the product got those ratings.
Take a random sample of the true population of the data set (everyone that has seen a movie) and not just the people that logon to rate it.
http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2...
It's very accessible for an academic paper.
I think there was an article about that a while ago on HN.
The first time I really noticed the problem was when I published my own Flash game on Kongregate and started paying closer attention to the ratings. That led me to examine my own rating habits and I conjectured that is probably what happens to everyone else.
The bias I'm talking about is caused by the fact that most people can't be bothered to rate something. Most people only rate something when there's a powerful impulse to do so, so most of the votes will be 5 stars or 1 star. The 4-star ratings come from people who liked something enough to be moved to rate it, but not enough to gush about it; note that the group of people who makes that distinction is already substantially smaller than the 5- and 1-star reviewers. The rest comes from a very small minority, most of whom are people who didn't have anything better to do at that moment and decided to spend some time rating, but don't do it on the regular basis.
By the way, I realize that this is just a conjecture, but from what I've seen so far, it seems to be pretty accurate.
I think that introducing an additional axis will only exacerbate this, by raising the bar for rating. If the act of rating starts demanding more effort, you'll get a distribution that is even more skewed than now.
The two improvements I would like to see are:
1. a system that infers ratings from users' actions
2. better mechanisms for gauging the relevance of someone's review/rating based on my preferences/tastes
The first would help reduce the bias and the second would help me extract more useful information from the biased dataset.
As a benefit, I've noticed I can usually find the best reviews on Amazon by looking for 3 star ratings, and to a lesser extend 4 and 2 stars. People who rate a 3 have looked at pros and cons of the product, and generally compare to similar goods. 3 star reviews usually provide FAR more information than glowing or glowering 4 and 1's.
E.g. when I go to Amazon I don't buy some random product with a 4.5 star review -- I search for a specific product or a specific kind of product and then reject candidates which are lousy. How is that not INCREDIBLY useful? Similarly, who goes to a movie simply based on whether it's good or not.
In general, if you create any point rating system people who like a thing will tend to rate it towards the top of the scale, e.g. 4/5 or 9/10.
I actually did an informal experiment -- I used to run role-playing tournaments, and do exit surveys on participants. For the first few years we asked players to rate us on a 5-point scale and scored slightly over 4/5 on average. Then we switched to a 10-point scale and scored slightly over 9/10. Not scientific -- but I don't think we suddenly got better.
This finding is backed up by serious research (which is why when a psychologist creates a scale, the numerical ranges need to stay constant in follow-up studies or the results are not statistically comparable).
Netflix, which tries to give users customized ratings, actually subtracts value (in my opinion) from its scores because it tries to make ratings mean "how much will you enjoy this?" BZZZT. I pick stuff for me, my wife, my au pair, and my kids. We don't all like the same stuff, and we don't want to track ratings individually. My kids want good kid stuff. I want good me stuff. Don't try to guess what I like based on our collective tastes.
Early on in the Netflix Challenge, I was able to get myself (very briefly) a leaderboard score with nothing more than analyzing every user's ratings; re-centering them by their mean, and re-scaling them according to their standard deviation. The by remembering their translations and scales, I could put a globally-predicted score back into their own language.
So just some very basic statistics is sufficient to erase much of the bias toward higher numbers, as well as halo effects and the like.
(I was pretty surprised that Netflix's own algorithm apparently wasn't doing anything this simple)
Netflix does have really interesting blind spots. They claim to take ratings seriously, to the point of offering a million dollars for the best rating algorithm. Then, as the GP says, they implement the rating algorithm in a way that renders it completely worthless to any household with more than one viewer.
Netflix does offer us a good demonstration of the failings of absolute technocracy, but it leaves the question of how best to rate movies wide open.
Considering how many Amazon 1-star reviews I find that can be summed up as, "UPS sucked," averages are kinda useless.
http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3...
might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well informed opinions. But how do you ever know that when you see an online rating of something that you haven't personally experienced?
Amazon has had star ratings for a long time. I largely ignore them. I read the reviews. For mathematics books (the thing I shop for the most on Amazon), I look for people writing reviews who have read other good mathematics books and who compare the book I don't know to books I do know. If an undergraduate student whines, "This book is really hard, and does a poor job of explaining the subject" while a mathematics professor says, "This book is more rigorous than most other treatments of the subject," I am likely to conclude that the book is a good book, ESPECIALLY if I can find comments about it being a good treatment of the subject on websites that review several titles at once, as for example websites that advise self-learners on how to study mathematics.
The problem with any commercial website with ratings (Amazon, Yelp, etc., etc.) is that there is HUGE incentive to game the ratings. Authors post bad ratings for books by other authors. The mother and sister and cousins of a restaurant owner post great ratings for their relative's restaurant, and lousy ratings for competing restaurants. I usually have no idea what bias enters into an online rating. So I try to look for the written descriptions of the good or service being sold, and I try to look for signals that the rater isn't just making things up and really knows what the competing offerings are like. When I am shopping for something, I ask my friends (via Facebook, often enough) for their personal recommendations of whatever I am shopping for. Online ratings are hopelessly broken, because of lack of authentication of the basis of knowledge of the raters, so minor details of dimensions of rating or of data display are of little consequence for improving online ratings.
However, the type of rating mentioned in the OP and the type of rating on Netflix only seem to work in specific niches. I can't imagine how a website like Amazon would implement anything even close to what Goodfilms is doing.
When you first get on the website, you're asked to rate a number of products. The more you rate, the more accurate the solution becomes.
When it can couple your test with users having the same test, you know see these users ratings in priority for new products. You can even ask them why they disliked/liked something if they didn't write a review. Because their opinion matters to you now.
While I agree that this is a problem, I think a bigger problem is a simple matter of scale:
Amazon is huge, and many people buy things, but they don't split reviews and ratings by what kind of person is rating them. If they wanted to make an improvement, why not show me only ratings and reviews by people who are similar to me? They have tons of data about me and other people who use the service, so it should be possible for them to say "people like you rated this on average a 4, but everyone in the world rates it an average of 2.5."
That's much easier than having to read all the reviews and decide if the person is in my demographic or whether I agree with their review.
While words do express the point better, this rating system is a step in the right direction.
N people rated this better than X movie, but less than Y movie.
Ranking movies can be easy. Show 5 movie posters instead of 5 stars or have an auto-complete field for this movie is up there with:.
I've often thought about some start-up ideas around relative ratings, and this book was the reason
[1] - http://danariely.com/
I think Amazon's "Was this rating helpful (Yes/No)?" provides a good filter for ratings. A lot of mindlessly negative reviews get filtered out by the users who come along afterwards and rate the rating in their own self-interest.
Netflix does. By cross referencing your likes and dislikes against those of your fellow Netflix members, the company is able to create a meta rating system, in which the score you see for a movie is your own. You see that score because that's how much Netflix thinks you'll like it, based on how similar people liked it.
This is the only good way of going about this method. The trick is, it's easy to do this with movies, but much more difficult with product ratings and the like. Maybe this is an opportunity for someone to build something on top of Facebook or Amazon.
Once you realize that people have different tastes and you know someone's preferences that is the obvious solution. Or is the process of crawling through that much statistical data that expensive that it can only be offered to paying subscribers?
It works exceptionally well. You just listen to the stream of incoming songs, you never pick songs yourself. After a good song you click like, after a mediocre song you keep listening, during a bad song you click skip. After a few days you will only get good songs! (with a few exceptions of course) It's like magic, i can't even count how many new bands i found through pandora without any effort.
Too bad it doesn't work outside US anymore. :(
Indeed, my suspicion is that organizational politics have more to do with the lack of a better rating system than any technical limitation.
The approach I use is to read 3 star ratings first before biasing myself with the more extreme ratings. I also check to see what else the reviewer has rated and if there's nothing there then I immediately dismiss the review.
I take rewatchability as possibly meaning complex or rich, in that there is something to be gained by watching it again. For example, Prometheus is rewatchable for me. I can also see it as meaning "a vacuous thrill ride I want to try again". Transformers is rewatchable, but only because it appeals to my childhood daydreams of bads, giant robots. It is certainly not deep.
Second, are there movies in the database high in rewatchability but low in quality? If not, then the variable are essentially dependent and the scatter plot useless.
And frankly who has ever mentally rated a film in terms of "re-watchableness".? People just think in terms of of "good" or "bad" and current ratings systems a la Amazon leverage that. It's simple, fast and given the histogram presentation tells me everything I need to know about the number and distribution of votes in a flash. Plus whether I want to rewatch a film or re-read a book is largely down to my mood at the time. But my opinion on whether it's "good" or "bad" is pretty static.
Maybe Amazon's system is not statistically bullet-proof, but who cares? We're talking movies here: a cheap, casual and discretionary purcahse.