I do not use the sites listed in the article, but I have found that Epicurious (http://www.epicurious.com/) has plenty of bad recipes that are rated as such. I have had good success following the ratings and checking the comments on the recipes before trying them.
Of course, with any crowd sourced system it is good to take the time to really read through who you are getting your trusted advice from. I still remember reading a review of a Korean restaurant on yelp that had two or three one star reviews because "they did not have sushi".
I just checked and I unfortunately do not have the epicurious rating data. I did just check their website and they have a guide for rating 1-Okay 2-Good 3-Delicious 4-Exceptional.
Their rating system is not broken to the degree of these other websites.
I'd suggest that a bell curve in this case isn't a particularly good goal. Most people who cook already know what collection of flavours they're likely to enjoy and will make more dishes that combine these flavours in interesting ways.
A good example is dishes containing coriander, a large number of people hate it with a passion while others love it. People who hate it won't make the dish and won't review it. Those who love coriander will make it and most likely enjoy it.
Applying statistical analysis with the goal of getting a bell curve isn't the correct direction to reason in.
I guess I just want a recipe rating system that is consistent and has some sort of meaning. I want to know of the people that like coriander and made a dish with coriander in it, what recipes did they find the tastiest (the ones with extra cumin or not). This is not possible when coriander loving people rate the coriander containing recipe 5/5 9 times out of 10.
How about a two-stage version of social bookmarking: in the first stage, there's just "save" and "hide." (You save things you'll make; you hide things you know ahead of time that you won't—because it has coriander, etc.) Then, after a suitable period (e.g. how long it would take to cook all the stuff you saved), you can vote on your saved/cooked dishes (and only those.)
Cilantro really is awful from my perspective so I would be one of those people that hate it, I love Spanish food but I live in New Mexico so I really just love New Mexican food which really is quite different from Mexican food.
It really is hard to find a native New Mexican, other than those raised in Mexican households, who actually prefer or even like cilantro compared to the other ingredients we have available.
I guess I'm just furthering your point but some people may prepare those dishes with no Idea that coriander tastes the way it does and just don't have so much hate to decide to review it (or maybe they just think they lack the cooking skills to make a dish).
I don't think recipes themselves can really be rated; we probably have to rate the person who created the recipe or the person evaluating the recipe. Let me explain:
The number of people that come back and rate a recipe is probably small. And how many people who had a middling experience with the recipe would make the effort to come back and rate it? Or even a good experience, for that matter? A bad one? You bet they're going to rate it.
So right there, you have all kinds of bias problems with the data. And just to make it more fun, how many of those people rating the recipes prepared it properly? How many substituted ingredients that just completely ruined the dish? You see this all the time on these sites - "This recipe is horrible. I vomited for 3 hours straight. BTW I didn't have any red onions so I used pickled cocktail cherries instead."
Somehow you have to find a more indirect way of evaluating a recipe's worth. Maybe it could be a measure of how many times a person comes back to look at it (on the theory that each hit is presumably them coming back to look up that recipe again)? Or perhaps how many people chose to make it.
I think your are on to something. Report how many people saved a recipe (and unsaved it?), how many printed it and how many views it received.
Additionally, I think the mentioned websites ignore an important fact about recipes: they are not static. Most recipes have variations in technique or ingredients. Many cooks, as you stated, substitute ingredients that subtly, or not so subtly, change a recipe. If a mechanism exists to capture those variations (wiki style, maybe?), those people might have an incentive to come back and share their experience rather than their impressions.
This is a good thing. The public usually does not have access to such detailed data, so I'm glad that other companies are copying what OKCupid pioneered.
YouTube noticed a similar issue.
Getting users to rate on a star rating just isn't effective. You may have more luck with a simple 'yes/no' option. Would you make this again? And offer the opportunity for comments.
I find the comments in the reviews on epicurious to be very influential in how I follow the recipe, and if I cook it at all.
I think this applies to almost all online rating systems, in particular ones where it's just pick a number out of five or ten. Most will be either very high (perfect, or perfect with a slight problem), or the lowest (completely shit). Barely anybody will mark something as 2/5 because it was utterly terrible, but this one feature was good. They'll remember the bad parts. No, no data at the moment and it's just a hunch, but one I hope to be able to measure soon.
I'm hoping to apply this to webhosting, rather than simply rate out of ten and leave a 2 sentence review, it'll be much more indepth, with multiple options (support, uptime, features etc). I'm betting that the more effort that you go through to leave a thorough rating will be balanced by the fact you won't have to rate that often (at most maybe once a year once you find a good host).
Oh well, still in the idea stage but I think people should reconsider just a simple rating out of five for anything non-trivial (like rating videos). Anything important (major purchase decision) should be more detailed and have more information for the prospective user.
I think you have another challenge when dealing with something like rating hosting.
The problem, I believe, comes in where you will likely have many people leaving reviews after a bad experience, rather than a good one.
For any negative experience, a person will tell 10 people, for a positive experience, they will share with 2 people.
Videos are a bit different, because people review as they are watching. However with something like hosting, you need to give considerable thought to how you get people to report positive or average experiences equally as often as negative ones.
Yeah, being filled with negatives would be annoying although I'm hoping to come up with something, probably along the lines of discounts/coupons regardless of whether they left a positive or negative one (as long as it's an informative one).
It's definitely something to think about and it's a pretty difficult balance I'll have to achieve but I'm confident I can figure something out.
It's a delicate balance. I personally don't want to fill out a massive questionnaire (even 5 questions may be too many) just to rate something, usually these fixed axises don't always fit what I want to say either. People seem to respond well to a simple rating and a free-form paragraph, both as readers and reviewers -- what is the incentive for the person writing the detailed review to actually write it? All the data needs to be boiled down to a single yes or no in the mind of the reader trying to make a decision anyway.
One of the things I've been contemplating is incentives, though not to the point where people will just fill out a lot of reviews to earn points or anything, more along the lines of random prizes or discount codes for hosting packages. Although then I have to be careful not to trivialize it and turn it into a game since I still want it to be a good resource for prospective customers.
Slight tangent: anyone built a recipe site that works by presenting a common item (chicken, flour, milk, etc) with "Got this? Yep or Nup" and so on to build a list of search terms?
CookThing seems similar I guess, but not quite the same.
Not sure if it wasn't working when I tried it, but I entered 'capsicum' and it showed no recipes. What I envisaged is a site that would randomly pick a common ingredient (rice, potato, coconut milk) and confirm that the visitor had it at hand. Then go from there, rather than starting by having them name something they had.
Recipes are interesting in that most reviewers actually alter the recipe, making star reviews misleading.
So someone might post a bad recipe and have other members alter it and rate it highly after their own modifications. You can only figure this out by reading the comments.
So some kind of yes / no on whether you have modified the recipe is pretty essential.
You can observe this same multi-factor effect in restaurant or product reviews.
When a restaurant serves one customer unsatisfactorily or a product is delivered in poor condition, the rating suffers substantially. This rating no longer represents the quality of the thing itself but also the individual's experience.
The same is true with a cook's experience preparing a recipe. Depending on the reviewer's bias, a recipe rating might describe circumstance and culinary skill better than the quality of the instructions.
As other folks here have mentioned, there's also the issue of average reviewers being not-so-great at representing their true opinions with linear rating systems, but that's a whole other topic.
"So how does the user actually find the best recipe for a particular dish?"
Users may have better luck finding quality recipes by going to curated sites Cooks Illustrated rather than mob-rule sites like RecipeZaar. As with social sites (Digg, Reddit, HN), the chemistry of the community and the influence of the admins can have a powerful effect on the quality of the content. It also depends on what you're looking for. If all you want is cheesy enchiladas, recipes from the popular sites will work just fine.
There are many sites (not to mention books and periodicals) which have thought and care put into their recipes, often with supplementary material which helps you understand what you're doing and become a better cook, rather than just an ingredient-assembling automaton.
Another aspect to consider is that many people like to read (or watch) things about food but don't actually cook much. That's why the Food Network has become so popular (ever notice how all their commercials are for ready-made products?). A pet peeve of mine is to see 5-star reviews with a comment like, "Looks delicious! Will try making it sometime." Pretty annoying, but some people just like to watch. Maybe that's why recipes with pictures get higher ratings.
Another thing you can't do with recipe ratings is prove the the rater actually made the item in question. At least on shopping sites, you can say "jrockway bought this item on 5/19/2009", so you have some idea whether or not the person has any clue.
Yes, online recipe ratings are broken, which is why I don't pay attention to them. Of course, I'm a fairly good cook, so I know how to look at a recipe and tell whether it makes sense or not. Often, I'll end up with 60% of one recipe, 15% of another, 10% of a third, and 15% my own experience to pull together a recipe for a dish I want to try to make.
If you're looking for something on how to make a dish (like I did recently for baguettes) and find several recipes to choose from, I find a few things make it easy to choose from:
1. Real, individual ingredients are better than composite ingredients (that is, flour, baking soda, oil, etc. rather than Bisquick™).
2. Simpler recipes are usually better than complex recipes, especially if you've never made the dish before. There are exceptions, of course. (I'm certain that the King Arthur baguette recipe that I found would be better than the one that I tried, but it requires making a sourdough starter some fourteen hours in advance. I will try it some day.)
3. Well-written descriptions suggest better recipes. This is a bit like code comments, though—there is such a thing as too much description.
4. Recipes that depend on exact timing of multiple pots and pans are probably not a good idea to experiment with if you're not already good at making similar dishes.
That said, I'd pay attention to recipe ratings if there were only two yes/no questions: I made this; I'd make it again. Report the resulting value as a simple ratio of (repeat)/(made once). Seriously, I don't make recipes more than once if I wasn't enthusiastic about the results or weren't convinced that I could make it better by changing the ingredients or cooking instructions better. I don't know anyone who does. Star ratings are a waste of time.
There is a factor at work in this recipe ratings systems that is a common bane to all rating systems which present a user with a range of choices. One commenter (pedalpete) noted already that people gravitate high or low. But the larger issue is the number of people that will gravitate toward the current average rating.
This is a natural desire to conform, and it isn't always conscious. If a user perceives that most people have rated this item X, he will naturally want to rate it X, and will even re-evaluate his judgment based upon the ratings of others.
Of course, for an online-rating system with instant feedback, this gives the earliest ratings the highest weight, immediately skewing all the results which follow.
But what effect does the average rating have on a person who disagrees? They will naturally tend toward the opposite extreme. They want to make their disagreement obvious. If the average rating is high, the negative rater will gravitate toward the lowest ratings, when in fact he might have gone 2 or 3 stars had he had no such prior feedback.
This is a rather intractable problem, since about the only way to prevent this sort of feedback would be to hide the rating until the person has actually tried the item in question and rated it themselves. But that, of course, makes a rating system useless.
29 comments
[ 3.3 ms ] story [ 79.5 ms ] threadOf course, with any crowd sourced system it is good to take the time to really read through who you are getting your trusted advice from. I still remember reading a review of a Korean restaurant on yelp that had two or three one star reviews because "they did not have sushi".
Their rating system is not broken to the degree of these other websites.
A good example is dishes containing coriander, a large number of people hate it with a passion while others love it. People who hate it won't make the dish and won't review it. Those who love coriander will make it and most likely enjoy it.
Applying statistical analysis with the goal of getting a bell curve isn't the correct direction to reason in.
It requires more of a slider approach though, which is something you have to spend more time teaching users about.
It really is hard to find a native New Mexican, other than those raised in Mexican households, who actually prefer or even like cilantro compared to the other ingredients we have available.
I guess I'm just furthering your point but some people may prepare those dishes with no Idea that coriander tastes the way it does and just don't have so much hate to decide to review it (or maybe they just think they lack the cooking skills to make a dish).
The number of people that come back and rate a recipe is probably small. And how many people who had a middling experience with the recipe would make the effort to come back and rate it? Or even a good experience, for that matter? A bad one? You bet they're going to rate it.
So right there, you have all kinds of bias problems with the data. And just to make it more fun, how many of those people rating the recipes prepared it properly? How many substituted ingredients that just completely ruined the dish? You see this all the time on these sites - "This recipe is horrible. I vomited for 3 hours straight. BTW I didn't have any red onions so I used pickled cocktail cherries instead."
Somehow you have to find a more indirect way of evaluating a recipe's worth. Maybe it could be a measure of how many times a person comes back to look at it (on the theory that each hit is presumably them coming back to look up that recipe again)? Or perhaps how many people chose to make it.
Additionally, I think the mentioned websites ignore an important fact about recipes: they are not static. Most recipes have variations in technique or ingredients. Many cooks, as you stated, substitute ingredients that subtly, or not so subtly, change a recipe. If a mechanism exists to capture those variations (wiki style, maybe?), those people might have an incentive to come back and share their experience rather than their impressions.
YouTube noticed a similar issue. Getting users to rate on a star rating just isn't effective. You may have more luck with a simple 'yes/no' option. Would you make this again? And offer the opportunity for comments.
I find the comments in the reviews on epicurious to be very influential in how I follow the recipe, and if I cook it at all.
I think this applies to almost all online rating systems, in particular ones where it's just pick a number out of five or ten. Most will be either very high (perfect, or perfect with a slight problem), or the lowest (completely shit). Barely anybody will mark something as 2/5 because it was utterly terrible, but this one feature was good. They'll remember the bad parts. No, no data at the moment and it's just a hunch, but one I hope to be able to measure soon.
I'm hoping to apply this to webhosting, rather than simply rate out of ten and leave a 2 sentence review, it'll be much more indepth, with multiple options (support, uptime, features etc). I'm betting that the more effort that you go through to leave a thorough rating will be balanced by the fact you won't have to rate that often (at most maybe once a year once you find a good host).
Oh well, still in the idea stage but I think people should reconsider just a simple rating out of five for anything non-trivial (like rating videos). Anything important (major purchase decision) should be more detailed and have more information for the prospective user.
Videos are a bit different, because people review as they are watching. However with something like hosting, you need to give considerable thought to how you get people to report positive or average experiences equally as often as negative ones.
That's just my initial reaction.
It's definitely something to think about and it's a pretty difficult balance I'll have to achieve but I'm confident I can figure something out.
CookThing seems similar I guess, but not quite the same.
So someone might post a bad recipe and have other members alter it and rate it highly after their own modifications. You can only figure this out by reading the comments.
So some kind of yes / no on whether you have modified the recipe is pretty essential.
When a restaurant serves one customer unsatisfactorily or a product is delivered in poor condition, the rating suffers substantially. This rating no longer represents the quality of the thing itself but also the individual's experience.
The same is true with a cook's experience preparing a recipe. Depending on the reviewer's bias, a recipe rating might describe circumstance and culinary skill better than the quality of the instructions.
As other folks here have mentioned, there's also the issue of average reviewers being not-so-great at representing their true opinions with linear rating systems, but that's a whole other topic.
Users may have better luck finding quality recipes by going to curated sites Cooks Illustrated rather than mob-rule sites like RecipeZaar. As with social sites (Digg, Reddit, HN), the chemistry of the community and the influence of the admins can have a powerful effect on the quality of the content. It also depends on what you're looking for. If all you want is cheesy enchiladas, recipes from the popular sites will work just fine.
There are many sites (not to mention books and periodicals) which have thought and care put into their recipes, often with supplementary material which helps you understand what you're doing and become a better cook, rather than just an ingredient-assembling automaton.
Another aspect to consider is that many people like to read (or watch) things about food but don't actually cook much. That's why the Food Network has become so popular (ever notice how all their commercials are for ready-made products?). A pet peeve of mine is to see 5-star reviews with a comment like, "Looks delicious! Will try making it sometime." Pretty annoying, but some people just like to watch. Maybe that's why recipes with pictures get higher ratings.
If you're looking for something on how to make a dish (like I did recently for baguettes) and find several recipes to choose from, I find a few things make it easy to choose from:
1. Real, individual ingredients are better than composite ingredients (that is, flour, baking soda, oil, etc. rather than Bisquick™). 2. Simpler recipes are usually better than complex recipes, especially if you've never made the dish before. There are exceptions, of course. (I'm certain that the King Arthur baguette recipe that I found would be better than the one that I tried, but it requires making a sourdough starter some fourteen hours in advance. I will try it some day.) 3. Well-written descriptions suggest better recipes. This is a bit like code comments, though—there is such a thing as too much description. 4. Recipes that depend on exact timing of multiple pots and pans are probably not a good idea to experiment with if you're not already good at making similar dishes.
That said, I'd pay attention to recipe ratings if there were only two yes/no questions: I made this; I'd make it again. Report the resulting value as a simple ratio of (repeat)/(made once). Seriously, I don't make recipes more than once if I wasn't enthusiastic about the results or weren't convinced that I could make it better by changing the ingredients or cooking instructions better. I don't know anyone who does. Star ratings are a waste of time.
This is a natural desire to conform, and it isn't always conscious. If a user perceives that most people have rated this item X, he will naturally want to rate it X, and will even re-evaluate his judgment based upon the ratings of others.
Of course, for an online-rating system with instant feedback, this gives the earliest ratings the highest weight, immediately skewing all the results which follow.
But what effect does the average rating have on a person who disagrees? They will naturally tend toward the opposite extreme. They want to make their disagreement obvious. If the average rating is high, the negative rater will gravitate toward the lowest ratings, when in fact he might have gone 2 or 3 stars had he had no such prior feedback.
This is a rather intractable problem, since about the only way to prevent this sort of feedback would be to hide the rating until the person has actually tried the item in question and rated it themselves. But that, of course, makes a rating system useless.