Is there any rolling-extension each time a new top submission arrives -- like in some auctions -- or is the deadline firm at X days after the first +10% entry?
BK/PC submitted their last RMSE 2009-07-25 15:53:34, three hours before The Ensemble. If they can hit the Ballmer Peak or otherwise beat the Ensemble RMSE before tomorrow morning, it should be very interesting.
Reducing the RMSE isn't quite that simple. Teams have been working for years to improve the algorithm by 10%. You can't just "boost it up" in order to win. It takes serious work. BPC is still working to improve their algorithm like everyone else but questioning whether they "expected this to happen" is ridiculous. None of the top teams "deserve to lose", they've all done great work and deserve a round of applause, regardless if they are the ultimate winner.
It's a standard ml technique that's been shown to work well, although no ones shown a proof of this. This is related to the observation "more data beats better models." This does seem to have it's limits - at some point you do need better features before you can expect your data to pay off.
It seems like just a consequence of basic statistics (central limit theorem?). Any algorithm you create has some random bias. By blending, you take several algorithms with random bias and average them together, reducing the overall bias (assuming the algorithms converge toward the correct result).
I believe it is a consequence of A) netflix evaluating the algorithms in terms of the squared error and B) different classification algorithms making different types of mistakes.
Take a simple example, with two movies (A and B), and two algorithms (X and Y). Say Alg. X rates movie A perfectly, but is off by 2 for movie B. Meanwhile, Alg. Y rates movie B perfectly, but is off by 2 for movie A.
The squared errors for the different algorithms will be:
The Ensemble is actually a meta-team of most of the teams that didn't participate in BPC, and I'm hearing that BPC had lately tried to get some other teams to join, so they were kinda expecting that (but I don't think they will be able to fire back).
Bottom line, I think that the most massive open data set out there (next to a competition) has really helped the research to move on, but I don't see the point in 5 or 10 teams joining forces to improve the results for a tiny percentage and win the prize.
On the other hand, maybe this is actually the right thing, for all these teams that have struggled for two years to be in the top-15 to receive the prize...
> I don't see the point in 5 or 10 teams joining forces to improve the results for a tiny percentage and win the prize.
I think the point is who gets the million dollars. It makes sense that people would group up, probably into as few additional participants as it would take to ensure winning. If you and I were in the vicinity of being able to win independently and we knew we would lose otherwise, I think we have every incentive to team up.
When the RMSE of a submitted prediction set on the quiz subset improves beyond the qualifying RMSE an electronic announcement will inform all registered Participants that they have thirty (30) days to submit additional candidate prediction sets to be considered for judging. At the end of this period, qualifying submissions will be judged (see Judging below) in order of the largest improvement over the qualifying RMSE on the test subset.
Maybe it should, maybe it shouldn't, but Netflix defined it as 30 days from the first submission over a 10% improvement. I'm sure there's some reason why they did it that way, but I don't know what it is.
25 comments
[ 3.7 ms ] story [ 71.4 ms ] threadTake a simple example, with two movies (A and B), and two algorithms (X and Y). Say Alg. X rates movie A perfectly, but is off by 2 for movie B. Meanwhile, Alg. Y rates movie B perfectly, but is off by 2 for movie A.
The squared errors for the different algorithms will be:
Alg X: 4 (=0^2 + 2^2)
Alg Y: 4 (=2^2 + 0^2)
Mean of Alg X and Alg Y: 2 (=1^2 + 1^2)
I think the point is who gets the million dollars. It makes sense that people would group up, probably into as few additional participants as it would take to ensure winning. If you and I were in the vicinity of being able to win independently and we knew we would lose otherwise, I think we have every incentive to team up.
I can completely understand joining forces to win the prize. $1,000,000 isn't chump change, and a smaller percentage of it is better than none.
When the RMSE of a submitted prediction set on the quiz subset improves beyond the qualifying RMSE an electronic announcement will inform all registered Participants that they have thirty (30) days to submit additional candidate prediction sets to be considered for judging. At the end of this period, qualifying submissions will be judged (see Judging below) in order of the largest improvement over the qualifying RMSE on the test subset.