The Netflix Prize winners regularly compete on Kaggle. One of the top 10 Netflix Prize winners joined Kaggle early on (he got a job out of it). Kaggle competitions are a game. Most notorious players already have a good job.
I think the original intent was great. And you can learn from it still. However, as the job market for data scientists heated up, Kaggle became something else entirely.
The Netflix Prize was interesting, and drew a lot of attention to it, but ultimately wasn't that simply stuffed into the trash bin? Not long after that, Netflix basically abandoned both user ratings of significance, and realistic recommendations. Now it's just a nonsense engine with some sort of meaningless overlap or whatever they call it.
They did not implement the winning solution as is, but a lot of good stuff came from the winning teams, including techniques (such as SVD) that are in use as of today (or maybe a few years back).
> just a nonsense engine
No, it is safe to assume the recommendation engine of Netflix is close to the state-of-the-art. A lot of money and talent went into it.
"They did not implement the winning solution as is"
Shortly after that contest finished Netflix removed the five star rating system, and dramatically subdued the recommendation engine. Now the vast majority of the content surfacing is universal beyond some small category filtering (e.g. You like crime dramas and horrors so here's a bunch of the most popular stuff from those categories).
Now they have a "match" rating that I have not met a single person who finds useful (it is almost at the point of farce and seems more like a randomization engine).
A lot of money and talent went into something that Netflix clearly decided just wasn't important or useful enough for them. Now they just push Don't F*ck With Cats on everyone.
I don't see a problem with that. It seems to me most companies sponsoring the contests have legitimate problems they wanted solved.
Now it really illustrated to me how data science is still an incipient field. Even the basic Titanic example has a lot of fake solution and a bit of "cheating" (overfitting).
And don't get me started on the multitude of tutorials and examples that only show the loss decreasing graph but the actual results are not evaluated (and thus might have some glaring defects).
It’s so fascinating to see how easily a human brain can be hijacked by status. I wonder if there’s a way to harness that power for improving individual performance...
There is and it's very effective ... A company I worked in briefly used to have this badge system to honor people that go the extra mile at work, and people were very keen on getting those even though it didn't have any financial compensation behind it
Um that is kindof missing the point. A badge is nothing more than a number expressed in symbols, so you demonstrated just another variation of how to hijack the brain, even if you just read an article showing it to you with numbers...
If "improving individual performance" means to you that you perform better at your job, you are already lost.
Elementary shchool teachers use it all the time. Just some stickers and a leaderboard in the classroom and it works wonders for behavior and achievement.
I have noticed that it doesn't have much effect after a while. The students are like, meh, someone gets an award every week, and eventually it rolls around to my turn unless I really screw up.
The effect on parents seems bigger, which is a good thing, I suppose, since parents then give their kids to some positive feedback.
I participated in some of them before due to the visibility.
People from your class may know you but others, not so much. In the absence of time and other signals, management would often pick someone who is highlighted to represent something.
A lot of big people at those events generally don't mind giving contact information to young kids asking for help as much.
From there, you can effectively build a nice list of people that you can leverage in future for your resume or career.
It's like free consultation you otherwise would need to buy later.
Interesting idea. I have a very confident and talkative kid who is not shy about approaching adults. I bet he could collect lots of contact information.
No chance. HN’s goal is to have every effective programmer in the world checking it at least once a week. Anything that gets in the way of that is de facto bad.
Everything in moderation, right? Including games we play for imaginary Karma. I must say, though, I generally like the way HN does it. It isn't nearly as in your face and I don't find that motivates my behavior very much.
If I get too many positive karma posts in a row I start worrying I'm censoring myself too much. Positive karma usually means I'm just saying the most mainstream thing and that's boring.
The proposal is to see if site behavior changes as a whole when integers are removed. Individual opt-in doesn't permit such a test, nor does permitting opt-out.
In my opinion, keeping him onboard compromises h2o.ai's image as a trustworthy AI company. To many people, AI is magic, so the last thing you want is the impression that the magic is really a con.
"bringing the company into disrepute" is gross misconduct and normally a firing offence, even in countries with more liberal employment laws than the USA
This is a perfectly valid reason to fire someone, even in Europe.
If you admitted to cheating on your degree, and your university found out and revoked your degree, then you'd probably expect to get fired from your job if you no longer have the qualification you said you did. Being a top 10 kagglester is a qualification (and a highly prestigious one at that).
I never understood Kaggle. Most competitions don't require code to be submitted, just predictions to be made on a test set with missing labels. So, you don't even need to apply machine learning and I'd bet money that lots of winners don't and label by hand or outsource. I don't understand the fascination and appeal of these so-called "grandmasters".
Without very careful contest design, the best performers are obviously going to be over-fitting. Especially if the entire distribution is public. That's exactly what this team did.
This is true of academic contests in general, btw, even without cheating. They stop being interesting/fun/good signals as soon as people start treating them as an independent skill set. Comparing performance on chess games for the first N games between two new players might be a good signal for some general intellectual capabilities. Comparing experienced players against one another is mostly just testing who's spent more time learning about chess.
Winners have to submit code most of the times. There are sporadic computer vision competitions where it is possible to hand-label (such as geo-int competitions where people reverse engineered the flight paths), but vast majority it is impossible to hand-label. Nobody becomes a grandmaster without applying world-class machine learning skills.
The "cheater" in this competition is both a world-class data scientist and a reverse engineering hacker. Heck, people used to write papers about how they crawled the ground truth. Now they see their name in the newspaper.
> I never understood Kaggle. Most competitions don't require code to be submitted
Your point of view is outdated ;-)
In recent ML competitions, participants do submit code that is run on a held-out dataset - as was the case in the PetFinder.my challenge in question here.
Most competition platforms are migrating to this format, as otherwise you can just label by hand as you said.
Note that this competition went even further: not only was the evaluation code run on Kaggle, the training code was also run there. This means that you couldn't even train a gigantic model then submit it: your model had to be trainable within well defined time and resource constraints, which is a great way to level the playing field.
Of course, there's still some unfairness as people with more resources can try out more solutions before submitting a model to be trained on the platform. No platform has a solution for this yet!
It's an odd way to cheat too, if they realised they had data from the validation set, couldn't they have over-trained a model with the validation set in the training data?
Yes, that would have been harder to detect if it is allowed. My understanding is that for a kernel competition (like this one), you can't use model weights that you've trained outside the kernel. Oddly, I can't find a rule explicitly prohibiting it.
Not if the underlying model was bad, no tweaking of hyperparameters can change that. It's safe to say, he probably did consider this (and it probably didn't work well enough).
Am I missing something? It seems like "adoption speed" really isn't something you want to over optimize.
The whole point is to match abandoned animals with suitable homes. Adoption speed seems secondary to post adoption measures of adopter satisfaction and adoptee welfare.
It looks like the shelter has limited capacity, and too many dogs to be taken care of, therefore the faster dogs are adopted, the less need to be euthanized.
Also what you are writing are extremely sparse and weak signals compared to adoption speed.
It's a classic optimization problem. With high speed, but poor suitability, animals are bound to return. With low speed, but good suitability, there's low return rate - but slow turnover, which leads to overcrowding and rejection of new animals.
Like in business and tech, you want a high as posible throughput / inventory turnover rate - but obviously with constraints.
> The goal was to create an algorithm that could predict how quickly a pet would be adopted based on its profile details, from its photo to its breed, sex, size, age, and whether it had been vaccinated or not.
> These predictions would be used to optimize and tweak future critters' profiles so that they are adopted as soon as possible.
Sorry but, how is this useful? You can't just change the age of an animal to make it more likely to be adopted. The profile is meant to be an accurate representation of the animal so people know what they're getting. What exactly was the algorithm meant to achieve aside from being a predictor?
Exactly my thought when I read this also, and I think it generalizes to a lot of cases where the value of trying to predict something is dubious at best, when there's nothing obvious you can do with the prediction itself.
Sadly, many ML comps involve exactly this. Making a predictor for something, where the prediction has no real world value, and even dubious value at a thought experiment level.
I think one of the biggest skills to have in the ML space, is knowing what is worth training a model to know, and what isn't. Just like in engineering, the most successful products are those that solve a real world problem, no matter how elegantly the others might have been made.
You can use this to select which pets to put on a platform. For instance, no-kill shelters have to decide which animals they intake since they have finite room. They can save more animals if they pick animals that are likely to be adopted quickly. Obviously, kill shelters have a similar calculus when deciding which animals to cull (and indeed, animals that don't fit in the no-kill shelter go to the kill shelter).
I'm not sure how this website manages "inventory", but they might have similar problems.
I'm pretty sure so-called no-kill shelters don't outsource their killing by simply refusing less adoptable animals. And if this contest were advertised as "help us decide which animals to kill first" it probably wouldn't gain traction.
This contest sounds ridiculous. It sounds like an attempt to get in on that AI gravy but do so with some sort of feel-good element. Only there is no feel good to it, and the basic premise seems outlandish.
They limit the number of animals they have at any given time by not taking in more when they are at capacity. This is very different from running a DNN model on every applicant and refusing to intake those that aren't adoptable enough, which is a preposterous concept.
And just to provide the full picture, most no-kill shelters of course have scenarios where they euthanize -- violent animals, sick animals, etc -- but they don't need a neural network to accomplish this.
This is all neither here nor there, as the contest had positively nothing to do with any of this. Instead they wanted to determine the most adoptable traits so they could adjust the less adoptable traits with the more adoptable traits: The poodle goes through the hair straightener and gets a blonde hair color treatment (clearly I am being satirical) to make it more like a lab, for instance.
A limited number of parameters can be genuinely altered - a better photo can be taken, and vaccinations can be administered for example.
Animal rehoming centres have to balance throughput with cost; reduced per pet costs mean that they are able to expand or support more complex cases.
Whilst keeping a pet in a "space" and feeding it does cost, this cost can easily be significantly less than vaccinating, particularly as some vaccines require a few days hold post vaccine. Similarly if the vaccine will not alter a pets rehoming chance, then it is an unnecessary cost.
Pictures may be more easily applied as a tighter feedback loop (of the 5, use the 3rd) however they may also indicate other issues that could be addressed (over / underweight, coat damage, etc.) and addressing those issues have costs to balance and predict.
I have my doubts too, since there are quite likely ceiling effects. Perhaps the competition was set up out of desperation, or because the non-profit still had 25k to burn who knows?
But I would be really interested to see if it really has an effect, and if that effect can be sustained.
> because the non-profit still had 25k to burn who knows?
I think this is the closest to the truth, but that Google spent the money.
Google wants to show usefulness of ML. Marketing person comes up with competition and backs it. Pet adoption org just has to provide some data and says why not? At the very least, it is free publicity for minimal effort.
Competive ML model grading with a common training set using unseen data.
Cheat was to scrape the data that would then be used as unseen by the organisers. Unseen is now seen for this model. Then, instead of training the model with the "unseen" data, which would have been cheating and an advantage it apparently wasn't enough of an advantage so they hard code 10% of the cases to boost metrics and win.
Having more data to train your model is google & facebricks competitive advantage. Their attempts to use that advantage for something actually useful to society rather than just as a method of selling ads seems to have been a complete bust so far. If that is wrong you know better, please link us up.
I'm suspicious that their predictive power to sell ads actually works for the people who buy those ads but I guess we aren't likely to know for sure. I do wonder "who dominated their industry segment in sales by being an early adopter of google ads" I don't know anyone. It's not a great metric but what else do we have?
They scraped data from the website of the org that wanted the results and funded the competition. That data was supposed to be unseen for the competitors and used to grade the models. This cheat was to use that scraped data in its training set and, beyond that, hard code some predictions.
It's looking up the answers in the grading sheet while taking an exam.
You have a model that you have trained on some provided data - the training set. You give kaggle this model. Kaggle grades your model on some different data your model has never seen. The better your model classifies this data it hasn't seen the higher it scores and the more money you win.
So again if you trained your model (code) on a training set that you have illegally obtained. That is broken the rules of the competition to get the additional data. Data that is also going to be used for competition verification and grading, then you have cheated. Doubly so if you just hardcode outputs to boost your score, which they did here.
They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught.
It really is equivalent to looking up the answer sheet while taking an exam.
People are saying that you do not just submit your trained model to Kaggle. You also submit the code that was used to train the model from the training set, which is used in the winning models to train them from scratch on the training set. Of course, that wouldn't have prevented this type of cheating of course, but it does mean that you can't submit a model that was trained on your own private data set.
You can overfit a model to your hold out set quite easily with repeated trials. It's a trap you have to avoid in normal circumstances! (Feynman: "You are the easiest person to fool"). Even if you have to submit code to generate your model parameters from "the training set" (which hasn't been explained at all well by "People" if that is indeed the case) you could do that overfitting deliberately here with the illegal unseen data as your hold out set. Aside from the advantage of a bigger training set. Aside from the advantage in model selection, which is not done with code from a training set. Aside from the advantage to your feature engineering also not done in code. Aside from the advantage to your regularization choices, bias parameters etc etc.
So yes you absolutely /can/ submit a model trained on your own private data set even if what you submit is a model code that will be re-trained. Even if "the training set" is different to the provided - you still have that scraped data so you can slice it up with the provided training set so that any selected training set does well against the rest. Now the overfit you've just carefully engineered should win against the honest models unless you suck, right? It's kind of risible that they had to go further and hard code certain results, don't you think? Perhaps if they still couldn't win with scraped, illegal additional data then everyone else had illegal data too? Perhaps Kaggle is not a good indicator of how good ML techniques are in practise? Perhaps Kaggle systematically overstates ML effectiveness due to this kind of uncaught cheating in many of their competitions? I bet kaggle won't look too hard at that.
I'm by no means an expert in ML, but my understanding is there's some code that is run to train the modal. I meant that by "training code". My regrets if my terminology was unclear.
> They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught.
No, this is wrong
I suggest you look at one of the other comment where people have explained why this is wrong. They did a better job than me.
No it isn't. See above. Overfittiing to a holdout set is so easy and common you usually need to take steps to avoid it. See my comment directly above. Best.
This competition allowed submissions to include extra data files that can be used by the model. The cheaters added a file with data from another website that seemed innocent, but secretly encoded extra information (perfect answers) in IDs. For 10% of predictions, the code via a set of obfuscated operation retrieved this information and presented it as the answer.
I'm not going to defend Pleskov but organizers shouldn't have put out the competition with money attached that can simply be solved by scraping data. Good ML competition in fact should even invite cheats because the end goal is not ML for the sake of ML but rather cracking the prediction problem by whatever shortest path possible.
I wouldn't necessarily describe him as having a 'good' career in quant finance. He spent 18 months as a quant and then left to go and cheat at ML competitions. It's not a traditional sign of a successful quant - giving up after 18 months.
How? According to the article his model without the cheat rated at ~100th place, and the article mentions him cheating the same way before (by scraping Quora for some Quora related competition).
"Winning" by anything that can reasonably called cheating, as in this case, does not advance the general state of the art. Innovation is best served through appropriate rules and competition structure.
Yes, and that's the right thing to do in the academic research setting ("advance the state of the art"). But the public competitions with monetary rewards are not the same setting. I can imagine scenarios where the guy stole the test set from Kaggle servers (i.e. unlawful access) should disqualify him permanently. But the essence of the competition should be the focus on cracking a given problem, not about a specific technique.
One test a good of ML competition: Can it be solved by simply hiring lots of humans to make predictions without incurring significantly more costs than the prize money?
One test a good of ML competition: Can it be solved by simply hiring lots of humans to make predictions without incurring significantly more costs than the prize money?
What value to the organizers, to society or to whatever are you imagining coming out of a free-for-all style competition?
I think the organizers now imagine that the result would identifying good, generic prediction algorithms along with identifying good AI programmers capable of producing general prediction algorithms.
It seems like the contest framework already has become a bit problematic through context winners just being good at contests and not otherwise achieving anything.
But what are you thinking of? There are already hacking competitions btw.
Again, I'm not defending Pleskov. If he had come forth with the hack, things would have been different. Instead, he pretended that he had ML solution, pocket the money and put an extraordinary effort into making sure that people can't actually figure out his true doings. He was disingenuous, fully self-aware that he was in the wrong and did his best to cover up his tracks. It wasn't fair to other competitor and it was most certainly not fair to the organization trying to do something good. So yes, Pleskov, remains indefensible.
I disagree... The hope with competitions is that we learn something new through massive semi collaborative exploration. Raising the barrier to hosting useful competitions means we learn less.
Indeed you do, and I know someone who got screwed by a couple of plagiarizing cheaters. This probably contributed to his abandoning his studies and possibly to his suicide. I take a dim view of suggestions that cheating is reasonable, let alone that it is the smart option.
I would have a different interpretation. If he cheated once, he has most likely done it before and since the competition in question. Usually when 'good people' cheat there is a series of escalating transgressions before they get caught.
He scraped the test set's labels. How is that useful? It's not about "ML for the sake of ML", it's the equivalent of stealing the answers to a math test then writing them down. Why should that be rewarded?
The goal of this competition is to build a system (using ML or not) which is useful for predicting how quickly pets will be adopted. Any information used during the competition should be realistically available at inference time for future predictions... clearly, the expected answer cannot be available at the time you're trying to predict when a pet will be adopted.
People do scrape test answers and use them to optimise hyperparameters in ML models and blends of them in Kaggle. That's cheating, but at least they built a learning system here.
Before reading they'd scraped public data that was likely to be the "hidden" evaluation set, I thought they might have cheated using Python introspection: inspect the caller's frame, find some variable already loaded with the expected answer, return that.
Has anyone cheated at Kaggle/similar using that approach?
They would have able to win and get away with it if they incorporated the knowledge of the external dataset directly into the ML model, provided they had a reasonable estimate on the fraction of overlap between the external data and the test set. A weak version of this would be to just train on the external data in addition to the provided data. A stronger version would train regularly on the provided training data and in addition overfit on a random subset of some percentage of the external data (with some small random prediction error thrown in to obfuscate), which would get equivalent results to what they did with logic.
Considering the guy was smart (he is kaggle grandmaster), I would really like to know what prevented him from training on the scraped data, and what motivated him to obfuscate the known sample lookup.
Maybe there's some technicality they made it impossible to tune the model on the additional scraped training data.
I'm surprised by some of the overly sympathetic comments here, the guy cheated, not "cheated" etc.
Of course, we're all human, and he's come clean, but his actions potentially had a negative effect on the non-profit and the animals it places; and competing talents were denied their rightful places.
This comment isn't about condemning him or anything, just let's be honest about what happened here; it wasn't ok, or just system-gaming caught out.
> competing talents were denied their rightful places
Absolutely. What are the odds he (and/or others in his team?) did this just for this one competition? It's possible, but unlikely he/they invented this (or other) code hiding technique(s) just for this single occasion.
Also, the employer did the right thing and kick him out. Now they only have to scrutinize the last few months of his work instead of looking over his shoulders the next few years.
Also there may not be much correlation with profiles and speed of adoption. Why don’t they prove that first then have the competition instead of assuming it would be the solution?
It really worries me how many people are so quick to forgive him and tell him so.
In my family if someone cheated they got called a cheater and suffered consequences. At least, they would have, if someone did something like that. But my parents didn’t raise mendacious villains.
Look at this crap on Twitter:
“Everyone makes mistakes. Thank you for the apology”.
“Kagglers will still love to have you back”
“It's great that you realize your mistakes. Looking forward to see your comeback with more cool DS solutions and ethics than before.”
“Thanks for doing this. It's okay to make errors in judgement, we've all been there to varying degrees. Y'all be gonna be fine.1!”
Those are the worst. I’m not so crazy about these below, either, although there’s just a hint of steel in them, at least:
“I’m glad to see that you had a change of heart after sleeping on it and that you will be returning the prize money. I hope you will consider donating to or volunteering at a local animal shelter as well. Atonement here is more than returning the money and apologizing.”
“I hope this can be used as a teaching moment as well. Many people clearly look up to you because of your work. What can we learn from this? Something to ponder in the days to come.”
Imagine you have 100 sociopaths, all of them aiming for the top, all able to recognize each other in a crowd. Now you both compete and cooperate together to get to the top; the ones caught are written off, the uncaught ones sooner or later achieve the goal, even if only 10 out of 100 make it there.
Some headhunters specifically try to identify high-performing sociopaths for top management positions.
Kaggle should be looking to the police on how to handle cheaters. He stole 10,000 dollars, he defrauded h20.ai, etc. That's "go to jail" level crimes, not just "get banned from a competition and fired" level crimes.
I'm confused by this reaction here - this was a _brilliant_ hack of the system and I think his work should be celebrated. Was it in keeping with the intention of the competition? Of course not. Were lives threatened by their creative solution to the competition? Also no. At the end of the day it was a fun, inventive approach to a made-up problem.
That's not a hack, that's blatant cheating, their solution litterally looked at the anwsers. It bypassed the ML model prediction, so that's not a ML solution, which to my understanding was the constraint of the competition. And in the end, that solution is useless for the adoption site, since the objective is to get adoption predictions (the animal has not been adopted yet). I'm confused too, by how can anyone think it's an acceptable behavior.
Their solution made use of the data available to them in a "prize" that was nothing more than a made-up competition for fun. Did the rules expressly say somewhere that one could not make use of available datasets?
I think you misunderstood that the team in question scraped prospective test profiles from the website in question outside of the competition data flow. They blatantly cheated the competition process using data that was not supposed to be available to them.
For this particular problem, they were working on pet adoption timing prediction algorithms. The proposed solution wouldn’t work as efficiently in production as one of the other competitors. That leads to inaccurate predictions on pet adoption times, which increases costs for adoption centers, and maybe euthanasia rates among kill shelters. So animal lives would be impacted by this cheat.
More importantly, Kagggle does competitions across dozens of industries. If a culture of getting better at hiding your cheat pervades the platform, that could impact finance, transportation, and medical research. In those scenarios, lives would either be threatened or at least subject to sub-optimal systems.
Not trying to be argumentative here, but is what they did actually against the rules? I'm not at all familiar with this competition which might be why I'm not quite so worked up about this, so maybe I have misunderstood something. Did the competition really require that they only train on the provided data?
I compare this to my favorite sport, Formula 1 racing. In F1, teams of engineers with nearly unlimited budgets spend an absurd amount of effort doing everything in their power to bend the regulations (the "formula") to squeeze out some extra advantage.
For example, in this past season Ferrari was suddenly outperforming the pack (and their own recent performance) and it was clear something had changed on the car, they had power in places they didn't before. What finally came down is a clarification of the rules around fuel-rate metering, without directly calling out Ferrari. After the clarification, Ferrari power was back where it used to be. Nothing more was said of the matter by the FIA.
What we all _think_ happened is that Ferrari, knowing the fuel rate meters ran at 10kHz, discovered they could pulse their fuel pump so that the low-end of the flow rate cycle happened during that sampling interval. This means they could increase their overall fuel rate beyond what was technically allowed, due to how that technical requirement was being measured on the car (and reported back to the FIA).
Is it in keeping with the spirit of the rules? Of course not! Does it make for an interesting engineering puzzle on top of an already-exciting sport? Sure does!
Clearly I'm in the minority here, but I think this sort of problem-solving approach can be useful. If you're looking to compete against a field of entrants who are all looking for obvious and well-understood approaches to solving the problem at hand, I think sometimes the best solution to stand out is to look where the other teams aren't looking.
The issue is that the entire reason the competition exists is because the company is sponsoring it and putting forward the prize money so that the top performing models can then be put into production, thereby solving some problem the company has. This type of cheating is dishonest and against the spirit of the competition, but it also defeats the entire purpose of the exercise. Simply keeping a lookup table of answers for the data isn't machine learning, and will not generalize into a production system. As stated in the article, without these hacks, he wouldn't have even placed in the top 100.
To use your F1 analogy, this isn't the equivalent of tweaking the cars in whatever way possible is within the rules. This is the equivalent of completely cutting across the grass and bypassing 90% of the track, which is indeed illegal and would get you penalized.
That reminds me of another _brilliant_ hack I had my kid do at a youth checkers tournament. I got him to swat the other players pieces off of the board so the opponent would eventually run out of pieces before my child. I think the opposing kids parents were biased against me just because I'm an inventive genius.
Jesus fucking christ. He fucking ran MD5 on some shit he pulled down from a web crawler.
Over-training a model on the validation set would be a lot more "brilliant", and even that is a dumb script kiddie level hack. Maybe finding an algorithm that computes weights s.t. the preimage of the training algorithm on the training set matches the result of training with truly random weights using the validation set. That could be a "clever hack". And even then _brilliant_ would be.... a real fucking stretch.
>In my family if someone cheated they got called a cheater and suffered consequences.
You picked a random sample of ratio'd comments from a platform which is big on performative wokeness to strengthen your opinion, which still fails to explain how Pleskov managed to zugzwang his way into pulling off a Kobayashi Maru style move and the failure of the platform to monitor such abuses. Only time will tell, whether or not he has avoided being a part of the Dark Triad; without condoning his behaviour ─ what more can he do to atone for his sins?
My two nuggets.
This reminded me of Ijon Tichy's saying after one of his voyages:
" Thus concluded one of the most unusual of my adventures and voyages. Notwithstanding all the hardship and pain it had occasioned me, I was glad of the outcome, since it restored my faith, shaken by corrupt cosmic officeholders, in the natural decency of electronic brains. Yes, it’s comforting to know, when you think about it, that only man can be a bastard. "
How is this not criminal fraud for $10k? He deserves to go to prison. "Boo cheater" would be an appropriate response if he did it purely for ranking, not for money (or something easily sold for money).
How _is it_ criminal though? Please do tell which law his team broke here.
Put down the pitchfork and calm down. The guy is a raging a* and a cheater, but a criminal he is not. He lost his job, was publicly shamed and his reputation is tarnished basically forever. I think he got enough coming his way.
Are you sure? Aren't they just trying to show the most appropriate pets to the most appropriate people in order to make adoptions quicker? Where do you read the euthanization part?
Unfortunately, most pet shelters get more pets than their facilities are able to handle. Unless they're a no-kill shelter (in order to be considered one they need to kill <10%), they will need to euthanize in order to make room for incoming animals. This is as opposed to promoting adoption, fostering, etc. An algorithm that can prematurely decide what animals will be adopted, or have the best chance to be, can create a perverse incentive to euthanize early the less optimal pets.
THIS! Drop the "slightly"... I can't understand how people focus so much on the competition cheating, and so less on "wait, wtf are they doing here"... I mean, even if they are not deciding whether to euthanize or not based on this, you're still building a system that introduces "good looks" as a factor in a life-and-death decision regarding a living being.
It's not hard to jump from this a system that would use your facebook file to grant or deny medical health coverage or a similar life-and-death thing. Shift the Overton-window a little, push is a few notches further, and you're re-inventing phrenology with deep learning...
This is bone-chilling! I mean the fact that so many people overlook this...
If some animals are going to be euthanised, would you rather you euthanise at random, or euthanise those who are likely not going to be adopted anyway.
The problem here isn't AI, it's the problem of pet ownership. This problem has existed. And you are criticising an imperfect attempt at solving a part of this problem, because the solution isn't perfect.
The euthanization is already there. Machine learning can't justify what economy requires. The only thing machine learning promises here (and you're free to doubt the effect) is to improve adoption rates and thus reduce euthanization.
Of course this is highly problematic when applied to humans. But to be clear, some humans are already judged by models (transparent ones so far) in live and death decisions. That's how organ transplant decisions are taken. The candidates with the best prospects get the scarce organs. Similarily, when looking at populations, decisions that protect many are often taken in full knowledge of the danger to a few. Vaccination for example.
Resource optimization problems mix badly with absolute morals.
Having read the original description of the challenge[1] I don't think you're correct. I think what they're doing is trying to identify the types of photos and descriptions that are successful so they can do more of them. Like: Do you want the dog bounding through a field or do you want them snuggling up on someone's lap in the photo? That sort of stuff.
I mean, you're right, you could just run the tool, find the ones that are unlikely to get adopted and euthanize them, but I don't think there's any reason to believe that's actually their intention.
Kaggle doesn't make a lot of sense to me. It's a good self education platform but that's it. It is kind of sad to see job interview candidates for ML positions whose only ML experience is Kaggle.
196 comments
[ 2.3 ms ] story [ 270 ms ] thread"Sorry, you are likely a cheater, we logged you out and banned forever."
They did not implement the winning solution as is, but a lot of good stuff came from the winning teams, including techniques (such as SVD) that are in use as of today (or maybe a few years back).
> just a nonsense engine
No, it is safe to assume the recommendation engine of Netflix is close to the state-of-the-art. A lot of money and talent went into it.
Shortly after that contest finished Netflix removed the five star rating system, and dramatically subdued the recommendation engine. Now the vast majority of the content surfacing is universal beyond some small category filtering (e.g. You like crime dramas and horrors so here's a bunch of the most popular stuff from those categories).
Now they have a "match" rating that I have not met a single person who finds useful (it is almost at the point of farce and seems more like a randomization engine).
A lot of money and talent went into something that Netflix clearly decided just wasn't important or useful enough for them. Now they just push Don't F*ck With Cats on everyone.
Now it really illustrated to me how data science is still an incipient field. Even the basic Titanic example has a lot of fake solution and a bit of "cheating" (overfitting).
And don't get me started on the multitude of tutorials and examples that only show the loss decreasing graph but the actual results are not evaluated (and thus might have some glaring defects).
How is that different from "LeetCode PhD" doing "DP Hard" for 6 months to get into FAANG?
Accumulated integer counts that represent social value and/or standing encourage destructive behavior.
If "improving individual performance" means to you that you perform better at your job, you are already lost.
The effect on parents seems bigger, which is a good thing, I suppose, since parents then give their kids to some positive feedback.
I participated in some of them before due to the visibility. People from your class may know you but others, not so much. In the absence of time and other signals, management would often pick someone who is highlighted to represent something.
A lot of big people at those events generally don't mind giving contact information to young kids asking for help as much.
From there, you can effectively build a nice list of people that you can leverage in future for your resume or career.
It's like free consultation you otherwise would need to buy later.
Perhaps someday we'll get a week without them.
(I’d love to be surprised, though.)
You can get the same effect with a Tampermonkey script I wrote: https://news.ycombinator.com/item?id=14456200
It even scrambles the karma count in your profile, so there’s no opportunity for karma to affect you.
Jeff Atwood on Coding Horror. (He can't be the first person to have said this, though.)
https://twitter.com/ppleskov/status/1215983188876709888?s=19
The person was originally employed at h2o.ai and as a consequence of this was fired. Not sure if that was completely appropriate.
Wasn't this a personal participation? Or are there "company teams" on Kaggle ?
In my opinion, keeping him onboard compromises h2o.ai's image as a trustworthy AI company. To many people, AI is magic, so the last thing you want is the impression that the magic is really a con.
He has only been at h20 for a few months.
He was almost certainly hired at least in part because of his participation in the contest.
If you admitted to cheating on your degree, and your university found out and revoked your degree, then you'd probably expect to get fired from your job if you no longer have the qualification you said you did. Being a top 10 kagglester is a qualification (and a highly prestigious one at that).
This is true of academic contests in general, btw, even without cheating. They stop being interesting/fun/good signals as soon as people start treating them as an independent skill set. Comparing performance on chess games for the first N games between two new players might be a good signal for some general intellectual capabilities. Comparing experienced players against one another is mostly just testing who's spent more time learning about chess.
The "cheater" in this competition is both a world-class data scientist and a reverse engineering hacker. Heck, people used to write papers about how they crawled the ground truth. Now they see their name in the newspaper.
Your point of view is outdated ;-)
In recent ML competitions, participants do submit code that is run on a held-out dataset - as was the case in the PetFinder.my challenge in question here.
Most competition platforms are migrating to this format, as otherwise you can just label by hand as you said.
Note that this competition went even further: not only was the evaluation code run on Kaggle, the training code was also run there. This means that you couldn't even train a gigantic model then submit it: your model had to be trainable within well defined time and resource constraints, which is a great way to level the playing field.
Of course, there's still some unfairness as people with more resources can try out more solutions before submitting a model to be trained on the platform. No platform has a solution for this yet!
I feel as though comments are less informative, more judgemental, and chalk full of emotions.
...as if a younger crowd of users has started using it.
Have the older-wiser HNers started using another site?
The whole point is to match abandoned animals with suitable homes. Adoption speed seems secondary to post adoption measures of adopter satisfaction and adoptee welfare.
Also what you are writing are extremely sparse and weak signals compared to adoption speed.
Like in business and tech, you want a high as posible throughput / inventory turnover rate - but obviously with constraints.
I'm not even certain of how that could be objectively measured for children, let alone pets.
> These predictions would be used to optimize and tweak future critters' profiles so that they are adopted as soon as possible.
Sorry but, how is this useful? You can't just change the age of an animal to make it more likely to be adopted. The profile is meant to be an accurate representation of the animal so people know what they're getting. What exactly was the algorithm meant to achieve aside from being a predictor?
I think one of the biggest skills to have in the ML space, is knowing what is worth training a model to know, and what isn't. Just like in engineering, the most successful products are those that solve a real world problem, no matter how elegantly the others might have been made.
I'm not sure how this website manages "inventory", but they might have similar problems.
This contest sounds ridiculous. It sounds like an attempt to get in on that AI gravy but do so with some sort of feel-good element. Only there is no feel good to it, and the basic premise seems outlandish.
They do, though. That's how they are able to limit the number of animals they have at any given time.
And just to provide the full picture, most no-kill shelters of course have scenarios where they euthanize -- violent animals, sick animals, etc -- but they don't need a neural network to accomplish this.
This is all neither here nor there, as the contest had positively nothing to do with any of this. Instead they wanted to determine the most adoptable traits so they could adjust the less adoptable traits with the more adoptable traits: The poodle goes through the hair straightener and gets a blonde hair color treatment (clearly I am being satirical) to make it more like a lab, for instance.
A limited number of parameters can be genuinely altered - a better photo can be taken, and vaccinations can be administered for example.
Animal rehoming centres have to balance throughput with cost; reduced per pet costs mean that they are able to expand or support more complex cases.
Whilst keeping a pet in a "space" and feeding it does cost, this cost can easily be significantly less than vaccinating, particularly as some vaccines require a few days hold post vaccine. Similarly if the vaccine will not alter a pets rehoming chance, then it is an unnecessary cost.
Pictures may be more easily applied as a tighter feedback loop (of the 5, use the 3rd) however they may also indicate other issues that could be addressed (over / underweight, coat damage, etc.) and addressing those issues have costs to balance and predict.
But I would be really interested to see if it really has an effect, and if that effect can be sustained.
I think this is the closest to the truth, but that Google spent the money.
Google wants to show usefulness of ML. Marketing person comes up with competition and backs it. Pet adoption org just has to provide some data and says why not? At the very least, it is free publicity for minimal effort.
Competive ML model grading with a common training set using unseen data.
Cheat was to scrape the data that would then be used as unseen by the organisers. Unseen is now seen for this model. Then, instead of training the model with the "unseen" data, which would have been cheating and an advantage it apparently wasn't enough of an advantage so they hard code 10% of the cases to boost metrics and win.
Having more data to train your model is google & facebricks competitive advantage. Their attempts to use that advantage for something actually useful to society rather than just as a method of selling ads seems to have been a complete bust so far. If that is wrong you know better, please link us up.
I'm suspicious that their predictive power to sell ads actually works for the people who buy those ads but I guess we aren't likely to know for sure. I do wonder "who dominated their industry segment in sales by being an early adopter of google ads" I don't know anyone. It's not a great metric but what else do we have?
It's looking up the answers in the grading sheet while taking an exam.
You have a model that you have trained on some provided data - the training set. You give kaggle this model. Kaggle grades your model on some different data your model has never seen. The better your model classifies this data it hasn't seen the higher it scores and the more money you win.
So again if you trained your model (code) on a training set that you have illegally obtained. That is broken the rules of the competition to get the additional data. Data that is also going to be used for competition verification and grading, then you have cheated. Doubly so if you just hardcode outputs to boost your score, which they did here.
They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught.
It really is equivalent to looking up the answer sheet while taking an exam.
So yes you absolutely /can/ submit a model trained on your own private data set even if what you submit is a model code that will be re-trained. Even if "the training set" is different to the provided - you still have that scraped data so you can slice it up with the provided training set so that any selected training set does well against the rest. Now the overfit you've just carefully engineered should win against the honest models unless you suck, right? It's kind of risible that they had to go further and hard code certain results, don't you think? Perhaps if they still couldn't win with scraped, illegal additional data then everyone else had illegal data too? Perhaps Kaggle is not a good indicator of how good ML techniques are in practise? Perhaps Kaggle systematically overstates ML effectiveness due to this kind of uncaught cheating in many of their competitions? I bet kaggle won't look too hard at that.
I'm by no means an expert in ML, but my understanding is there's some code that is run to train the modal. I meant that by "training code". My regrets if my terminology was unclear.
> They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught.
No, this is wrong
I suggest you look at one of the other comment where people have explained why this is wrong. They did a better job than me.
https://news.ycombinator.com/item?id=22124760
https://news.ycombinator.com/item?id=22126489
https://news.ycombinator.com/item?id=22124193
How? According to the article his model without the cheat rated at ~100th place, and the article mentions him cheating the same way before (by scraping Quora for some Quora related competition).
One test a good of ML competition: Can it be solved by simply hiring lots of humans to make predictions without incurring significantly more costs than the prize money?
What value to the organizers, to society or to whatever are you imagining coming out of a free-for-all style competition?
I think the organizers now imagine that the result would identifying good, generic prediction algorithms along with identifying good AI programmers capable of producing general prediction algorithms.
It seems like the contest framework already has become a bit problematic through context winners just being good at contests and not otherwise achieving anything.
But what are you thinking of? There are already hacking competitions btw.
The goal of this competition is to build a system (using ML or not) which is useful for predicting how quickly pets will be adopted. Any information used during the competition should be realistically available at inference time for future predictions... clearly, the expected answer cannot be available at the time you're trying to predict when a pet will be adopted.
If one can excuse scraping the data to build a better model, I can't see how one can excuse this.
He wasn't really making predictions at all, just looking up the answer.
And the goal really is a prediction system, not something that looks up previously known answers.
Has anyone cheated at Kaggle/similar using that approach?
In this competition, the training code was run on Kaggle's system, so you'd still need to smuggle in the extra data.
You've got the testing set. Create random HPs and tune them to fit. The way they cheated is stupid.
And the way the testing set can be obtained is silly.
Considering the guy was smart (he is kaggle grandmaster), I would really like to know what prevented him from training on the scraped data, and what motivated him to obfuscate the known sample lookup.
Maybe there's some technicality they made it impossible to tune the model on the additional scraped training data.
Of course, we're all human, and he's come clean, but his actions potentially had a negative effect on the non-profit and the animals it places; and competing talents were denied their rightful places.
This comment isn't about condemning him or anything, just let's be honest about what happened here; it wasn't ok, or just system-gaming caught out.
Absolutely. What are the odds he (and/or others in his team?) did this just for this one competition? It's possible, but unlikely he/they invented this (or other) code hiding technique(s) just for this single occasion.
Also, the employer did the right thing and kick him out. Now they only have to scrutinize the last few months of his work instead of looking over his shoulders the next few years.
In my family if someone cheated they got called a cheater and suffered consequences. At least, they would have, if someone did something like that. But my parents didn’t raise mendacious villains.
Look at this crap on Twitter:
“Everyone makes mistakes. Thank you for the apology”.
“Kagglers will still love to have you back”
“It's great that you realize your mistakes. Looking forward to see your comeback with more cool DS solutions and ethics than before.”
“Thanks for doing this. It's okay to make errors in judgement, we've all been there to varying degrees. Y'all be gonna be fine.1!”
Those are the worst. I’m not so crazy about these below, either, although there’s just a hint of steel in them, at least:
“I’m glad to see that you had a change of heart after sleeping on it and that you will be returning the prize money. I hope you will consider donating to or volunteering at a local animal shelter as well. Atonement here is more than returning the money and apologizing.”
“I hope this can be used as a teaching moment as well. Many people clearly look up to you because of your work. What can we learn from this? Something to ponder in the days to come.”
Some headhunters specifically try to identify high-performing sociopaths for top management positions.
[Citation needed]
A lifetime ban would not be out of the place here, considering it went on to win the competition with no admission until caught.
He was publicly shamed, banned from Kaggle and he lost his job (and I guess he's basically unhireable right now).
I wonder what would be your family reaction, if these things don't sound like consequences to you.
So what's the problem?
The prize was $10,000, not "fun".
More importantly, Kagggle does competitions across dozens of industries. If a culture of getting better at hiding your cheat pervades the platform, that could impact finance, transportation, and medical research. In those scenarios, lives would either be threatened or at least subject to sub-optimal systems.
I compare this to my favorite sport, Formula 1 racing. In F1, teams of engineers with nearly unlimited budgets spend an absurd amount of effort doing everything in their power to bend the regulations (the "formula") to squeeze out some extra advantage.
For example, in this past season Ferrari was suddenly outperforming the pack (and their own recent performance) and it was clear something had changed on the car, they had power in places they didn't before. What finally came down is a clarification of the rules around fuel-rate metering, without directly calling out Ferrari. After the clarification, Ferrari power was back where it used to be. Nothing more was said of the matter by the FIA.
What we all _think_ happened is that Ferrari, knowing the fuel rate meters ran at 10kHz, discovered they could pulse their fuel pump so that the low-end of the flow rate cycle happened during that sampling interval. This means they could increase their overall fuel rate beyond what was technically allowed, due to how that technical requirement was being measured on the car (and reported back to the FIA).
Is it in keeping with the spirit of the rules? Of course not! Does it make for an interesting engineering puzzle on top of an already-exciting sport? Sure does!
Clearly I'm in the minority here, but I think this sort of problem-solving approach can be useful. If you're looking to compete against a field of entrants who are all looking for obvious and well-understood approaches to solving the problem at hand, I think sometimes the best solution to stand out is to look where the other teams aren't looking.
To use your F1 analogy, this isn't the equivalent of tweaking the cars in whatever way possible is within the rules. This is the equivalent of completely cutting across the grass and bypassing 90% of the track, which is indeed illegal and would get you penalized.
Jesus fucking christ. He fucking ran MD5 on some shit he pulled down from a web crawler.
Over-training a model on the validation set would be a lot more "brilliant", and even that is a dumb script kiddie level hack. Maybe finding an algorithm that computes weights s.t. the preimage of the training algorithm on the training set matches the result of training with truly random weights using the validation set. That could be a "clever hack". And even then _brilliant_ would be.... a real fucking stretch.
You picked a random sample of ratio'd comments from a platform which is big on performative wokeness to strengthen your opinion, which still fails to explain how Pleskov managed to zugzwang his way into pulling off a Kobayashi Maru style move and the failure of the platform to monitor such abuses. Only time will tell, whether or not he has avoided being a part of the Dark Triad; without condoning his behaviour ─ what more can he do to atone for his sins?
" Thus concluded one of the most unusual of my adventures and voyages. Notwithstanding all the hardship and pain it had occasioned me, I was glad of the outcome, since it restored my faith, shaken by corrupt cosmic officeholders, in the natural decency of electronic brains. Yes, it’s comforting to know, when you think about it, that only man can be a bastard. "
(source: The Star Diaries, Stanisław Lem)
Similar with school: how many times you, excellent grades holder, was cheating because no one scrutinizes high grade holders too much?
Put down the pitchfork and calm down. The guy is a raging a* and a cheater, but a criminal he is not. He lost his job, was publicly shamed and his reputation is tarnished basically forever. I think he got enough coming his way.
They're essentially letting an algorithm decide which dogs have the best chance of adoption and which to euthanize, aren't they?
It's not hard to jump from this a system that would use your facebook file to grant or deny medical health coverage or a similar life-and-death thing. Shift the Overton-window a little, push is a few notches further, and you're re-inventing phrenology with deep learning...
This is bone-chilling! I mean the fact that so many people overlook this...
The problem here isn't AI, it's the problem of pet ownership. This problem has existed. And you are criticising an imperfect attempt at solving a part of this problem, because the solution isn't perfect.
shake my head
Of course this is highly problematic when applied to humans. But to be clear, some humans are already judged by models (transparent ones so far) in live and death decisions. That's how organ transplant decisions are taken. The candidates with the best prospects get the scarce organs. Similarily, when looking at populations, decisions that protect many are often taken in full knowledge of the danger to a few. Vaccination for example.
Resource optimization problems mix badly with absolute morals.
I mean, you're right, you could just run the tool, find the ones that are unlikely to get adopted and euthanize them, but I don't think there's any reason to believe that's actually their intention.
[1]:https://www.kaggle.com/c/petfinder-adoption-prediction/overv...