This highlights something people on HN don't appreciate about machine learning: how hard it is to actually trust results, and how likely it is that the results were affected by bugs in the code or how the dataset was handled. In this case the second author was only able to replicate if he didn't shuffle the dataset. Graduate students almost never write tests for their code.
You're right, and I'm surprised (and concerned) that there haven't been any retractions or errata published for the paper cited. That said, the sentence-to-vector and paragraph-to-vector models are obvious extensions of Mikolov's original word2vec architecture (the performance of which has been extensively verified by people all over and by a myriad of reimplementations), and I can attest that they've outperformed traditional text vectorization techniques (different bag of words parameterizations, largely) as features for every task I've evaluated them on (can't be too specific publicly, unfortunately). I guess what I'm saying is that ideas can be decent in spite of a poor first implementation.
> I'm surprised (and concerned) that there haven't been any retractions or errata published for the paper cited
there is a subsequent paper by Mikolov and Mesnil at ICLR with the correct results. And code to replicate them.
Retraction? You want to retract a great paper because there is a 1-3% accuracy discrepancy for a result on a more or less random text classification task?
No, I think errata would be more appropriate; the gist of my comment was in support of the idea behind the paper, but it's disingenuous to pretend that the model works as well as described on that task.
It's also disingenuous to trivialize the difference as 1-3% - that difference is more than a third of the actual error rate (7.42% vs. 11.27%) assuming you're referring to [1]. True, the IMDB dataset isn't a very important one, but it's important to clarify when you've made a mistake, especially if your paper has been cited hundreds of times.
> about machine learning: how hard it is to actually trust results
I find the opposite true: code is easy to replicate and the datasets for algorithm comparison are open (e.g. imdb used in the PV paper). If you show very good results (especially with a simple approach such as PVs) people will immediately implement your algorithm and if their results don't match your published results, it will be known. PS: I implemented PVs shortly after it was published - though I don't care so much for the 1-3% or wtv accuracy discrepancy on the imdb dataset, the idea is great.
> Graduate students almost never write tests for their code
1) I doubt a standard software test would've helped here (probably cross-val would've caught it); 2) Who writes tests for experiment code? 3) The graduate student story is concerning: either a) someone doing a lot of the heavy lifting for the paper w/o being credited or b) this someone doesn't exist
Code is only easy to replicate when they give you or publish the code. This is not true of many ML papers. In the words of the second author, a 3% accuracy difference on this particular dataset is a "huge difference."
In fact, dismissing a 3% difference is actually reflective again of how delicate understanding ML results is. A jump from 90% accuracy to 93% accuracy is massively different than a jump from 50 to 53% or even a jump from 80 to 83%.
Almost nobody writes tests for experiment code. You're proving my point :)
> Code is only easy to replicate when they give you or publish the code
No. Graduate ML students can implement the papers they read w/o a reference implementation - just search github. As I said, I implemented PV w/o the reference code. Many others did the same even before I did.
> dismissing a 3% difference is actually reflective again of how delicate understanding ML results
Not really. I understand very well results in ML (Otherwise I would be a pretty incompetent graduate student). But does a 3% increase on say imdb translate to an increase on a another text classification task? possibly - but usually not. If it does translate well across text classification datasets, you will almost certainly see the different datasets and the results in the paper.
> Almost nobody writes tests for experiment code. You're proving my point :)
It's a good point but in my experience, the kinds of mistakes that I've usually found with my own or others experimental code would not be possible to catch with a software test. Only with analysis of the results do they become obvious.
"Each entity is
represented by a pattern of activity distributed over many computing
elements, and each computing element is involved in representing
many different entities."
I experimented with gensim's implementation of doc2vec this year; despite not being able to achieve similar results in sentiment analysis (because unshuffled datasets in the original paper) it's still really impressive. I analysed some document relations in wikipedia, and it finds some really unusual / neat relationships, e.g. Autism - Cat + Dog ~= ADHD.
Anybody have tried this algorithm compared to simpler strategy, like average of word vector, for document classification task? Or compared to using skipthought sent2vec pre-trained model?
13 comments
[ 3.0 ms ] story [ 35.8 ms ] threadHe was merely advising the first author, who actually wrote the code. Source: (requires Google login) https://groups.google.com/forum/#!topic/word2vec-toolkit/XC7... and https://groups.google.com/forum/#!msg/word2vec-toolkit/Q49FI....
This highlights something people on HN don't appreciate about machine learning: how hard it is to actually trust results, and how likely it is that the results were affected by bugs in the code or how the dataset was handled. In this case the second author was only able to replicate if he didn't shuffle the dataset. Graduate students almost never write tests for their code.
Document Embedding with Paragraph Vectors (2015) https://arxiv.org/pdf/1507.07998.pdf
there is a subsequent paper by Mikolov and Mesnil at ICLR with the correct results. And code to replicate them.
Retraction? You want to retract a great paper because there is a 1-3% accuracy discrepancy for a result on a more or less random text classification task?
It's also disingenuous to trivialize the difference as 1-3% - that difference is more than a third of the actual error rate (7.42% vs. 11.27%) assuming you're referring to [1]. True, the IMDB dataset isn't a very important one, but it's important to clarify when you've made a mistake, especially if your paper has been cited hundreds of times.
[1] https://arxiv.org/pdf/1412.5335v7.pdf
I find the opposite true: code is easy to replicate and the datasets for algorithm comparison are open (e.g. imdb used in the PV paper). If you show very good results (especially with a simple approach such as PVs) people will immediately implement your algorithm and if their results don't match your published results, it will be known. PS: I implemented PVs shortly after it was published - though I don't care so much for the 1-3% or wtv accuracy discrepancy on the imdb dataset, the idea is great.
> Graduate students almost never write tests for their code
1) I doubt a standard software test would've helped here (probably cross-val would've caught it); 2) Who writes tests for experiment code? 3) The graduate student story is concerning: either a) someone doing a lot of the heavy lifting for the paper w/o being credited or b) this someone doesn't exist
In fact, dismissing a 3% difference is actually reflective again of how delicate understanding ML results is. A jump from 90% accuracy to 93% accuracy is massively different than a jump from 50 to 53% or even a jump from 80 to 83%.
Almost nobody writes tests for experiment code. You're proving my point :)
No. Graduate ML students can implement the papers they read w/o a reference implementation - just search github. As I said, I implemented PV w/o the reference code. Many others did the same even before I did.
> dismissing a 3% difference is actually reflective again of how delicate understanding ML results
Not really. I understand very well results in ML (Otherwise I would be a pretty incompetent graduate student). But does a 3% increase on say imdb translate to an increase on a another text classification task? possibly - but usually not. If it does translate well across text classification datasets, you will almost certainly see the different datasets and the results in the paper.
> Almost nobody writes tests for experiment code. You're proving my point :)
It's a good point but in my experience, the kinds of mistakes that I've usually found with my own or others experimental code would not be possible to catch with a software test. Only with analysis of the results do they become obvious.
Distributed Representations (1986) http://stanford.edu/~jlmcc/papers/PDP/Chapter3.pdf
"Each entity is represented by a pattern of activity distributed over many computing elements, and each computing element is involved in representing many different entities."
Full Book: http://stanford.edu/~jlmcc/papers/PDP/