The abstract is damning, but not surprising to anyone outside the machine learning bubble. It's a symptom of a major problem in meta-science: breakthroughs are hard, and when you shovel billions of dollars into trying to incentivize breakthroughs, you end up with a lot of envelope-pushing dressed up like breakthroughs, and most of them don't even successfully push the envelope.
biology, especially genomics, is this way as well. Every day you see research articles where a scientist studied something and the PR folks (or even the paper) present it as a solution to a major medical problem. I spent a good part of my graduate career attempting to write code that duplicated the results in papers; I'd say about 75% of the time, I convinced myself the paper was simply wrong due to technical errors that a bit of data science was able to uncover.
12 hours, that's early, according to a friend of mine doing his PhD in ML/NN (or as he put it: everyone uses L²-loss and SGD - it's just not original)... He also thinks that the quality of papers from a formal standpoint is very low (not speaking of content yet). Riddled with spelling errors and illogical sentences. But as published around NIPS gives all the credits and fixing those things does not (moreso, it probably taints results), this won't change anytime soon...
Seems like the desired outcome here is an unfixed target. What does Facebook want to show you next in the Newsfeed, and why? It's gonna be different from the factors Amazon evaluates to determine your home page suggested items. So while it's worth trying to replicate academic research on these datasets like MovieLens, the real-world outputs from collaborative filtering are inherently subjective.
Yeah, at least in the short term. Which makes me think that beyond obvious patterns, the question of accuracy (is this a successful recommender system or not?) is hard to define in a way that can be demonstrated in a lab.
In general, if you provide a social media service, optimising for ads will probably not lead to you becoming a world-dominating destroyer of democracy, like Facebook.
You probably want to optimise for keeping people on your service, first (and last).
"Probing Neural Network Comprehension of Natural Language Arguments
Timothy Niven, Hung-Yu Kao
(Submitted on 17 Jul 2019)
We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average untrained human baseline. However, we show that this result is entirely accounted for by exploitation of spurious statistical cues in the dataset. We analyze the nature of these cues and demonstrate that a range of models all exploit them. This analysis informs the construction of an adversarial dataset on which all models achieve random accuracy. Our adversarial dataset provides a more robust assessment of argument comprehension and should be adopted as the standard in future work."
In the field of computer vision, there are similar suspicions that state of the art computer vision models are overfit to ImageNet datasets and the like. The issue being that even average research labs do not have hundreds of thousands of dollars to reproduce extremely expensive and highly tuned models. Reproducing and advancing the field of deep learning is quickly becoming inaccessible to almost everyone except for a few of the highest funded industrial labs (Google, FB, OpenAI, Microsoft Research among a few others). This is not all negative, it's just that not everything can be taken as gospel.
The paper is relevant to the original post because it points out that NLP models show state of the art results that are overfit to their respective datasets. This means that we may not be advancing as fast as we think. The issue here is also that few people have the monetary resources to provide a secondary analysis on these models. It's too expensive.
I think this is way too broad an interpretation to draw from a paper that shows that a single dataset was bad. NLP datasets have had a good chunk of these issues, but people have been aware of this issue for a few years and newer datasets, e.g. SQuAD 2.0 explicitly try to address these sorts of issues.
Any chance you can provide some more recent results that justify your lack of concern?
I'm an outsider to NLP, but have been very surprised by the recent improvements, so would be interested in anything that covers their limitations and newer datasets which help us to get over them.
The main breakthrough in the last 2 years is increasing success in unsupervised/self-supervised pre-training on very large datasets transfering to other tasks, e.g. ELMo, BERT, XLNet.
Have a look at the BERT & XLNet papers, they have results on a wide range of datasets, including SQuAD 2.0.
I think it's worth keeping in mind that all of the headlines about "surpases human performance" are 100% bullshit regardless of whether they are about vision, ASR or NLP. Performance on a dataset is not the same thing as real world usefulness.
But this takes nothing away from the fact that ELMo/BERT/XLNet make it far easier to pull a model off the shelf and fine tune it for your task and get good performance.
And the gains here are so broad and not task specific that issues with individual datasets are not really an issue, it is the massive gains in pretraining that are the real story, and if that even helps you overfit a test set, it shows that they now have enough built-in linguistic knowledge that we previously didn't have to even overfit a bad test set.
This is obviously bad news for you if you thought that these tasks were basically solved, but it's completely irrelevant if you were building systems since these methods all result in relative gains on whatever language task they were pointed at.
The bigger issue IMO with many recent NLP papers is not that they are expensive to train from scratch, which is a one time cost, but that they are expensive to even do inference with, which makes it hard to actually deploy them in real systems.
But if your question is "is recent progress in NLP illusory" I think the answer is clearly no since the gains were across a broad range of tasks, and frankly I've used BERT and it makes my application better, the main issue I have is the runtime issue.
"Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques. "
Ha. I get a feeling there's alotta overpaid data scientists out there.
Basically when you see a RNN in industry, chances are somebody has made a huge and expensive mistake. And quite likely the person behind it thinks that they've applied cutting edge research to a real world problem (when they have done neither).
I met a dude once who was using LSTMs to predict HVAC failures for some sort of HVAC support firm. On some interrogation, it turned out that the input data was garbage, and the only way the model "worked" is if it was repeatedly overfit on only the last week's worth of data... comprising about 5k rows, iirc.
This struck me as completely terrifying... it's not like the set of HVAC units was changing out every week. And one would hope that the historical data would help arrive at a general solution. My guess is they had some sort of church-meets-state problem in the test/train split, so that the short-term data allowed 'predicting' the test set.
Dude was quite convinced that they were making HVAC history, though.
My experience (working in industry) is machine learning projects tend to go something like this:
Someone on the company board has just read a pop sci article about "machine learning" and now as a result the company has engaged some expensive consultants to advise us on "potential machine learning opportunities." The consultant spends maybe a day or two touring one of the company's plants and has only the most broad and high level view of how our industrial process actually works. The consultant then promises the world "machine learning is great, you can use it here, here and here for sure. It will save you millions..." It gets the board members excited.
So the company picks some toy problem and gives the consultant and his team a six month contract to deliver a proof of concept. The deadline passes a solution is delivered - it is underwhelming. The contract is not extended.
edit: All cynicism aside I appreciate the possibilities but for ML to succeed the developers really need to be embedded/integrated into the company have a good understanding of problem space and fundamentals. Not really something you can outsource easily and expect to succeed.
Summarizing, they tried replicating the results of 18 recent papers, and determined that because of lack of available code or data, they were only able to attempt replications of 7 of them. Of these, they found that the "improvements" in 6 of them could also be achieved with a simpler non-machine learning approach (such as nearest neighbor). For the 7th (Mult-VAE), they were able to almost match the improvement with better tuning of one of the baselines used in that paper.
Does this mean that the published results are garbage? No! But what does it mean?
First, don't miss the main claim: less than half the published papers have sufficient code and data available for replication. While one might argue exactly what the standards for availability should be, if journals and reviewers were to demand it, this number can definitely be improved. Greater availability of data and code makes for better science.
Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication. They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement. Unfortunately, they are probably right.
The useful change (in my opinion) would be to change the standards by which papers are judged to be publishable. Reviewers should push back against weak baselines, but be more accepting of results that make modest claims. You can still have a useful paper even if the degree of improvement is small --- or even nonexistent. Maybe it will inspire someone else, maybe it's useful in an ensemble, or maybe it applies to a case where the baseline wouldn't. It's the puffery that's the problem, not the publication. Creating a system that rewards authors who more honestly evaluate their work would be an overall win.
Is there some background reason that academics generally don't provide code in their papers? Hell, in a lot of papers I've tried looking at recently[0], there will be paragraphs and paragraphs of english text describing an algorithm rather than even just plunking down some pseudo code.
It makes pulling out the actual content of the papers such a slog.
Research code is generally an abomination, scraped together by MS/PhD students who may never have had any industry exposure to best practices, norms, testing, etc. There may also be the fear of releasing buggy code, which if/when discovered, will discredit the paper. Far safer to just not release the code, since it isn't mandated (which should absolutely 100% change)
Source: am former producer/publisher of abomination research code.
I wonder if a journal could be established that emphasizes code integrated papers, something that might accept submissions as jupyter notebooks for example.
Edit: Perhaps one way to encourage this is via topic "reuse" papers, which present further detail on a previously published papers topic, but this time with code and more detailed discussion. It would at least help the situations in that the author gets some reuse of effort, but in a way that still shows new information and advances the field.
> One good test for whether your article is a fit for Distill is whether your collaborators and you are willing to put in whatever time is necessary to write and illustrate an outstanding article. In our experience, this often takes 100+ hours.
Reproducibility is one of the core problems of the article. I'm not sure any format can assure understanding by itself - that's more the purview of the reviewers I would think. Applying the criticism to my own suggestion: even the reproducibility issue is only tangentially helped by a Jupyter format, it may only bring reviewers to the point where one might execute the notebook (though there are still issues there if its an exceeding long computation or needs special hardware setup, or the notebook executes w/o demonstrating actual computation).
This is a current area of research as the prevalence of interactive notebooks has peaked. As of now, they simply "EXIST" and we know we like them. Its still unclear as to why; obviously having code intertwined other common communication mediums but what about that makes us like it. What benefits does it have over pseudocode or just code? What about code beyond small scripts (500+ lines with design patterns or other abstractions)?
Jupyter notebooks are risky business for reproducible work. No dependency data means they're highly prone to bit rot. Stored results and out-of-order execution means they're prone to subtle errors. Environmental leakage is relatively high.
Literate programming suffers from this in general because you don't want to clutter up your document with noise about versions and so forth - even as appendices, including package information and build instructions is a lot of noise.
Requiring that papers making claims about some code base publish that code base simultaneously with the paper via the peer review process should be sufficient to improve the overall state of affairs.
Don't underestimate the value of demos and competitions with companion papers either, those tend to get a lot more notice.
I dunno, I get where you are coming from with respect to literate programming, but I find that it's often better to show all the versions in an org file (or whatever tool you use) and write up a report separately including the final results.
In general, you'll have a lot of approaches that don't work out which are nice to have a record of, but definitely don't merit being in the final paper.
A published code base doesn't necessarily mean a full revision history, just something others can reproducibly build and run.
If your claims don't depend on specific behaviours of some body of code, you wouldn't need it - eg, an article claiming some asymptotic performance of an algorithm should describe the algorithm in the abstract s.t. the performance bound can be proven, not some specific language's implementation of the algorithm.
Seems like one should be able to tag some metadata to a notebook - as little as a git url and a hash or as exotic as an IPFS link or (I hate to say it) some other blockchain info. Somehow dependencies would seem to be something needed to be specified in the submittal standards of a code-required journal
My experience with ML notebooks in particular has been poor. But most of the publications I read are not in the field of ML, and I tend to prefer ones that use an abstract definition of an algorithm rather than code in any case - whatever the language of the day is, it probably won't be compilable in fifteen years, but the algorithm will probably still be relevant.
There are some journals who have started taking it seriously. eLife does that: asks authors to put their code on GitHub/gitlab etc. Fork their repo and use that as snapshot for the published paper. I like the eLife approach.
But for most journals, it's the story as usualy. Personally I dont feel very confident about a paper result if they don't publish the code. Why hold back on the code?
I disagree that it is a lack of industry exposure. "Best practices, norms, testing, etc." is a waste of time when you develop well written code that doesn't produce good results. After months of tweaking even a good foundation of code, it begins looking atrocious. But you get graduated for publications based on good results, not for code that someone else can use.
testing, refactoring, code organization, consistent code styling for easy search-replace, and other best practice norms will save you months of effort in tracking down bugs, fixing new ones you introduced while scrambling around researching, figuring out what's going on when you have an idea that uses code you wrote six months ago, etc. Sure, maybe a tutorial and documentation might not be worth the effort for something that hasn't produced results (does publishable research still count as not having produced results?), but most industry best practices I can think of will save you much more than the time invested even on relatively modest (< 1 month) projects.
> After months of tweaking even a good foundation of code, it begins looking atrocious.
Months of tweaking code is exactly what you want some test coverage for - so that you end up with a tweaked version of something where you still know what it does instead of something that does god knows what.
> The CRAPL is an open source "license" for academics that encourages code-sharing, regardless of how much how much Red Bull and coffee went into its production.
In addition, you may want to milk your code for more papers. Releasing the code makes it easier for others to also do that milking, too, finding and publishing some results before you can find them.
If you saw the code academics and students write, you'd know why they don't want to publish it.
Also, more likely and non- nefarious, but no less harmful, there's just an academic culture that says "this is what a paper looks like and this is what it contains". Moving from this pattern increases the odds you'll not be published, and it would also likely result in a scandal when someone uses it to point out that the peers in peer review are neither peers (they can't read nor understand nor test the code) nor reviewing (they won't actually put resources into code review).
Oh I don't wonder :) After a few years experience with analysis you sometimes see certain patterns in papers which strongly suggest bugs you've experienced, phenomenon in the data set you're already familiar with, and hacking/overfitting. And some of the languages are specifically designed to keep running and produce wrong results rather than crash and abort: a cynical man might say this is a feature not a bug and that's why they're popular.
As a non-wealthy, non-academic, this isn't a battle I'm interested in fighting directly. There are other hills I'd rather die on :)
Well. In theory, one should be able to reproduce (implement) an algorithm described in a paper just by reading the paper. It's better if the algorithm is typeset as, you know, an algorithm, but if the algorithm can be better described in plain English...
Besides, _ideally_, machine learning reseach is not about building a system but about making a theoretical claim that is supported by empirical results. The main claim should be about a property of the algorithm: its time complexity, its convergence, its correctness, etc. Experiments should directly test the theoretical claim and verify it in practice.
... ideally. In practice most machine learning papers are vehicles for a little table that compares your system with a baseline that you chose based on the fact that your system beats it (or you wouldn't have put it in the paper). Whether your experiments prove anything beyond the fact that you beat some benchmark is not really important.
And so whether someone publishes a reproducible paper is of no consequence because the only point of papers is to pass review and convince someone that the researchers should stay in the (research) money. In any case, things move so fast that most of the time nobody wants to reproduce your old and unsexy paper from last week.
Then some busybody comes along and points out the fact that you're wearing no clothes. Like in the paper above.
Once again, I'll attribute to malice what many here are so eager to attribute to stupidity (or something rather similar). Basically, they don't have any incentive to do so. The goal of the researcher is to be marked as an author of something that will be considered as an important result in the domain, not to actually help the others to understand the result and to be able to build upon it.
The paragraphs of English text help them to be able to claim that they are indeed the authors of some novel method (exactly what the author wants).
Code/data accessibility helps others to both find errors/imperfections in their approach and to build upon their results (exactly what the author doesn't want).
I did my thesis on OCR, and even simple binary thresholding algorithms (to decide a cutoff point for "background" vs "text") that might boil down to a dozen lines of code if that in some cases, would often be presented as a formula (great; at least something) or imprecise prose, but where you then when translating it into code would find that their results depended on setting certain parameters to certain values that they'd not specified anywhere.
In some cases I basically had to run a test harness and brute force the parameters they'd used, because the algorithm was basically useless without picking parameters in the right range.
It was hard to accept then (~2008-ish), but now we're at a point where frankly at least for the more basic CS papers we ought to be able to expect source and some runnable format (e.g. a VM image) with test data.
I handed in a DVD with gigabytes of test data, full source, built binaries, and a test harness for the work I did for my thesis. All it took was to follow practices that I needed to follow to satisfy myself that my results were valid anyway.
I realize that for some work that relies on much larger and/or proprietary datasets making everything available would be hard, but a small test case illustrating the code run with the right parameters etc. ought to be a minimum unless there are substantial barriers, in which case the paper ought to signpost clearly what is missing and why.
>> Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication. They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement. Unfortunately, they are probably right.
I fear this contradicts your assertion that published results are not garbage. If researchers are claiming their systems are strong based on the fact that they beat weak baselines, then their claims are rubbish and their scholarship shoddy. If, on top of that, their work is difficult to reproduce, or replicate (by the distinction in the paper) then the published results are beyond garbage- they are meaningles garbage, garbage that fails to prove even its weak and insignificant claim to beat a weak baseline.
Basically, this is exactly the kind of shoddy scholarship that Backus accused social scientists of, in his infamous speech about "Cargo-Cult Science". In that speech, he points out that basing new research on previous weak research can only lead in the proliferation of weak results (my interpretation of his words, from memory).
This seems very much to be the case in machine learning research today. This article only provides further evidence that there is something rotten to the core in machine learning research.
>> Reviewers should push back against weak baselines, but be more accepting of results that make modest claims.
Agreed about the second part, however, the paper in the OP says that the way things are, it is very difficult to know which baselines are strong. My interpretation: because they're all trying to beat weak baselines that became established by beating earlier weak baselines. It's weak baselines all the way down.
Weak baselines are not necessarily inappropriate. If using a new approach the point is to show that it is a viable direction for further exploration and refinement, not always whether it can beat finely tuned, well explored approaches.
I disagree with you. Weak baselines are inappropriate, especially when claiming 'state-of-the-art' results, which is very common in ML literature these days. If weak baselines are used as proof of superiority for a particular method, there are infinite other methods/minor adjustments which can be shown to be superior using the same logic.
State of the art is already a dubiously vague expression, but I think you miss my point. I would think usually the point is not that it is superior approach, but a viable one to explore.
You're right, it was Feyman. Somehow I got the two mixed in my head- apologies.
Also, sorry HN, for the controversial comment (judging by the rollercoaster vote activity). I hope it adds something to the discussion despite the tone and I'll remember to keep the tone down in the future (but I keep saying that and I always end up giving in to the temptation to use more colourful language. I suck and I'm sorry).
Lately I was entertaining an idea of how this whole thing may be improved, but I don't really see how can I possibly influence it, so let me splash it out here.
What do we have now:
1. We have a common platform, which is a de-facto standard for sharing (somewhat math-related) papers: arxiv.org. Basically, anyone can upload anything to arxiv.org, there's not much of quality review required. Which, all in all, I think is great. I mean, it's orders of magnitude better than other fields, where both publishing and accessing a published paper is expensive, painstakingly slow and inconvenient process which is obviously great for the journals that have a monopoly over it, but less so for everyone else. All being justified by the need of peer-review (which is not even sure to be working alright, because (1) we all know the quality of the vast number of studies in some social field is shit, and (2) you can even publish a completely bullshit paper in a very reputable journal if you try hard enough).
2. Much of the research is done in the academia (because, well, that's supposedly the reason it exists), but, paradoxically, the main moving force of the progress isn't really interested in progress. It is a competitive environment, so everybody is interested in publishing a paper which will be recognized as an "important one" later on, so the. They are not interested in anyone learning anything and especially not interested in making their rivals (and that is what any other researcher is, a rival) able to make a follow-up contribution easily.
3. Hence, many important studies by reputable scientists lack quality, which everyone knows, including the scientists. Papers are written in a passive-agressive tone, in particularly confusing terms an inconvenient format, they often lack the source code and the source data, a minor one-sentence contribution to the current state-of-the-art is spread around 20 pages which you have to read fully to understand what exactly the contribution is.
4. To summarize: everyone is interested in publicity of their work, nobody is interested in improving their work because it provides them the high ground, but everyone (including even those outside of the academia) is interested in improving others work, since it helps them to learn. Improvements on the others work, obviously, are not shared unless it can be framed as a follow-up study.
How (I think) it might be improved. Assume a platform everyone uses to share their work, let's call it arxiv2. arxiv2 has a ranking system, which pushes the papers it "thinks" are of the better quality to the top. "Quality" is (maybe not exhaustively, but that's the main idea) a set of domain-specific flags which are considered to be beneficial for a study to be conclusive in this field, like "has a source code", "has a source data", "has a benchmark/accuracy/specificity/precision metrics", "summary/abstract succinctly captures the main result". Depending on the field, features may be more complicated in order to address some common problems like p-hacking or non-representative sampling in the study. But all the metrics have to be concrete. Nobody is required to include anything, but if they don't, and if others mark the work as lacking some features, it has less chance to get to the top. Everyone can vote for the paper, some minimal moderation may take place. Disputes may be solved similarly to how they are solved on wikipedia/stackoverflow (which are, admittedly, awful and frustrating, but order of magnitudes better than the status-quo in the research).
>Greater availability of data and code makes for better science.
Lack of availability of data and code should be an immediate disqualifier from publication for research. In any field. There is no substantial difference between withholding source code or data and withholding experimental procedures or list of materials tested. I can only think that the people who withhold such things are either actively malicious or are completely ignorant of the stupendous danger of cutting corners in science. Given the history we have to work with, it's pretty clear that the distance from cutting a corner in a way that seems harmless to the deaths of millions of people is very short. This is not something people should trifle with.
The dirty secret of ML research is almost all of it is shit (much worse than Sturgeon's law), and something like KNN with minor tweaks will almost always win.
I'm out in the trenches trying to turn ding dong papers into tools that does something useful, and the "muh mape 50% better" papers are effectively always something that should be ignored. Actually better algorithms and implementations (something like gradient boost machines/xgboost) are rare enough attempting to find them in the literature is futile.
> For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques.
Even if these models are outperformed, would they be useful anyway, to serve as insights into the functioning of an agent that can deduce solutions to these tasks "from scratch?"
After all, the human brain is a wondrous AI, but it is often also outperformed by simple heuristics.
It's outperformed at basic arithmetic by fairly simple electronic circuits too.
Which is probably your point :P
Right tool for the right job. The problem is that when you have a hammer called machine learning, every problem looks like a machine-learning-sized nail.
Eh, this isn't really surprising. The purpose of research is to explore- there are no guarantees that newer, complex methods will be better than simpler methods.
I think the greater issue is that researchers are afraid to admit that their results aren't great. And I don't blame them- it could impact funding, access to journals, etc. But mediocre results are still important for research, and it's a shame the scientific community doesn't recognize that.
> And I don't blame them- it could impact funding, access to journals, etc.
No, being "understanding" about this makes you part of the problem.
Part of research is trying new things, and exploring, but you should always be checking this against simpler techniques and currently accepted best performance. If it isn't an improvement, you don't publish at all unless you have something interesting to say about why it isn't improving things. Or perhaps in a methods survey.
One thing that happens when a technique like deep learning gets popular and things like python toolkits pop up making it easy to try, is that you get researchers in adjacent areas who don't really understand what they are doing trying to apply it to their favorite problem domain, and then comparing almost exclusively to other naive groups doing the same thing. It can be hard to tell if there is anything interesting there, even when there are dozens of conference papers, etc.
Basically the same thing happened with kernel methods when SVMs were hot, to a smaller scale.
Compare this to say, AlexNet. The reason that was immediately obvious something interesting was happening there was the fact that lots of people who did know what they were doing in models, had tried lots of other approaches, and you could make direct comparisons.
So yes, blame them. I do think negative results should be valued higher, but the fact you did some work doesn't make it publishable.
Frame it another way, if you give me a paper proposing a complex model and I grab your data an play with it a bit and it turns out linear regression with appropriate pre treatment works better ... well then I'm forced to believe you are either incompetent or being dishonest. Or, if students, your supervisors are.
This generalizes well. You should always be comparing against a known (or considered) good solution on your data under the same conditions, not just comparing against last years conference paper and variants you are trying to "improve". The right choice of baseline comparison will depend a bit on the domain, but not including it at all is shockingly poor scholarship.
I've even seen paper submissions with no direct comparisons at all, because the "researchers" didn't have access to the comparators data, and were too lazy to implement the other methods. Which leads to another sloppiness - methods that get pulled into comparison not because they are the right choice, but because there is a publicly available implementation. In the best case this forms a useful baseline. In the worst case, well, I guess it's good for your citation count if you implemented it :)
Whether an approach is currently better than other well understood approaches shouldn't be the focus of researchers. It might be the focus of private sphere engineers, but researchers should be interested in whether a new approach has potential to advance machine learning.
I don't mean to suggest that papers should boil down to performance comparisons with baseline results, not at all.
What I'm saying is that if you don't do this comparison somewhere, it can be very hard to tell what your numeric results do mean. In the worse case you see people offering numerical comparisons to other approaches that are similarly unpinned, and you can't tell if they are interesting even if they are apples to apples.
As a researcher, you are being lax if you never do that work if for nothing else than a sanity check on your implementation. If you've done it, it's good to include in the paper as a point of reference. If you currently aren't performing better than baseline, that's fine - but you should understand why and discuss that with insight too.
So it isn't the focus. But it is table stakes that you understand this stuff.
There are plenty of valid reasons to be critical / skeptical of the data science / machine learning bubble, but the last few articles I've read about the research actually makes it sound not-too-bad. 7/18 releasing enough code to reproduce sure beats the rate at which other fields release code, even when the code is still critical to confirming the results. In fact it's almost more understandable since the code is a larger portion of the intellectual property that might be valuable to whoever is funding the research, if it's a private entity, or university staff who might still want to monetize the research. I saw another one recently about reproducibility, something like 85%. Which sounds terrible - until you remember that something like 90% [1] of science papers weren't successfully reproduced in an attempt to do so. I'm gonna go dig up citations if I can find them quickly...
Yeah, but we'd expect papers that predominantly run code to be much, much better than this. Like, there is definitely a set of code that produced the results, and presumably a repository somewhere with some of the scripts that did this. Supplying this to the world is not too much to ask (and should 100% be a requirement of funding agencies (which would work as long as users of data and code were also required to cite the original article (isn't nesting parentheses fun?))).
I agree, but all of that is also true of a lot of papers in a lot of other fields. I've seen papers about better complex simulations of natural processes - I don't think I've ever seen one with complete code. I've seen papers that described the experiment and the results of their analysis with hand-wavy descriptions of which algorithms and statistical methods they used - but no code.
My point is that the findings of this article are not necessarily indicative of machine learning being junk science, but it's more likely indicative of a systemic problem with all research, where there's incentive to publish the paper, but not the data, code, models, or even generally enough information of any kind to successfully replicate the experiment more often than 10-15% of the time.
I remember when LIGO first detected a black-hole collision, they released the raw data and the Jupyter notebooks that were actually used to go from the raw data to the published results. Everyone was floored. It shouldn't be that amazing - it should be the standard. For every field.
To me this seems less understandable than the reproducibility crisis in psychology research, for at least in psychology research there is always the possibility that different samples can belegitimately different, and thus legitimately provide different results, but in machine learning, there are standardized training and testing samples, and the methods can be entirely encapsulated into immutable computer code. In this respect, the lack of reproducibility in machine learning papers suggests to me either distorting incentives or rampant unreported tweaking of parameters and/or falsification, or probably, I suspect, intentional obsfuscation of methods to retain value as IP in the private market. I'm not sure which or which mixture, but publishing code, and datasets if funded with public money should be made more manditory.
Interestingly --- and in my impression unlike psychology research --- I don't think this paper ever attempted to replicate and got contradictory results. Instead, I think the complaint is that for the majority of papers they were unable to begin the replication for lack of working code or available data sets. When they had these, they appear to have obtained the same results.
I guess this shows that the data really are what matter. Most ML methods can get signal that are reasonably approximate to each other. So are we hyper optimizing the methods? So my guess is that the data is the advantage. Having a lot of it and having exclusive access to it. Is it also time to move beyond derivative me too type publications?
There is plenty of work out now showing that we may have gone as far as we can with the 'throw more data' at it approach. Work out of the Adversarial AI area showing that models are learning shallow representations regardless of the amount of data used, work showing that even with deep learning we are hitting diminishing marginal returns with larger and larger corpus sizes, almost complete lack of progress in dealing with 'context'... I think that is a large part of the sudden resurgence of interest in ontologies, knowledge graphs, etc...
I agree with the context argument. But essentially with an ontology though we are adding more “data”. In this case it would be relationships between things. The hope is that implicit connections between other things not directly connected might be revealed by the AI.
7 were repoducible with considerable effort but only 1 clearly outperformed the baseline but still didn't outperform a well-tuned linear ranking method
--
Travis Ebesu et al. 2018. Collaborative Memory Network for Recommendation Systems. In Proceedings SIGIR ’18. 515–524.
was the only paper out 18 that was both reproducible and outperformed the baseline but did not consistently outperform a well-tuned non-neural linear ranking method
Reproducible with considerable effort but outperformed by comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques.
====================
They are:
kDD:
[17] Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S Yu. 2018. Leveraging meta- path based context for top-n recommendation with a neural co-attention model. In Proceedings KDD ’18. 1531–1540. [SIGIR]
[23] Xiaopeng Li and James She. 2017. Collaborative variational autoencoder for recommender systems. In Proceedings KDD ’17. 305–314.
[48] Hao Wang, Naiyan Wang, and Dit-Yan Yeung. 2015. Collaborative deep learning
for recommender systems. In Proceedings KDD ’15. 1235–1244.
RecSys:
[53] LeiZheng,Chun-TaLu,FeiJiang,JiaweiZhang,andPhilipS.Yu.2018.Spectral Collaborative Filtering. In Proceedings RecSys ’18. 311–319.
WWW:
[14] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings WWW ’17. 173–182.
[24] Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018. Variational Autoencoders for Collaborative Filtering. In Proceedings WWW ’18. 689–698.
Non-reproducible:
====================
KDD:
[43] Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Multi-Pointer Co-Attention
Networks for Recommendation. In Proceedings SIGKDD ’18. 2309–2318.
RecSys:
[41] Zhu Sun, Jie Yang, Jie Zhang, Alessandro Bozzon, Long-Kai Huang, and Chi Xu.
2018. Recurrent Knowledge Graph Embedding for Effective Recommendation.
In Proceedings RecSys ’18. 297–305.
[6] Homanga Bharadhwaj, Homin Park, and Brian Y. Lim. 2018. RecGAN: Recurrent Generative Adversarial Networks for Recommendation Systems. In Proceedings RecSys ’18. 372–376.
[38] Noveen Sachdeva, Kartik Gupta, and Vikram Pudi. 2018. Attentive Neural Archi- tecture Incorporating Song Features for Music Recommendation. In Proceedings RecSys ’18. 417–421.
[44] Trinh Xuan Tuan and Tu Minh Phuong. 2017. 3D Convolutional Networks for Session-based Recommendation with Content Features. In Proceedings RecSys
’17. 138–146.
[21] Donghyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, and Hwanjo Yu. 2016. Convolutional Matrix Factorization for Document Context-Aware Recom- mendation. In Proceedings RecSys ’16. 233–240.
[45] Flavian Vasile, Elena Smirnova, and Alexis Conneau. 2016. Meta-Prod2Vec: Prod-
uct Embeddings Using Side-Information for Recommendation. In Proceedings
RecSys ’16. 225–232.
SIGIR:
[32] Jarana Manotumruksa, Craig Macdonald, and Iadh Ounis. 2018. A Contextual Attention Recurrent Architecture for Context-Aware Venue Recommendation. In Proceedings SIGIR ’18. 555–564.
[7] Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat- Seng Chua. 2017. Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In Proceedings SIGIR ’17. 335–344.
WWW:
[42] Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Latent relational metric learn-
ing via memory-based attention for collaborative ranking. In Proceedings WWW
’18. 729–739.
[11] Ali Mamdouh Elkahky, Yang Song, and Xiaodong He. 2015. A multi-view deep learning approach for cross domain user modeling in recommendation systems. In Proceedings WWW ’15. 278–288.
I did not see any author names there that I recognized.
I would imagine most people in academia know that most papers published are published "just to publish" to help one's academic career, mostly by MSc and PhD students. They are essentially "write-only" papers. Everyone has to start somewhere.
Now, if we saw something like this out of DeepMind or OpenAI or in Nature, that would be worrying.
In my opinion, the most provocative point that this paper makes isn't just about general reproducibility issues or problems with comparing to a weak baseline — it’s that a number of these papers used improper methods to obtain their results in the first place.
For instance the NCF and MCRec papers tuned model parameters on the test set and the SpectralCF paper used a non-randomly sampled test set for evaluation.
That to me is even more surprising than their revelations that a well-tuned statistical baseline outperforms these models.
83 comments
[ 3.5 ms ] story [ 148 ms ] threadOr in it, either. ML researchers are painfully aware of how the sausage gets made. (About 12 hours before the NIPS deadline, typically.)
Facebook -> make the user see more ads
Amazon -> make the user buy more things
You probably want to optimise for keeping people on your service, first (and last).
https://arxiv.org/abs/1907.07355
"Probing Neural Network Comprehension of Natural Language Arguments
Timothy Niven, Hung-Yu Kao (Submitted on 17 Jul 2019)
We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average untrained human baseline. However, we show that this result is entirely accounted for by exploitation of spurious statistical cues in the dataset. We analyze the nature of these cues and demonstrate that a range of models all exploit them. This analysis informs the construction of an adversarial dataset on which all models achieve random accuracy. Our adversarial dataset provides a more robust assessment of argument comprehension and should be adopted as the standard in future work."
In the field of computer vision, there are similar suspicions that state of the art computer vision models are overfit to ImageNet datasets and the like. The issue being that even average research labs do not have hundreds of thousands of dollars to reproduce extremely expensive and highly tuned models. Reproducing and advancing the field of deep learning is quickly becoming inaccessible to almost everyone except for a few of the highest funded industrial labs (Google, FB, OpenAI, Microsoft Research among a few others). This is not all negative, it's just that not everything can be taken as gospel.
I'm an outsider to NLP, but have been very surprised by the recent improvements, so would be interested in anything that covers their limitations and newer datasets which help us to get over them.
Have a look at the BERT & XLNet papers, they have results on a wide range of datasets, including SQuAD 2.0.
I think it's worth keeping in mind that all of the headlines about "surpases human performance" are 100% bullshit regardless of whether they are about vision, ASR or NLP. Performance on a dataset is not the same thing as real world usefulness.
But this takes nothing away from the fact that ELMo/BERT/XLNet make it far easier to pull a model off the shelf and fine tune it for your task and get good performance.
And the gains here are so broad and not task specific that issues with individual datasets are not really an issue, it is the massive gains in pretraining that are the real story, and if that even helps you overfit a test set, it shows that they now have enough built-in linguistic knowledge that we previously didn't have to even overfit a bad test set.
This is obviously bad news for you if you thought that these tasks were basically solved, but it's completely irrelevant if you were building systems since these methods all result in relative gains on whatever language task they were pointed at.
The bigger issue IMO with many recent NLP papers is not that they are expensive to train from scratch, which is a one time cost, but that they are expensive to even do inference with, which makes it hard to actually deploy them in real systems.
But if your question is "is recent progress in NLP illusory" I think the answer is clearly no since the gains were across a broad range of tasks, and frankly I've used BERT and it makes my application better, the main issue I have is the runtime issue.
Ha. I get a feeling there's alotta overpaid data scientists out there.
This struck me as completely terrifying... it's not like the set of HVAC units was changing out every week. And one would hope that the historical data would help arrive at a general solution. My guess is they had some sort of church-meets-state problem in the test/train split, so that the short-term data allowed 'predicting' the test set.
Dude was quite convinced that they were making HVAC history, though.
Someone on the company board has just read a pop sci article about "machine learning" and now as a result the company has engaged some expensive consultants to advise us on "potential machine learning opportunities." The consultant spends maybe a day or two touring one of the company's plants and has only the most broad and high level view of how our industrial process actually works. The consultant then promises the world "machine learning is great, you can use it here, here and here for sure. It will save you millions..." It gets the board members excited.
So the company picks some toy problem and gives the consultant and his team a six month contract to deliver a proof of concept. The deadline passes a solution is delivered - it is underwhelming. The contract is not extended.
edit: All cynicism aside I appreciate the possibilities but for ML to succeed the developers really need to be embedded/integrated into the company have a good understanding of problem space and fundamentals. Not really something you can outsource easily and expect to succeed.
Does this mean that the published results are garbage? No! But what does it mean?
First, don't miss the main claim: less than half the published papers have sufficient code and data available for replication. While one might argue exactly what the standards for availability should be, if journals and reviewers were to demand it, this number can definitely be improved. Greater availability of data and code makes for better science.
Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication. They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement. Unfortunately, they are probably right.
The useful change (in my opinion) would be to change the standards by which papers are judged to be publishable. Reviewers should push back against weak baselines, but be more accepting of results that make modest claims. You can still have a useful paper even if the degree of improvement is small --- or even nonexistent. Maybe it will inspire someone else, maybe it's useful in an ensemble, or maybe it applies to a case where the baseline wouldn't. It's the puffery that's the problem, not the publication. Creating a system that rewards authors who more honestly evaluate their work would be an overall win.
It makes pulling out the actual content of the papers such a slog.
[0] exhibit A: https://ieeexplore.ieee.org/document/1017616
Source: am former producer/publisher of abomination research code.
Edit: Perhaps one way to encourage this is via topic "reuse" papers, which present further detail on a previously published papers topic, but this time with code and more detailed discussion. It would at least help the situations in that the author gets some reuse of effort, but in a way that still shows new information and advances the field.
distill.pub has a journal that strongly encourages interactive articles: https://distill.pub/journal/
It's a lot of work, tho:
> One good test for whether your article is a fit for Distill is whether your collaborators and you are willing to put in whatever time is necessary to write and illustrate an outstanding article. In our experience, this often takes 100+ hours.
Literate programming suffers from this in general because you don't want to clutter up your document with noise about versions and so forth - even as appendices, including package information and build instructions is a lot of noise.
Requiring that papers making claims about some code base publish that code base simultaneously with the paper via the peer review process should be sufficient to improve the overall state of affairs.
Don't underestimate the value of demos and competitions with companion papers either, those tend to get a lot more notice.
In general, you'll have a lot of approaches that don't work out which are nice to have a record of, but definitely don't merit being in the final paper.
If your claims don't depend on specific behaviours of some body of code, you wouldn't need it - eg, an article claiming some asymptotic performance of an algorithm should describe the algorithm in the abstract s.t. the performance bound can be proven, not some specific language's implementation of the algorithm.
But for most journals, it's the story as usualy. Personally I dont feel very confident about a paper result if they don't publish the code. Why hold back on the code?
Months of tweaking code is exactly what you want some test coverage for - so that you end up with a tweaked version of something where you still know what it does instead of something that does god knows what.
http://matt.might.net/articles/crapl
> The CRAPL is an open source "license" for academics that encourages code-sharing, regardless of how much how much Red Bull and coffee went into its production.
I know you know this, but this is absolutely a feature.
The big question is why this failure to publish code is tolerated by program+steering committees.
Also, more likely and non- nefarious, but no less harmful, there's just an academic culture that says "this is what a paper looks like and this is what it contains". Moving from this pattern increases the odds you'll not be published, and it would also likely result in a scandal when someone uses it to point out that the peers in peer review are neither peers (they can't read nor understand nor test the code) nor reviewing (they won't actually put resources into code review).
As a non-wealthy, non-academic, this isn't a battle I'm interested in fighting directly. There are other hills I'd rather die on :)
Besides, _ideally_, machine learning reseach is not about building a system but about making a theoretical claim that is supported by empirical results. The main claim should be about a property of the algorithm: its time complexity, its convergence, its correctness, etc. Experiments should directly test the theoretical claim and verify it in practice.
... ideally. In practice most machine learning papers are vehicles for a little table that compares your system with a baseline that you chose based on the fact that your system beats it (or you wouldn't have put it in the paper). Whether your experiments prove anything beyond the fact that you beat some benchmark is not really important.
And so whether someone publishes a reproducible paper is of no consequence because the only point of papers is to pass review and convince someone that the researchers should stay in the (research) money. In any case, things move so fast that most of the time nobody wants to reproduce your old and unsexy paper from last week.
Then some busybody comes along and points out the fact that you're wearing no clothes. Like in the paper above.
The paragraphs of English text help them to be able to claim that they are indeed the authors of some novel method (exactly what the author wants).
Code/data accessibility helps others to both find errors/imperfections in their approach and to build upon their results (exactly what the author doesn't want).
In some cases I basically had to run a test harness and brute force the parameters they'd used, because the algorithm was basically useless without picking parameters in the right range.
It was hard to accept then (~2008-ish), but now we're at a point where frankly at least for the more basic CS papers we ought to be able to expect source and some runnable format (e.g. a VM image) with test data.
I handed in a DVD with gigabytes of test data, full source, built binaries, and a test harness for the work I did for my thesis. All it took was to follow practices that I needed to follow to satisfy myself that my results were valid anyway.
I realize that for some work that relies on much larger and/or proprietary datasets making everything available would be hard, but a small test case illustrating the code run with the right parameters etc. ought to be a minimum unless there are substantial barriers, in which case the paper ought to signpost clearly what is missing and why.
https://www.youtube.com/watch?v=42QuXLucH3Q
“What gets me is the thought that even trying our best to figure out what's
true, using our most sophisticated and rigorous mathematical tools: peer review,
and the standards of practice, we still get it wrong so often; so how frequently
do we delude ourselves when we're not using the scientific method? As flawed as
our science may be, it is far away more reliable than any other way of knowing
that we have.”
I fear this contradicts your assertion that published results are not garbage. If researchers are claiming their systems are strong based on the fact that they beat weak baselines, then their claims are rubbish and their scholarship shoddy. If, on top of that, their work is difficult to reproduce, or replicate (by the distinction in the paper) then the published results are beyond garbage- they are meaningles garbage, garbage that fails to prove even its weak and insignificant claim to beat a weak baseline.
Basically, this is exactly the kind of shoddy scholarship that Backus accused social scientists of, in his infamous speech about "Cargo-Cult Science". In that speech, he points out that basing new research on previous weak research can only lead in the proliferation of weak results (my interpretation of his words, from memory).
This seems very much to be the case in machine learning research today. This article only provides further evidence that there is something rotten to the core in machine learning research.
>> Reviewers should push back against weak baselines, but be more accepting of results that make modest claims.
Agreed about the second part, however, the paper in the OP says that the way things are, it is very difficult to know which baselines are strong. My interpretation: because they're all trying to beat weak baselines that became established by beating earlier weak baselines. It's weak baselines all the way down.
Also, sorry HN, for the controversial comment (judging by the rollercoaster vote activity). I hope it adds something to the discussion despite the tone and I'll remember to keep the tone down in the future (but I keep saying that and I always end up giving in to the temptation to use more colourful language. I suck and I'm sorry).
What do we have now:
1. We have a common platform, which is a de-facto standard for sharing (somewhat math-related) papers: arxiv.org. Basically, anyone can upload anything to arxiv.org, there's not much of quality review required. Which, all in all, I think is great. I mean, it's orders of magnitude better than other fields, where both publishing and accessing a published paper is expensive, painstakingly slow and inconvenient process which is obviously great for the journals that have a monopoly over it, but less so for everyone else. All being justified by the need of peer-review (which is not even sure to be working alright, because (1) we all know the quality of the vast number of studies in some social field is shit, and (2) you can even publish a completely bullshit paper in a very reputable journal if you try hard enough).
2. Much of the research is done in the academia (because, well, that's supposedly the reason it exists), but, paradoxically, the main moving force of the progress isn't really interested in progress. It is a competitive environment, so everybody is interested in publishing a paper which will be recognized as an "important one" later on, so the. They are not interested in anyone learning anything and especially not interested in making their rivals (and that is what any other researcher is, a rival) able to make a follow-up contribution easily.
3. Hence, many important studies by reputable scientists lack quality, which everyone knows, including the scientists. Papers are written in a passive-agressive tone, in particularly confusing terms an inconvenient format, they often lack the source code and the source data, a minor one-sentence contribution to the current state-of-the-art is spread around 20 pages which you have to read fully to understand what exactly the contribution is.
4. To summarize: everyone is interested in publicity of their work, nobody is interested in improving their work because it provides them the high ground, but everyone (including even those outside of the academia) is interested in improving others work, since it helps them to learn. Improvements on the others work, obviously, are not shared unless it can be framed as a follow-up study.
How (I think) it might be improved. Assume a platform everyone uses to share their work, let's call it arxiv2. arxiv2 has a ranking system, which pushes the papers it "thinks" are of the better quality to the top. "Quality" is (maybe not exhaustively, but that's the main idea) a set of domain-specific flags which are considered to be beneficial for a study to be conclusive in this field, like "has a source code", "has a source data", "has a benchmark/accuracy/specificity/precision metrics", "summary/abstract succinctly captures the main result". Depending on the field, features may be more complicated in order to address some common problems like p-hacking or non-representative sampling in the study. But all the metrics have to be concrete. Nobody is required to include anything, but if they don't, and if others mark the work as lacking some features, it has less chance to get to the top. Everyone can vote for the paper, some minimal moderation may take place. Disputes may be solved similarly to how they are solved on wikipedia/stackoverflow (which are, admittedly, awful and frustrating, but order of magnitudes better than the status-quo in the research).
Lack of availability of data and code should be an immediate disqualifier from publication for research. In any field. There is no substantial difference between withholding source code or data and withholding experimental procedures or list of materials tested. I can only think that the people who withhold such things are either actively malicious or are completely ignorant of the stupendous danger of cutting corners in science. Given the history we have to work with, it's pretty clear that the distance from cutting a corner in a way that seems harmless to the deaths of millions of people is very short. This is not something people should trifle with.
I'm out in the trenches trying to turn ding dong papers into tools that does something useful, and the "muh mape 50% better" papers are effectively always something that should be ignored. Actually better algorithms and implementations (something like gradient boost machines/xgboost) are rare enough attempting to find them in the literature is futile.
Even if these models are outperformed, would they be useful anyway, to serve as insights into the functioning of an agent that can deduce solutions to these tasks "from scratch?"
After all, the human brain is a wondrous AI, but it is often also outperformed by simple heuristics.
Which is probably your point :P
Right tool for the right job. The problem is that when you have a hammer called machine learning, every problem looks like a machine-learning-sized nail.
I think the greater issue is that researchers are afraid to admit that their results aren't great. And I don't blame them- it could impact funding, access to journals, etc. But mediocre results are still important for research, and it's a shame the scientific community doesn't recognize that.
No, being "understanding" about this makes you part of the problem.
Part of research is trying new things, and exploring, but you should always be checking this against simpler techniques and currently accepted best performance. If it isn't an improvement, you don't publish at all unless you have something interesting to say about why it isn't improving things. Or perhaps in a methods survey.
One thing that happens when a technique like deep learning gets popular and things like python toolkits pop up making it easy to try, is that you get researchers in adjacent areas who don't really understand what they are doing trying to apply it to their favorite problem domain, and then comparing almost exclusively to other naive groups doing the same thing. It can be hard to tell if there is anything interesting there, even when there are dozens of conference papers, etc.
Basically the same thing happened with kernel methods when SVMs were hot, to a smaller scale.
Compare this to say, AlexNet. The reason that was immediately obvious something interesting was happening there was the fact that lots of people who did know what they were doing in models, had tried lots of other approaches, and you could make direct comparisons.
So yes, blame them. I do think negative results should be valued higher, but the fact you did some work doesn't make it publishable.
Frame it another way, if you give me a paper proposing a complex model and I grab your data an play with it a bit and it turns out linear regression with appropriate pre treatment works better ... well then I'm forced to believe you are either incompetent or being dishonest. Or, if students, your supervisors are.
This generalizes well. You should always be comparing against a known (or considered) good solution on your data under the same conditions, not just comparing against last years conference paper and variants you are trying to "improve". The right choice of baseline comparison will depend a bit on the domain, but not including it at all is shockingly poor scholarship.
I've even seen paper submissions with no direct comparisons at all, because the "researchers" didn't have access to the comparators data, and were too lazy to implement the other methods. Which leads to another sloppiness - methods that get pulled into comparison not because they are the right choice, but because there is a publicly available implementation. In the best case this forms a useful baseline. In the worst case, well, I guess it's good for your citation count if you implemented it :)
Research (noun): "the systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions."
Q: How can one reach new conclusions without being aware of existing approaches and conclusions?
I don't mean to suggest that papers should boil down to performance comparisons with baseline results, not at all.
What I'm saying is that if you don't do this comparison somewhere, it can be very hard to tell what your numeric results do mean. In the worse case you see people offering numerical comparisons to other approaches that are similarly unpinned, and you can't tell if they are interesting even if they are apples to apples.
As a researcher, you are being lax if you never do that work if for nothing else than a sanity check on your implementation. If you've done it, it's good to include in the paper as a point of reference. If you currently aren't performing better than baseline, that's fine - but you should understand why and discuss that with insight too.
So it isn't the focus. But it is table stakes that you understand this stuff.
...so, we are in agreement then?
[1] 1 quick citation before I get back to work: https://www.bewellbuzz.com/technology/many-published-studies...
It's terrible, but no more terrible than many other comparable fields.
My point is that the findings of this article are not necessarily indicative of machine learning being junk science, but it's more likely indicative of a systemic problem with all research, where there's incentive to publish the paper, but not the data, code, models, or even generally enough information of any kind to successfully replicate the experiment more often than 10-15% of the time.
I remember when LIGO first detected a black-hole collision, they released the raw data and the Jupyter notebooks that were actually used to go from the raw data to the published results. Everyone was floored. It shouldn't be that amazing - it should be the standard. For every field.
18 papers/algorithms were tested
11 weren't reproducible
7 were repoducible with considerable effort but only 1 clearly outperformed the baseline but still didn't outperform a well-tuned linear ranking method
--
Travis Ebesu et al. 2018. Collaborative Memory Network for Recommendation Systems. In Proceedings SIGIR ’18. 515–524. was the only paper out 18 that was both reproducible and outperformed the baseline but did not consistently outperform a well-tuned non-neural linear ranking method
Reproducible with considerable effort but outperformed by comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques.
====================
They are:
kDD:
[17] Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S Yu. 2018. Leveraging meta- path based context for top-n recommendation with a neural co-attention model. In Proceedings KDD ’18. 1531–1540. [SIGIR]
[23] Xiaopeng Li and James She. 2017. Collaborative variational autoencoder for recommender systems. In Proceedings KDD ’17. 305–314.
[48] Hao Wang, Naiyan Wang, and Dit-Yan Yeung. 2015. Collaborative deep learning for recommender systems. In Proceedings KDD ’15. 1235–1244.
RecSys:
[53] LeiZheng,Chun-TaLu,FeiJiang,JiaweiZhang,andPhilipS.Yu.2018.Spectral Collaborative Filtering. In Proceedings RecSys ’18. 311–319.
WWW:
[14] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings WWW ’17. 173–182.
[24] Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018. Variational Autoencoders for Collaborative Filtering. In Proceedings WWW ’18. 689–698.
Non-reproducible:
====================
KDD:
[43] Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Multi-Pointer Co-Attention Networks for Recommendation. In Proceedings SIGKDD ’18. 2309–2318.
RecSys:
[41] Zhu Sun, Jie Yang, Jie Zhang, Alessandro Bozzon, Long-Kai Huang, and Chi Xu. 2018. Recurrent Knowledge Graph Embedding for Effective Recommendation. In Proceedings RecSys ’18. 297–305.
[6] Homanga Bharadhwaj, Homin Park, and Brian Y. Lim. 2018. RecGAN: Recurrent Generative Adversarial Networks for Recommendation Systems. In Proceedings RecSys ’18. 372–376.
[38] Noveen Sachdeva, Kartik Gupta, and Vikram Pudi. 2018. Attentive Neural Archi- tecture Incorporating Song Features for Music Recommendation. In Proceedings RecSys ’18. 417–421.
[44] Trinh Xuan Tuan and Tu Minh Phuong. 2017. 3D Convolutional Networks for Session-based Recommendation with Content Features. In Proceedings RecSys ’17. 138–146.
[21] Donghyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, and Hwanjo Yu. 2016. Convolutional Matrix Factorization for Document Context-Aware Recom- mendation. In Proceedings RecSys ’16. 233–240.
[45] Flavian Vasile, Elena Smirnova, and Alexis Conneau. 2016. Meta-Prod2Vec: Prod- uct Embeddings Using Side-Information for Recommendation. In Proceedings RecSys ’16. 225–232.
SIGIR:
[32] Jarana Manotumruksa, Craig Macdonald, and Iadh Ounis. 2018. A Contextual Attention Recurrent Architecture for Context-Aware Venue Recommendation. In Proceedings SIGIR ’18. 555–564.
[7] Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat- Seng Chua. 2017. Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In Proceedings SIGIR ’17. 335–344.
WWW:
[42] Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Latent relational metric learn- ing via memory-based attention for collaborative ranking. In Proceedings WWW ’18. 729–739.
[11] Ali Mamdouh Elkahky, Yang Song, and Xiaodong He. 2015. A multi-view deep learning approach for cross domain user modeling in recommendation systems. In Proceedings WWW ’15. 278–288.
I would imagine most people in academia know that most papers published are published "just to publish" to help one's academic career, mostly by MSc and PhD students. They are essentially "write-only" papers. Everyone has to start somewhere.
Now, if we saw something like this out of DeepMind or OpenAI or in Nature, that would be worrying.
For instance the NCF and MCRec papers tuned model parameters on the test set and the SpectralCF paper used a non-randomly sampled test set for evaluation.
That to me is even more surprising than their revelations that a well-tuned statistical baseline outperforms these models.