> In the award ceramony [sic], the Bell Labs researcher presenting the award explicitly said he doesn't understand how InstaHide is secure, but, and I quote, “it works nonetheless”. No! It does not.
Bell Labs used to have a wonderful reputation. Not so much anymore, I suppose.
I think if one has paid attention to how many different entities Bell Labs has come under through mergers and acquisitions (and the fate of those parent companies), it's shocking why anyone would think the 1950s/1960s/1970s reputation should be applied to the present entity.
That's not to say that M&A automatically (or significantly dilutes) the value of the research arm of a commercial organization.
Rather, every few years, you have to re-evaluate what they're actually committed to.
He definitely doesn't mince words, but it's hard to be mad when he always makes very valid points. And I say that as a secondary author on one the criticized methods (not InstaHide).
Thoughtful criticism is like having good feedback. It should not be hard to receive that. It starts becoming hard when there are ulterior motives or when the criticism actually isn’t valid but getting a lot of attention. Most constructive feedback is actually best shared in private as well.
Shouldn’t be hard but still can be. It’s like misinterpreting signs of interest from someone until that conversation on the elevator ride down after work on a Friday. Thoughtful, private, polite, direct and final.
The article resonates with me a lot, and reminded of the couple of years of personal experience of applied machine learning in another field of science; because the points made are exactly what I encountered then and made me quit my job.
There is a lot of machine learning snake oil out there, even in the most prestigious journals. The issue is that most people wouldn't know about it until they actually try to reproduce those results (if they can at all in the first place).
It is a huge issue now that machine learning has made its way into every field of research, while almost nobody has adequate training to even gauge what is and isn't questionable. Even the grad students who end up doing all the "hard work" usually don't have much of an idea of what exactly they hacked together and how to access its validity.
The crazy thing is that, despite all of this, you can still get ahead if you have know just slightly more buzzwords than others and have a paper or two (don't have to be first author) to show for. At the end of the day, I suppose all of this is nothing new (machine learning practitioners are the new SEO consultants?), and I'm just venting my frustration for not being able to accept this is how things work.
> (machine learning practitioners are the new SEO consultants?)
If we want to continue the snake oil salesman analogy, SEO consultants were soothsayers, attempting to divine meaning from The Algorithm of Great Google.
Now the apothecaries have begun to legitimize their potion making into tonics, but there is still room for a charlatan or two to push hodgepodge concoctions onto the unsuspecting Product Manager or CTO.
> There is a lot of machine learning snake oil out there, even in the most prestigious journals.
I'm also deep in that industry, and can confirm this is the case. Even for research coming out of the biggest names: FB, Google, etc.
The problem of "story over substance" has been observed by many; it even has a name in the context of ML, the "Mummy Effect" [0].
Like you, I'm segueing out of "AI" because of the unfavourable signal-to-noise ratio. Frustrating indeed, but life is too short to try to outcompete pocketless BS generators, or clean up their mess on my dime.
Publish or perish - there are a lot of half baked ideas sold by researchers, particularly in fast advancing fields. As long as we as a society value output volume and showmanship over substance it continues. The best defense these days where distribution is instantaneous and global are critical readers and investments into education.
“Investment into education” translates into stupid politicians dumping more money into the status quo with no indication that this will better their own metrics (which are mostly irrelevant in the first place as they don’t teach any useful cognitive skills).
This isn't really much of a case of publish or perish - the two senior authors, to wit, are tenured, established, and very well off. They could each not write a paper for the rest of their careers and be fine. It's really a case of having a small idea - which, upon deeper examination, doesn't really work - and selling it way too hard. The (senior) authors should know better.
Well, that was a fairly comprehensive takedown. Kudos to the author for mentioning the things the primary author did well and correctly put the primary responsibility on the senior researchers.
ML privacy isn't at all in my wheelhouse, but I did do computer security research in school. The author makes a good point that researchers need to be ready for their methods to be broken. More importantly though, researchers have an ethical responsibility to publicize when a vulnerability in their work is discovered, so it's not used improperly.
I think this is something that's very clear in the security community, but isn't the norm in the machine learning community. Hopefully privacy researchers in ML take a few notes from their colleagues that have been doing computer security sooner rather than later.
Finally, it's pretty disappointing to see how far Bell Labs has fallen. I won't spoil the article, but the quote from the Bell Labs judge is telling.
Bell Labs has already fallen long before, see https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal (Though I only read about it earlier this year in the book Soft Machines.) Well, at least they fired him after it became clear what was going on, but pretty sad for it to go on under their noses for so long beforehand. Like this article there's a lingering, if a bit different, question of what should be expected of co-authors.
It's a small thing, but I'm really grateful that he very specifically aimed at the senior authors here. While I'm not sure that the first author can be completely absolved of responsibility (due to the simple fact that they are listed as an author), the power dynamics in academia are absurdly skewed and it's nice to see a tacit acknowledgement of that in an otherwise (justifiably) scathing critique.
i'm torn. I'd like to agree with the sentiment of the OP, but then when I surmise that the 1rst author will use this article as the basis for their job talk, and probably get something somewhere sunny and warm, I have second thoughts.
Sounds like the kind of show over substance and obfuscation to prevent review snake oil that all too often shows up in crypto-currency. At least in ML the author isn't likely to get swatted for being critical.
I'm not sure what it is about one-time-pads (and generalizations of OTP like shamir secret-sharing) that cause it to so reliably show up in unsound work. Something about these perfectly valid (in the right context) tools is very moth-to-flamey.
I thought it was perfectly polite and far more evenhanded than it needed to be. One could argue that the authors of the original paper and the bell labs judges were acting in a hostile fashion.
I think the post is frustrated with the attitude of authors who should really know better; Sanjeev Arora is a giant of theoretical computer science, and he should know to do better than this
The many spurious/misleading claims in this paper at least have some plausible deniability through vague language and complexity. Omitting a clear cut falsification of their methodology that they themselves put forward as a challenge is against basic scientific principles.
It seems this award would have awarded anything as 2nd place, what if there was nothing else better? Does the award really tell you how great this is, or maybe it simply shows how little progress has been made.
Edit: And I completely agree with author about Paper in the field need to do better.
Instead of downvotes, I'd appreciate being suggested alternative papers that same year that made more progress towards training data privacy in ML algorithms than the one mentioned in the article, and which should have won the prize instead.
Otherwise, it seems my assessment would hold. A ranking is relative, so doesn't tell you much of absolute progress towards the goal.
I misunderstood the title as being disappointed that they won second place (as opposed to first place), rather than being disappointed that they won an award at all.
I enjoyed the slight ambiguity of the title, like you I was expecting some whiny article about how someone should've gotten 1st but was pleasantly surprised by a thorough and well-written analysis instead.
It was a little bit shocking to me how obviously weak the image obfuscation performed is when you think about it.
Randomizing the sign is the same as eradicating 1 single bit out of however many were used to represent each sample. No matter how random it looks, there is no way removing 1/32 or 1/64 of the information in the image offers any actual security for something where samples are as highly correlated as in image data.
And using a non-cryptographic PRNG for security purposes... I'd really, really expect more from anybody working on privacy research.
This is a really interesting way of putting it. Is the presence of e.g. 31/32 of the original data the only reason InstaHide can train a model at all? I haven’t read the original paper, but it seems very possible that the entire approach runs into a fundamental tradeoff.
There's a book and UK newspaper column series called "bad science". It's not about people with good intentions failing.
> Ben Goldacre’s wise and witty bestseller, shortlisted for the Samuel Johnson Prize, lifts the lid on quack doctors, flaky statistics, scaremongering journalists and evil pharmaceutical corporations.
> Full spleen and satire, Ben Goldacre takes us on a hilarious, invigorating and ultimately alarming journey through the bad science we are fed daily by hacks and quacks.
Sure, I didn't mean to imply it was proof one way or the other, just that it's not a universal thing. I've actually never heard the term "quackery" but I do like it.
There's also "pseudoscience". Unfortunately all too common in ML, where the ratio of published papers to genuine new developments is (charitably) at least 2:1.
> There's this thing that machine learning research papers do where they propose Algorithm A, say in words it works, and then prove Theorem T. The problem is that Theorem T is often completely disjoint from whether or not Algorithm A works at all. Lipton and Steinhardt call these spurious theorems.
Is this unique to, or particularly common in, machine learning?
I was also confused by this. I guess that it might be equivalent to randomly flipping the top bit of the unsigned RGB components. The "absolute value" would always set the top bit to 1.
45 comments
[ 3.4 ms ] story [ 97.2 ms ] thread> In the award ceramony [sic], the Bell Labs researcher presenting the award explicitly said he doesn't understand how InstaHide is secure, but, and I quote, “it works nonetheless”. No! It does not.
Bell Labs used to have a wonderful reputation. Not so much anymore, I suppose.
That's not to say that M&A automatically (or significantly dilutes) the value of the research arm of a commercial organization.
Rather, every few years, you have to re-evaluate what they're actually committed to.
https://arxiv.org/pdf/2002.08347.pdf https://arxiv.org/pdf/1902.02322.pdf https://arxiv.org/pdf/1804.03286.pdf
He definitely doesn't mince words, but it's hard to be mad when he always makes very valid points. And I say that as a secondary author on one the criticized methods (not InstaHide).
There is a lot of machine learning snake oil out there, even in the most prestigious journals. The issue is that most people wouldn't know about it until they actually try to reproduce those results (if they can at all in the first place).
It is a huge issue now that machine learning has made its way into every field of research, while almost nobody has adequate training to even gauge what is and isn't questionable. Even the grad students who end up doing all the "hard work" usually don't have much of an idea of what exactly they hacked together and how to access its validity.
The crazy thing is that, despite all of this, you can still get ahead if you have know just slightly more buzzwords than others and have a paper or two (don't have to be first author) to show for. At the end of the day, I suppose all of this is nothing new (machine learning practitioners are the new SEO consultants?), and I'm just venting my frustration for not being able to accept this is how things work.
If we want to continue the snake oil salesman analogy, SEO consultants were soothsayers, attempting to divine meaning from The Algorithm of Great Google.
Now the apothecaries have begun to legitimize their potion making into tonics, but there is still room for a charlatan or two to push hodgepodge concoctions onto the unsuspecting Product Manager or CTO.
I'm also deep in that industry, and can confirm this is the case. Even for research coming out of the biggest names: FB, Google, etc.
The problem of "story over substance" has been observed by many; it even has a name in the context of ML, the "Mummy Effect" [0].
Like you, I'm segueing out of "AI" because of the unfavourable signal-to-noise ratio. Frustrating indeed, but life is too short to try to outcompete pocketless BS generators, or clean up their mess on my dime.
[0] https://rare-technologies.com/mummy-effect-bridging-gap-betw...
ML privacy isn't at all in my wheelhouse, but I did do computer security research in school. The author makes a good point that researchers need to be ready for their methods to be broken. More importantly though, researchers have an ethical responsibility to publicize when a vulnerability in their work is discovered, so it's not used improperly.
I think this is something that's very clear in the security community, but isn't the norm in the machine learning community. Hopefully privacy researchers in ML take a few notes from their colleagues that have been doing computer security sooner rather than later.
Finally, it's pretty disappointing to see how far Bell Labs has fallen. I won't spoil the article, but the quote from the Bell Labs judge is telling.
It's a small thing, but I'm really grateful that he very specifically aimed at the senior authors here. While I'm not sure that the first author can be completely absolved of responsibility (due to the simple fact that they are listed as an author), the power dynamics in academia are absurdly skewed and it's nice to see a tacit acknowledgement of that in an otherwise (justifiably) scathing critique.
I'm not sure what it is about one-time-pads (and generalizations of OTP like shamir secret-sharing) that cause it to so reliably show up in unsound work. Something about these perfectly valid (in the right context) tools is very moth-to-flamey.
I think this is the most abhorrent aspect: it's not in the spirit of science to arbitrarily delay findings
Anyway, why delaying results should be considered the worst transgression? I think dishonesty is the biggest problem here.
Edit: And I completely agree with author about Paper in the field need to do better.
Otherwise, it seems my assessment would hold. A ranking is relative, so doesn't tell you much of absolute progress towards the goal.
Randomizing the sign is the same as eradicating 1 single bit out of however many were used to represent each sample. No matter how random it looks, there is no way removing 1/32 or 1/64 of the information in the image offers any actual security for something where samples are as highly correlated as in image data.
And using a non-cryptographic PRNG for security purposes... I'd really, really expect more from anybody working on privacy research.
Science was never meant to happen here, this is just plain exercise in dishonesty.
"Bad science" is a real thing which happens when somebody tries to "do science" but fails.
> Ben Goldacre’s wise and witty bestseller, shortlisted for the Samuel Johnson Prize, lifts the lid on quack doctors, flaky statistics, scaremongering journalists and evil pharmaceutical corporations.
> Full spleen and satire, Ben Goldacre takes us on a hilarious, invigorating and ultimately alarming journey through the bad science we are fed daily by hacks and quacks.
I find it useful to have separate terms for when somebody is incompetent (bad science) and when somebody intentionally misleads (quackery).
https://news.ycombinator.com/item?id=25275637
Is this unique to, or particularly common in, machine learning?
If so, that is scary - what on earth is going on?
https://twitter.com/prfsanjeevarora/status/13374473410585681...
http://web.archive.org/web/20201211174253/https://hazelsuko0...