It also has a weird emphasis on their personal lives - particularly their gender:
> Satrajit Chatterjee, a more senior researcher at Google, used the cover of scientific debate to undermine the women personally, the employees claim. They spoke anonymously because they were not authorized to discuss company matters. Multiple complaints about Chatterjee’s behavior toward the women were made to Google’s personnel department, and he received a written warning, some employees said, but he continued to criticize the women’s results.
I half expect it to carry on: "The male researchman directed his critical male gaze at the delicate ladylike handwritten-in-flowery-pink-biro output of the female women researcheresses.."
I mean, I know I'm out of step with the zeitgeist in finding this irrelevant and patronising, but come on. Female researchers deserve to be treated not as fragile novelties but as regular professionals. Implying that a male researcher launched some kind of inappropriate sexed attack by criticising their work is thoroughly bizarre to me.
> Implying that a male researcher launched some kind of inappropriate sexed attack by criticising their work is thoroughly bizarre to me.
They’re not implying it they are outright claiming that was his motivation for not letting it drop before he got fired.
Most reasonable people would just stop when told “dude, you’re gonna get fired if you keep it up” but some people just can’t do that for whatever reason. Either way they decided back in the 70s women don’t have to deal with harassment as part of their job and whatever your opinion on how this is enforced it is pure and simple fact you will be fired over a corporation going to the mat to defend you against accusations related to this. Well, unless you’re a CEO of one of the Silicon Valley unicorns then you can do whatever you want.
I think there are two separate threads running through this. One is his academic criticism of their work through the academic publishing process. I hope no one would describe that as 'harassment', irrespective of how much his academic output focused on challenging their work. The other is his extremely poor and possibly slightly autistic Slack behaviour, fixating on his disapproval of their work and repeatedly bringing it up in even the most marginally related contexts. That part I would agree rises, or at least arguably rises, to the level of harassment.
Honestly, I find it kind of wild that someone would spend a big chunk of their time at work trying to drag down someone else's work. Fine, if you really think that someone has done something egregiously fraudulent, then you probably should pursue it. But to spend so long working on a rebuttal to a paper which basically boils down to "Well your benchmarks aren't good enough" is pretty weak. I've spent a lot of time looking at research papers - the benchmarks are always weak.
Also, if you're actually interested in writing a paper on "What are the best techniques for place and route" then write a paper "Best Modern Place and Route algorithms" not "Here's why this particular paper my colleagues wrote is wrong".
Personally, I think they should've published his paper - they can continue actually using the work internally to improve their TPU designs whilst he publicly tries to persuade their competitors it's a bad idea.
That said, I've also seen plenty of competitive drama in FAANG research labs, so this story is not hard to believe. More senior engineers often will use their seniority to power-grab control of projects. It sounds like Google execs did the right thing in the end.
> Personally, I think they should've published his paper - they can continue actually using the work internally to improve their TPU designs whilst he publicly tries to persuade their competitors it's a bad idea.
This analysis assumes he is wrong. I’ve got no idea if he is right or wrong.
But if he is right, Google would probably want to suppress stories like this for PR purposes. “Google AI published in Nature is actually beaten by old commercial technology” is not the headline Google wants to put out.
Although again - I’m not saying that he is right! Just that Google has commercial reasons for what it decides to pursue/implement/publish.
If he's right they've got a problem because they're actively using the research in their chip designs. Personally, I think if this technique really didn't work they probably would've figured that out when taking it to production.
Well they said they implemented it in their “pipeline”, which could mean anything from they are using it lots to almost not using it at all. How often have we deployed features in our own software that are barely touched in practice?
Maybe I’m being too cynical about press releases though!
From a quick scan of the paper, it seems the issue is not that the RL technique doesn't work (the chips do function) but that the claim that RL is just a really bad and inefficient way to do it. Key paragraph:
"We find that RePlAce [non AI] produces 26% better wirelength than RL [reinforcement learning] whilst using 5 orders of magnitude less computation"
If that's true then you can see why he criticized the Nature paper. It's not a small difference in performance. The classical techniques are just crushing the AI in this paper.
I was privy to some discussion with people related to this line of research. In my layman understanding Chatterjee's contention was final placement of components in the accelerator chip was better done with some human aided intervention, after RL had generated a "mostly" complete layout.
Its not hard to imagine why he could be challenging the other paper - The RL program could possibly be generating placements based on optimization related to defined tasks (goals). The Nature paper shows a blurred out image of the same, where the components aren't symmetrical placed. However its understandable that interconnect lengths influence I/O which you want symmetrical from the similar components. That makes the placement more optimal for generalized tasks instead of specific ones.
Do we know if that is done? Maybe or maybe not. But it would be interesting to find out. Anyway this is something we'll better understand if Sat Chatterjee's preprint is released.
This sounds like a preemptive hit piece on a researcher who is now free to publish an article that is critical of Google researchers work and perhaps even shows hints of outright fraud.
Nothing is brought forth except to say that this person had persistent claims of issues in the work with no proof provided to disprove these issues, and Google was preventing scientific publishing of the critical work to the point of threatening to fire the person and they now actually did.
The genders of the people involved are irrelevant but are pushed as some argument as to why the criticism is invalid. I’ll look forwards to seem the critique being published so the claims can be weighed against each other honestly.
See my above comment - the choice of metrics in table 3.2 doesn't really make any sense.
Also, they revived some ancient 1998 IBM chip, and report on that, for no clear reason. They call it a 2004 benchmark, but what actually happened is that in 2004 someone made some synthetic variants and put them into a dataset. This is not some widely used benchmark in the field. Given their highly questionable choice of metrics, I would not be surprised if there was some serious cherry-picking on the benchmark as well.
It continues to boggle my mind that this guy chose to get fired over this paper, which is the dumbest paper I have read in a long time.
As an example, his paper reports *median* congestion! I literally laughed out loud when I saw that. I don't think a single other chip design paper in the world uses that metric, and for good reason.
For those of you who don't know chip design, congestion is a measure of how many wires will need to go in the same place when you do routing for the chip. When wires overlap, they have to go above / below each other, and your chip only has so many layers. If congestion is too high, it can simply be impossible to make the chip.
Congestion is therefore all about the extreme values. Why in the world would you report median congestion, then? Median ignores extreme values, which are the only ones you care about!
I even checked the paper he cites as being prior work that uses median congestion, to see who else did something so stupid, and... it doesn't! It uses average, which makes way more sense. I mean, you really care about the max, but at least average is sensitive to extremes.
I could go on, but the TLDR is, his paper is hot garbage, and I cannot say I am surprised Google didn't want its name associated with it.
17 comments
[ 3.6 ms ] story [ 48.2 ms ] thread> Satrajit Chatterjee, a more senior researcher at Google, used the cover of scientific debate to undermine the women personally, the employees claim. They spoke anonymously because they were not authorized to discuss company matters. Multiple complaints about Chatterjee’s behavior toward the women were made to Google’s personnel department, and he received a written warning, some employees said, but he continued to criticize the women’s results.
I half expect it to carry on: "The male researchman directed his critical male gaze at the delicate ladylike handwritten-in-flowery-pink-biro output of the female women researcheresses.."
I mean, I know I'm out of step with the zeitgeist in finding this irrelevant and patronising, but come on. Female researchers deserve to be treated not as fragile novelties but as regular professionals. Implying that a male researcher launched some kind of inappropriate sexed attack by criticising their work is thoroughly bizarre to me.
They’re not implying it they are outright claiming that was his motivation for not letting it drop before he got fired.
Most reasonable people would just stop when told “dude, you’re gonna get fired if you keep it up” but some people just can’t do that for whatever reason. Either way they decided back in the 70s women don’t have to deal with harassment as part of their job and whatever your opinion on how this is enforced it is pure and simple fact you will be fired over a corporation going to the mat to defend you against accusations related to this. Well, unless you’re a CEO of one of the Silicon Valley unicorns then you can do whatever you want.
Also, if you're actually interested in writing a paper on "What are the best techniques for place and route" then write a paper "Best Modern Place and Route algorithms" not "Here's why this particular paper my colleagues wrote is wrong".
Personally, I think they should've published his paper - they can continue actually using the work internally to improve their TPU designs whilst he publicly tries to persuade their competitors it's a bad idea.
We've seen time and again that various trends in ML turn out to have actually been dead-ends.
A few examples: https://arxiv.org/abs/2102.06356 https://arxiv.org/abs/1905.03678
That said, I've also seen plenty of competitive drama in FAANG research labs, so this story is not hard to believe. More senior engineers often will use their seniority to power-grab control of projects. It sounds like Google execs did the right thing in the end.
This analysis assumes he is wrong. I’ve got no idea if he is right or wrong.
But if he is right, Google would probably want to suppress stories like this for PR purposes. “Google AI published in Nature is actually beaten by old commercial technology” is not the headline Google wants to put out.
Although again - I’m not saying that he is right! Just that Google has commercial reasons for what it decides to pursue/implement/publish.
Maybe I’m being too cynical about press releases though!
"We find that RePlAce [non AI] produces 26% better wirelength than RL [reinforcement learning] whilst using 5 orders of magnitude less computation"
If that's true then you can see why he criticized the Nature paper. It's not a small difference in performance. The classical techniques are just crushing the AI in this paper.
Its not hard to imagine why he could be challenging the other paper - The RL program could possibly be generating placements based on optimization related to defined tasks (goals). The Nature paper shows a blurred out image of the same, where the components aren't symmetrical placed. However its understandable that interconnect lengths influence I/O which you want symmetrical from the similar components. That makes the placement more optimal for generalized tasks instead of specific ones.
Do we know if that is done? Maybe or maybe not. But it would be interesting to find out. Anyway this is something we'll better understand if Sat Chatterjee's preprint is released.
The article has a very "inside baseball" feel and I didn't learn much of anything from it. Turns out, Google has internal politics - surprise!
Nothing is brought forth except to say that this person had persistent claims of issues in the work with no proof provided to disprove these issues, and Google was preventing scientific publishing of the critical work to the point of threatening to fire the person and they now actually did.
The genders of the people involved are irrelevant but are pushed as some argument as to why the criticism is invalid. I’ll look forwards to seem the critique being published so the claims can be weighed against each other honestly.
http://47.190.89.225/pub/education/MLcontra.pdf
Key to the dispute is section 3.2 (results).
Also, they revived some ancient 1998 IBM chip, and report on that, for no clear reason. They call it a 2004 benchmark, but what actually happened is that in 2004 someone made some synthetic variants and put them into a dataset. This is not some widely used benchmark in the field. Given their highly questionable choice of metrics, I would not be surprised if there was some serious cherry-picking on the benchmark as well.
As an example, his paper reports *median* congestion! I literally laughed out loud when I saw that. I don't think a single other chip design paper in the world uses that metric, and for good reason.
For those of you who don't know chip design, congestion is a measure of how many wires will need to go in the same place when you do routing for the chip. When wires overlap, they have to go above / below each other, and your chip only has so many layers. If congestion is too high, it can simply be impossible to make the chip.
Congestion is therefore all about the extreme values. Why in the world would you report median congestion, then? Median ignores extreme values, which are the only ones you care about!
I even checked the paper he cites as being prior work that uses median congestion, to see who else did something so stupid, and... it doesn't! It uses average, which makes way more sense. I mean, you really care about the max, but at least average is sensitive to extremes.
I could go on, but the TLDR is, his paper is hot garbage, and I cannot say I am surprised Google didn't want its name associated with it.