"Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting."
I get it, I do, and I'm a good boy around here as much as humanly possible, but this ain't no ordinary level of back-button-evil. There were hundreds of back-button edits. It was quite uncommon and interesting, IMO.
The question is, at what point does user-hostile site behavior become more than tangential? Surely if, for example, a page contained malware that would warrant at least a mention in the comments. Personally I appreciated your warning.
True, I did not interact with the page yet (did not scroll, klick or else and uBlock origin activated) and have as of this point 6 back entries in my Firefox-Tab for the page.
Yep. When that happens without users flagging the comment it doesn't get a [flagged] tag. It's unfortunate, because you can end up temporarily muted from commenting with a history full of nothing but positive-scored unflagged comments and have no idea how you angered the powers-that-be.
It seems to be quite the contrary to what you state: I hear from female colleagues (professors at CMU no less) all the time that their work is subjected to much greater scrutiny and harsher criticism than work by their male counterparts. To withstand a much greater volume and abrasiveness of attacks, work by women has to be higher quality and nearly unassailable. That the article might have amped up a sexism angle for buzz is one issue, but the underlying dynamic frankly isn't talked about enough. Was it the case here? Hard to know.
> I'll assume from this sentence they are imposter women who shouldn't be in computer science.
I honestly don't know if you meant that ironically or sarcastically, but holy flippety bippity, we need no such attitudes and certainly no such comments to mess up the culture even more. At all. If you think it's funny or insightful, it isn't.
That's what happens if you allow positive discrimination. People no longer trust the system since someone might have come into a position without being capable. People are forced to start scrutinizing members of the positively discriminated group more.
Having been a part of many hiring processes, positive discrimination is a complete mirage. Hiring unqualified, below-par candidates for diversity is something that I have never seen happen. Done right, hiring for diversity means better outreach and actively fighting cognitive biases (ironically) like the one you're alluding to, and the OP has gone all-in on. It's an utter fallacy that minority groups are imposters by default.
It's not that members of positively discriminated groups(you use minorities here but thats completely wrong, asians for example are a minority but are strongly discriminated against) are imposters by default. It's just that the probability is significantly higher.
On the other side of the coin: If you see an asian stem person for example you know for sure they were the cream of the crop since they had everything against them.
This both reinforces stereotypes held by society and in the long run I expect it to only make things worse.
The outcomes of positive discrimination are positively toxic. Storytime: just yesterday a candidate from a visibly marginalized group gave a research talk, and I caught myself thinking "holy shit this isn't a political candidate". It's easy to arrive at this dark spot when you have seen the media promote one political candidate too much.
Sure it is! If you think something substantially different solely because of someone's appearance or group status, then that's the very definition of a cognitive bias. If that makes you uncomfortable to even consider that possibility, then by all means, keep reinforcing that bias by not thinking about how you might be treating people differently because of something they cannot control.
If someone's a bad researcher or gives a bad talk, no need to jump to conclusions about hiring processes, imposters, reverse racism, positive discrimination, and whatever else. That's the real toxicity.
> "We open-sourced our code" but no one can get it to work
This is absolutely par for the course, open source means no warranty. It looks like these researchers are being subjected to unwanted scrutiny simply because they're women, which is quite unfair and very much not OK.
> This is absolutely par for the course, open source means no warranty.
This isn't about open source it's about scientific research. It 1) should be scrutinized and 2) if it's not reproducible, called out as such.
The whole point of presenting research is about advancing the state of the art. If someone else can't reproduce your results, then it's not science. It's just marketing.
There's a whole lot of hard-to-reproduce research out there. Why is these women's work - in EDA of all things, which has been far from friendly to open source historically - being singled out for such strong and apparently unjustified criticism? It's just not fair.
Because re-doing huge studies often isn't feasible or cost-effective for a bunch of reasons. 10+ year studies across a few hundred thousand cases isn't going to happen for many studies. But this is recreating an ML environment and seeing if it generates a similar-ish outcome.
Making someone's code work in a similar env is critical, and the article implies that there were a lot of "released at last minute" and "relearned every time" hurdles that implied that the ladies' work might have been dubious.
I did my PhD, which involved adding a new feature to another software project, verifying that it worked, and presenting benchmarks. But for Reasons, the code ended up being based on a development branch between v2 and v3 of the upstream project. (I used to try to rebase every week, but I ended up spending too much of my time rebasing, and eventually gave up.)
After going into industry, every so often someone would email me asking for my code. I'd mail them a link to a tarball of the Mercurial tree at the point I ran my benchmarks, offering to help get it working. I never heard from anybody a second time -- and nobody ever criticized my work for being un-reproducible. (In fact, eventually I received an ACM SigOps Hall of Fame Award for the work.)
(And since compilers get more paranoid every year, and the snapshot was taken in 2006, the chance of anyone getting it working continues to approach zero.)
> After going into industry, every so often someone would email me asking for my code… I never heard from anybody a second time…
I’m wondering if they wanted the code to clarify what you were doing or to try to port it to the current codebase?
I look at code all the time for stuff I’m interested in without any plans to use and/or to see if there’s anything I can steal for whatever I’m currently poking at.
Code I can read. Mathematical algorithmic notation of code, not so much.
> This is absolutely par for the course, open source means no warranty.
No warranty, sure, but if the original authors have a working program and what they publish doesn't work then did they really meaningfully open source it? Like, if you ask an Android OEM for their kernel source and they send you a tarball that when compiled can't actually boot their phone, is that above board?
(None of this is specific to the particular authors here, BTW; I will happily criticize literally anyone claiming to do Science or Open Source without publishing enough that others can actually reproduce their work.)
The team I work for recently open-sourced a version of our SaaS code. Thanks to docker, we are pretty sure that it will work for you if you follow the regular install instructions- containerization is awesome for that. The instructions on how to set up a local dev environment, without docker? That's a lot harder, has a ton more moving pieces, and if we don't actively maintain and test it regularly (to catch when libraries rot out from underneath us) I suspect that in 6-12 months those instructions will not work for anyone, because this is an incredibly complex beast doing machine-learning related things with a whole lot of dependencies and moving pieces (DB, API, Web interface, caching layer, etc.) so there are a lot of potential failure points.
This was the result of a team of 3-4 SW engineers and a UX engineer working on open-sourcing our existing, working, generating actual revenue SaaS code base for 5-6 months (not as our only task but as a very important one). So it was, say, somewhere around 2 man years of effort to take something that I know exists and works and turn it into a reasonable Open Source project. There is a huge difference in the amount of work required between simple "open sourcing" and actual effective Open Sourcing, and I suspect that few if anyone involved in "Open Science" is doing that work absent large funding sources specifically requiring and supporting with money the good kind. Just a simple fork of the repo? I think the result for us would have been the same as what happened here, and I know that our SaaS project works- I can look at my paycheck and see that.
This dispute has certainly had consequences, most of them negative.
There's an enormous amount of relevant details missing from this article and I don't know why. Why is there no response from Kahng to G&M's rebuttal of his experiments? What did G&M say when told their Github posting wasn't reproducible? Did they agree it was a problem or not? Why did G&M leave Google if the person who was personally undermining them was fired? What exactly did Chatterjee do to personally undermine G&M, and why?
The tangents people, even here at HN, take viz-a-viz these topics, are often mind-boggling. And HN generally has less bot protection than places like reddit, too.
The only reasonable explanation I can think of is that Chatterjee's motives and subsequent termination (which is not elaborated on) was related to gender discrimination. Otherwise I can't see what relevance anyone's genders has on the issue at hand.
It's explained in the Wired article that is linked that this was indeed the cause.
The independent reproduction that the IEEE article talks about also tested the method from the leaked paper from Chatterjee and concluded that his method indeed beats CT in almost all cases at a fraction of the computational cost.
The response from G&M to the independent reproduction emphasizes their method was used for production hardware at Google, was in a peer reviewed paper, and that their GitHub repo had a lot of forks and stars (I'm not making this up). Near the end it then raises several technical objections.
It doesn't look like they didn't train CT to convergence, so idk why anyone would care about those results.
What is even the point of benchmarking against a learning based method of you don't let it finish learning? Quality of output matters way more than compute cost.
Some background is EE professional and academic culture has, for more than a half century been ... less than fine with having women colleagues. Lots of ugly. Decades of improvement too, reflecting a whole lot of years of work from lots of people and leadership. But... CS changed faster.
So an IEEE pub is pointing out that an IEEE-affiliated conference reportedly had... an excursion from professionalism baseline.
Ultimately, what they needed to do was train to convergence, and they didn't do that.
They should have made it extremely clear in their original manuscript their CT results are against a non-pretrained version that was not always trained to convergence, and so the CT results presented are a just a lower bound on performance.
They got confirmation from a Google engineer on only one block, which they trained for 1 million steps. For the other blocks, they didn't train anywhere close to that many steps, and their tensorboards don't show convergence on total cost.
ML is difficult! It's not enough to confirm that your implementation and installation are correct - you also need to train to convergence. You are free to complain that this is onerous or expensive. It's a common problem in academia vs industry in other areas of ML -- ML takes a lot of compute, and academic budgets are extremely tight. That said, CT is still orders of magnitude smaller / cheaper than most modern ML models (not surprising! CT is pretty old at this point by ML standards (3 years is a long time in this field)).
Kahng's FAQ admits that pre-training led to a better final proxy cost in the Nature paper, and that it significantly reduced compute time, and that this is the method that the Nature authors recommend and used in their Table 1 evals.
This whole situation is such a shame. Prof Kahng wrote a really great article on CT a few years back, and it gave me hope that the whole field was going to embrace ML and really take chip design to the next level: https://www.nature.com/articles/d41586-021-01515-9
Most of the field in fact has, but Kahng himself seems to be backsliding here.
Unfortunately, I suspect that Kahng's been listening to some pretty toxic people (the same day his paper was posted to arxiv, it appeared in an SB author's court filing). It seems like his group ran into some difficulties jumping into machine learning, and rather than take that as an opportunity to grow and learn, they took that as an opportunity to attack.
While I'm glad that others in the field have built on the work successfully (see Appendix of https://www.annagoldie.com/home/statement ), it would be really tragic for Kahng himself to get left behind here, and so so unnecessary!
It is also covered in the ISPD'22 paper, "Scalability and Generalization of Circuit Training for Chip Floorplanning", by Summer Yue, Ebrahim M. Songhori, Joe Wenjie Jiang, Toby Boyd, Anna Goldie, Azalia Mirhoseini, and Sergio Guadarrama, which was written by authors from the Nature paper.
https://dl.acm.org/doi/pdf/10.1145/3505170.3511478
In their ISPD'22 paper, link above, Figure 7 shows a 5x speedup but 0.3% worse placement results with pre-training on diverse designs vs. running from scratch. Pre-training on previous versions of the same design does improve results by 1.7% vs. running from scratch, and has similar runtime benefit to pre-training with diverse designs. Given that the Circuit Training approach requires 41x to 646x wall time and an even higher factor of compute time vs. other standard techniques, the 5x speedup with pre-training is insufficient to close that very large gap. Furthermore, pre-training does not improve results vs. training from scratch, unless you are pre-training on previous versions of the same design but even then a 1.7% improvement in the circuit training results is not adequate to demonstrate improvement vs. standard techniques as can be seen by examining the results from Professor Kahng's group's work.
This concern about pre-training has been answered in multiple forums, including the discussion in ISPD'23 Session 8, when Mr. Washauer-Baker raised the same issue. It's unfortunate to see Gabriel Warshauer-Baker, Anna Goldie's spouse, making repeated personal attacks on Professor Kahng. Please focus on the scientific discussion.
53 comments
[ 11.8 ms ] story [ 136 ms ] threadhttps://news.ycombinator.com/newsguidelines.html
Thanks for what you do here, too, btw.
> I'll assume from this sentence they are imposter women who shouldn't be in computer science.
I honestly don't know if you meant that ironically or sarcastically, but holy flippety bippity, we need no such attitudes and certainly no such comments to mess up the culture even more. At all. If you think it's funny or insightful, it isn't.
On the other side of the coin: If you see an asian stem person for example you know for sure they were the cream of the crop since they had everything against them.
This both reinforces stereotypes held by society and in the long run I expect it to only make things worse.
If someone's a bad researcher or gives a bad talk, no need to jump to conclusions about hiring processes, imposters, reverse racism, positive discrimination, and whatever else. That's the real toxicity.
Definitely seems to be true.
This is absolutely par for the course, open source means no warranty. It looks like these researchers are being subjected to unwanted scrutiny simply because they're women, which is quite unfair and very much not OK.
This isn't about open source it's about scientific research. It 1) should be scrutinized and 2) if it's not reproducible, called out as such.
The whole point of presenting research is about advancing the state of the art. If someone else can't reproduce your results, then it's not science. It's just marketing.
Making someone's code work in a similar env is critical, and the article implies that there were a lot of "released at last minute" and "relearned every time" hurdles that implied that the ladies' work might have been dubious.
I did my PhD, which involved adding a new feature to another software project, verifying that it worked, and presenting benchmarks. But for Reasons, the code ended up being based on a development branch between v2 and v3 of the upstream project. (I used to try to rebase every week, but I ended up spending too much of my time rebasing, and eventually gave up.)
After going into industry, every so often someone would email me asking for my code. I'd mail them a link to a tarball of the Mercurial tree at the point I ran my benchmarks, offering to help get it working. I never heard from anybody a second time -- and nobody ever criticized my work for being un-reproducible. (In fact, eventually I received an ACM SigOps Hall of Fame Award for the work.)
(And since compilers get more paranoid every year, and the snapshot was taken in 2006, the chance of anyone getting it working continues to approach zero.)
I’m wondering if they wanted the code to clarify what you were doing or to try to port it to the current codebase?
I look at code all the time for stuff I’m interested in without any plans to use and/or to see if there’s anything I can steal for whatever I’m currently poking at.
Code I can read. Mathematical algorithmic notation of code, not so much.
No warranty, sure, but if the original authors have a working program and what they publish doesn't work then did they really meaningfully open source it? Like, if you ask an Android OEM for their kernel source and they send you a tarball that when compiled can't actually boot their phone, is that above board?
(None of this is specific to the particular authors here, BTW; I will happily criticize literally anyone claiming to do Science or Open Source without publishing enough that others can actually reproduce their work.)
This was the result of a team of 3-4 SW engineers and a UX engineer working on open-sourcing our existing, working, generating actual revenue SaaS code base for 5-6 months (not as our only task but as a very important one). So it was, say, somewhere around 2 man years of effort to take something that I know exists and works and turn it into a reasonable Open Source project. There is a huge difference in the amount of work required between simple "open sourcing" and actual effective Open Sourcing, and I suspect that few if anyone involved in "Open Science" is doing that work absent large funding sources specifically requiring and supporting with money the good kind. Just a simple fork of the repo? I think the result for us would have been the same as what happened here, and I know that our SaaS project works- I can look at my paycheck and see that.
There's an enormous amount of relevant details missing from this article and I don't know why. Why is there no response from Kahng to G&M's rebuttal of his experiments? What did G&M say when told their Github posting wasn't reproducible? Did they agree it was a problem or not? Why did G&M leave Google if the person who was personally undermining them was fired? What exactly did Chatterjee do to personally undermine G&M, and why?
ieee.org's editors are not doing a service to their members by making a chip design argument about AI a gender issue.
The tangents people, even here at HN, take viz-a-viz these topics, are often mind-boggling. And HN generally has less bot protection than places like reddit, too.
The independent reproduction that the IEEE article talks about also tested the method from the leaked paper from Chatterjee and concluded that his method indeed beats CT in almost all cases at a fraction of the computational cost.
The response from G&M to the independent reproduction emphasizes their method was used for production hardware at Google, was in a peer reviewed paper, and that their GitHub repo had a lot of forks and stars (I'm not making this up). Near the end it then raises several technical objections.
What is even the point of benchmarking against a learning based method of you don't let it finish learning? Quality of output matters way more than compute cost.
Some background is EE professional and academic culture has, for more than a half century been ... less than fine with having women colleagues. Lots of ugly. Decades of improvement too, reflecting a whole lot of years of work from lots of people and leadership. But... CS changed faster.
So an IEEE pub is pointing out that an IEEE-affiliated conference reportedly had... an excursion from professionalism baseline.
https://www.wired.com/story/google-brain-ai-researcher-fired...
The rebuttal was published less than 2 weeks ago, so that seems a bit early, and it is public, so reading it would answer several of your questions.
https://www.annagoldie.com/home/statement
The ACM article also has some relevant details: https://cacm.acm.org/news/271439-more-details-but-not-enough...
They should have made it extremely clear in their original manuscript their CT results are against a non-pretrained version that was not always trained to convergence, and so the CT results presented are a just a lower bound on performance.
They got confirmation from a Google engineer on only one block, which they trained for 1 million steps. For the other blocks, they didn't train anywhere close to that many steps, and their tensorboards don't show convergence on total cost.
ML is difficult! It's not enough to confirm that your implementation and installation are correct - you also need to train to convergence. You are free to complain that this is onerous or expensive. It's a common problem in academia vs industry in other areas of ML -- ML takes a lot of compute, and academic budgets are extremely tight. That said, CT is still orders of magnitude smaller / cheaper than most modern ML models (not surprising! CT is pretty old at this point by ML standards (3 years is a long time in this field)).
Kahng's FAQ admits that pre-training led to a better final proxy cost in the Nature paper, and that it significantly reduced compute time, and that this is the method that the Nature authors recommend and used in their Table 1 evals.
This whole situation is such a shame. Prof Kahng wrote a really great article on CT a few years back, and it gave me hope that the whole field was going to embrace ML and really take chip design to the next level: https://www.nature.com/articles/d41586-021-01515-9
Most of the field in fact has, but Kahng himself seems to be backsliding here.
Unfortunately, I suspect that Kahng's been listening to some pretty toxic people (the same day his paper was posted to arxiv, it appeared in an SB author's court filing). It seems like his group ran into some difficulties jumping into machine learning, and rather than take that as an opportunity to grow and learn, they took that as an opportunity to attack.
While I'm glad that others in the field have built on the work successfully (see Appendix of https://www.annagoldie.com/home/statement ), it would be really tragic for Kahng himself to get left behind here, and so so unnecessary!
It is also covered in the ISPD'22 paper, "Scalability and Generalization of Circuit Training for Chip Floorplanning", by Summer Yue, Ebrahim M. Songhori, Joe Wenjie Jiang, Toby Boyd, Anna Goldie, Azalia Mirhoseini, and Sergio Guadarrama, which was written by authors from the Nature paper. https://dl.acm.org/doi/pdf/10.1145/3505170.3511478
In their ISPD'22 paper, link above, Figure 7 shows a 5x speedup but 0.3% worse placement results with pre-training on diverse designs vs. running from scratch. Pre-training on previous versions of the same design does improve results by 1.7% vs. running from scratch, and has similar runtime benefit to pre-training with diverse designs. Given that the Circuit Training approach requires 41x to 646x wall time and an even higher factor of compute time vs. other standard techniques, the 5x speedup with pre-training is insufficient to close that very large gap. Furthermore, pre-training does not improve results vs. training from scratch, unless you are pre-training on previous versions of the same design but even then a 1.7% improvement in the circuit training results is not adequate to demonstrate improvement vs. standard techniques as can be seen by examining the results from Professor Kahng's group's work.
This concern about pre-training has been answered in multiple forums, including the discussion in ISPD'23 Session 8, when Mr. Washauer-Baker raised the same issue. It's unfortunate to see Gabriel Warshauer-Baker, Anna Goldie's spouse, making repeated personal attacks on Professor Kahng. Please focus on the scientific discussion.