If someone tries to run your method but messes it up, and then accuses you of fraud when the results don't match their expectations, I'm not sure they're entitled to a neutral tone in response. Maybe you're right, and a…
I think the pre-trained checkpoint uses the same 20 TPU blocks as the original paper, but it probably isn't the exact-same checkpoint, as the paper itself is from 2020/2021.
As Andrew Kahng was one of the co-authors of Cheng et al., all of the issues with his reproduction still matter here. The Nature paper went through an investigation and second round of peer review. AlphaChip is used to…
Three generations of TPU, Axion (ARM-based CPU), various other chips at AlphaBet, MediaTek's usage...
The UCSD paper didn't run the Nature method correctly, so I don't see how you can draw this conclusion. From Jeff's tweet: "In particular the authors did no pre-training (despite pre-training being mentioned 37 times in…
"Prior to publication of Cheng et al., our last correspondence with any of its authors was in August of 2022 when we reached out to share our new contact information." You don't stop being the corresponding authors of a…
"These major methodological differences unfortunately invalidate Cheng et al.’s comparisons with and conclusions about our method. If Cheng et al. had reached out to the corresponding authors of the Nature paper[8], we…
I mean, I think second is still "one of the first?" And, no offense to this project, but I don't know of it being used in a real industrial setting, whereas AlphaChip was used in TPU.
So much wasted time. He even ran a study internally (with Markov), but, as the AlphaChip authors describe: In 2022, it was reviewed by an independent committee at Google, which determined that “the claims and…
> Does the EDA ecosystem support a similarly open culture of benchmarking for commercial tools? If only. The comparison in Cheng et al. is the only public comparison with CMP that I can recall, and it is pretty suss…
> Did they mess up when they did not pre-train or they followed the "steps" described in the original repo and tried to get a fair reproduction? The Circuit Training repo was just going through an example. It is common…
Bitter Lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
Are you really suggesting that the TPU team does not stand behind the graphs in Google's own blog post? And that MediaTek does not stand behind their quoted statement?
His original complaint being dismissed matters because it suggests that he was fishing around for a complaint that was valid, and that perhaps his primary motivation was to get money out of Google. Legal nitpick - you…
> it might have a harder time with a chip designed by a third party (further from its pre-training). Then they could pre-train on chips that are in-distribution for that task. See also section 3.1 of their response…
They also changed the ratio of RL experience collectors to GPU workers (~1/20th the RL experience collectors, 1/2 the GPUs). I don't know what impact that has --- maybe each GPU episode has less experience? Maybe that…
> published a crappy article in Nature because it would never have passed editorial muster at something like DAC or an IEEE journal and now have to browbeat other people who are calling them out on it. I don't think…
Pre-training is just training on multiple chips. "If Cheng et al. had reached out to the corresponding authors of the Nature paper, we would have gladly helped them to correct these issues prior to publication"…
That is definitely a cool project, but I don't see how it contradicts "one of the first RL methods deployed to solve a real-world engineering problem". "One of the first" does not mean literally the first ever.
It is open: https://github.com/google-research/circuit_training
> The whole publication process seems dishonest, starting from publishing in Nature (why not ISCCC or something similar?) Why would you publish in ISCCC when you can get into Nature?
You're linking to his amended complaint - his original complaint was thrown out because it alleged things like "Google's motto is don't be evil, but they were evil, thus defrauding me." According to a Google…
What are you even talking about? Jeff had a hand in TPU, which is so successful that all other AI companies are trying to clone this project and spin up their own efforts to make custom AI chips.
> Some would say he got taken for a ride by a young charismatic grifter and is now in too deep to back out. Was the TPU physical design team also taken in? And also MediaTek? And also TF-Agents, which publicly said they…
See my comment above - the Nature authors already did this, and tried a huge hyperparameter sweep for SA, and RL still won. See appendix of the Nature article: rdcu.be/cmedX
If someone tries to run your method but messes it up, and then accuses you of fraud when the results don't match their expectations, I'm not sure they're entitled to a neutral tone in response. Maybe you're right, and a…
I think the pre-trained checkpoint uses the same 20 TPU blocks as the original paper, but it probably isn't the exact-same checkpoint, as the paper itself is from 2020/2021.
As Andrew Kahng was one of the co-authors of Cheng et al., all of the issues with his reproduction still matter here. The Nature paper went through an investigation and second round of peer review. AlphaChip is used to…
Three generations of TPU, Axion (ARM-based CPU), various other chips at AlphaBet, MediaTek's usage...
The UCSD paper didn't run the Nature method correctly, so I don't see how you can draw this conclusion. From Jeff's tweet: "In particular the authors did no pre-training (despite pre-training being mentioned 37 times in…
"Prior to publication of Cheng et al., our last correspondence with any of its authors was in August of 2022 when we reached out to share our new contact information." You don't stop being the corresponding authors of a…
"These major methodological differences unfortunately invalidate Cheng et al.’s comparisons with and conclusions about our method. If Cheng et al. had reached out to the corresponding authors of the Nature paper[8], we…
I mean, I think second is still "one of the first?" And, no offense to this project, but I don't know of it being used in a real industrial setting, whereas AlphaChip was used in TPU.
So much wasted time. He even ran a study internally (with Markov), but, as the AlphaChip authors describe: In 2022, it was reviewed by an independent committee at Google, which determined that “the claims and…
> Does the EDA ecosystem support a similarly open culture of benchmarking for commercial tools? If only. The comparison in Cheng et al. is the only public comparison with CMP that I can recall, and it is pretty suss…
> Did they mess up when they did not pre-train or they followed the "steps" described in the original repo and tried to get a fair reproduction? The Circuit Training repo was just going through an example. It is common…
Bitter Lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
Are you really suggesting that the TPU team does not stand behind the graphs in Google's own blog post? And that MediaTek does not stand behind their quoted statement?
His original complaint being dismissed matters because it suggests that he was fishing around for a complaint that was valid, and that perhaps his primary motivation was to get money out of Google. Legal nitpick - you…
> it might have a harder time with a chip designed by a third party (further from its pre-training). Then they could pre-train on chips that are in-distribution for that task. See also section 3.1 of their response…
They also changed the ratio of RL experience collectors to GPU workers (~1/20th the RL experience collectors, 1/2 the GPUs). I don't know what impact that has --- maybe each GPU episode has less experience? Maybe that…
> published a crappy article in Nature because it would never have passed editorial muster at something like DAC or an IEEE journal and now have to browbeat other people who are calling them out on it. I don't think…
Pre-training is just training on multiple chips. "If Cheng et al. had reached out to the corresponding authors of the Nature paper, we would have gladly helped them to correct these issues prior to publication"…
That is definitely a cool project, but I don't see how it contradicts "one of the first RL methods deployed to solve a real-world engineering problem". "One of the first" does not mean literally the first ever.
It is open: https://github.com/google-research/circuit_training
> The whole publication process seems dishonest, starting from publishing in Nature (why not ISCCC or something similar?) Why would you publish in ISCCC when you can get into Nature?
You're linking to his amended complaint - his original complaint was thrown out because it alleged things like "Google's motto is don't be evil, but they were evil, thus defrauding me." According to a Google…
What are you even talking about? Jeff had a hand in TPU, which is so successful that all other AI companies are trying to clone this project and spin up their own efforts to make custom AI chips.
> Some would say he got taken for a ride by a young charismatic grifter and is now in too deep to back out. Was the TPU physical design team also taken in? And also MediaTek? And also TF-Agents, which publicly said they…
See my comment above - the Nature authors already did this, and tried a huge hyperparameter sweep for SA, and RL still won. See appendix of the Nature article: rdcu.be/cmedX