So it's okay when big American corps raid the internet ignoring any terms of service or licenses they see in order to train models they rent back to us, but when a foreign entity trains off of Anthropic it's illegal?
I don’t think I’m the only one feeling some schadenfreude at this news. I suppose it’s ok when you’re a hot Silicon Valley scale-up to slurp up the rest of the worlds data for free and then hire hot shot lawyers to defend you against all the creatives you ripped off, but when it’s the “evil” Chinese doing the same to you it’s a dastardly “attack”?
Yeah - not only have we seend some of the same large companies that have trampled regular people and made examples of them in name of defending copyright fully ignore it when it was time to feed their AI models.
And now the hypocrisy went full circle with complains of others not respecting their rights!
I don't think this counts as distillation. Distillation is when you use a teacher model to train a student model, but crucially, you have access to the entire probability distribution of the generated tokens, not just to the tokens themselves. That probability distribution increases tremendously the strength of the signal, so the training converges much faster. Claude does not provide these probabilities. So, Claude was used for synthetic training data generation, but not really for distillation.
One difference between Anthropic and others is that Anthropic is crawling publicly visible information, and their argument is that this is fair use. Whereas these Chinese LLMs are circumventing an account creating process and terms of service to misuse non public information.
Lots of people think Anthropic training their own LLM is the same but it really isn’t.
If just 16 million examples were enough to significantly boost model quality (as Anthropic claims), it turns out that data quality beats quantity
Instead of vacuuming petabytes of trash from Common Crawl, you can just take high-quality distillate from a SOTA model and get comparable results. Bad news for anyone betting solely on massive compute clusters and closed datasets
9 comments
[ 2.6 ms ] story [ 26.6 ms ] threadI don’t think I’m the only one feeling some schadenfreude at this news. I suppose it’s ok when you’re a hot Silicon Valley scale-up to slurp up the rest of the worlds data for free and then hire hot shot lawyers to defend you against all the creatives you ripped off, but when it’s the “evil” Chinese doing the same to you it’s a dastardly “attack”?
And now the hypocrisy went full circle with complains of others not respecting their rights!
Lots of people think Anthropic training their own LLM is the same but it really isn’t.
They publish weights and useful research for everyone to benefit.
I mean this is incredibly tone deaf for a company facing multiple lawsuits over where they got their training data from.
https://news.ycombinator.com/item?id=47126614
Instead of vacuuming petabytes of trash from Common Crawl, you can just take high-quality distillate from a SOTA model and get comparable results. Bad news for anyone betting solely on massive compute clusters and closed datasets