Not to be a downer, but does anyone have a better source for this?
This website is (at least to me) full of ads for “AI stock set to soar” and even clicking on the site’s logo to visit the homepage first showed a full page popup with the same ad about an AI stock. I didn’t know gAds even offered a full page modal option.
If the headline explicitly anthropomorphizes a language model masquerading as AI, I doubt you're going to get anything reasonable by clicking through it.
The paper is quite approachable and worth a quick scan. In essence, reusing output from previous generations results in rather wonky results which are clearly separable from the original training data.
You probably have to clarify that it only applies to the direct transformer training with fixed weights for samples, and not RL variants or variable sample weights.
My car stereo loses its mind once a month or so and I have to reboot it. (SYNC, it sucks)
One time my garage door kept refusing to close, intermittently and at a random spot in the cycle. “Stupid thing is going crazy”. (Mental metaphor still there)
Often times I’ll notice that my golf balls “have a mind of their own”
This is not a weird use of the idiom. I think it’s a fine headline.
I think the whole site is LLM-generated. There are articles where the headline is not even directly related to the content, and the content is... well, the problem is that it is nearly content-free.
For example, their super guide to prompt engineering excellence: start by getting a bachelor's degree in the field you want to prompt about...
Same for the 70's and 80's music cassette bootlegging scene. Some of the tapes were quite unlistenable because they were so degraded from being a copy of a copy of a copy.
The irony is that AI is great at cleaning them up!
Its potentially an even worse problem than overfitting because of the error accumulation and undermining of the fitness functions. Its similar to asking students to create test questions to evaluate themself. As the proportion of self output questions grows, the context eventually drifts to be meaningless. In other words it leads to test "questions" like "The answer is A".
One measurement of this would be to watch the growth of probability of new models to say similar things to older model filters like "As a large language model ..."
Perhaps this is the salvation from too much influence by AI. Incest will eradicate itself. With language models in particular, it is easy to understand that only a multiplication of nonsense can be produced in a very short time if it has to reproduce itself because the influx of human content can no longer keep up.
That's why I worry we've approached the limits of what LLMs can do, because now that they're used heavily, the next one trying to train on even more data will just be training on the output of the others. It'll be really hard to filter it out.
26 comments
[ 25.9 ms ] story [ 1287 ms ] threadThis website is (at least to me) full of ads for “AI stock set to soar” and even clicking on the site’s logo to visit the homepage first showed a full page popup with the same ad about an AI stock. I didn’t know gAds even offered a full page modal option.
https://arxiv.org/abs/2307.01850
https://arxiv.org/pdf/2307.01850.pdf
The paper is quite approachable and worth a quick scan. In essence, reusing output from previous generations results in rather wonky results which are clearly separable from the original training data.
My car stereo loses its mind once a month or so and I have to reboot it. (SYNC, it sucks)
One time my garage door kept refusing to close, intermittently and at a random spot in the cycle. “Stupid thing is going crazy”. (Mental metaphor still there)
Often times I’ll notice that my golf balls “have a mind of their own”
This is not a weird use of the idiom. I think it’s a fine headline.
For example, their super guide to prompt engineering excellence: start by getting a bachelor's degree in the field you want to prompt about...
The irony is that AI is great at cleaning them up!
One measurement of this would be to watch the growth of probability of new models to say similar things to older model filters like "As a large language model ..."