I dont see this strategy ending up anywhere good for openAI. They don't have revenue and yet they ll be asked to subsidize the entire internet. They are priming their users to the idea that they won't use advertising, or manipulation. But when they do do that, their subscription model will collapse
seems like a way to slow down competitors more than anything. if they offer generous terms to sources of high quality data, they can anchor pricing out of reach for most others or maybe even snag exclusivity.
plus there's a high likelihood they were scraping and cleaning this data anyway. a formal agreement puts that effort on the publisher's side, letting them cut cost, complexity, and the potential for bad data.
What is the reason LLM training requires so much data? It appears they require the entire output of human civilization in order to stay competitive, but why? Is it simply that their trainers can get that much data, or are they just very inefficient in how they learn?
According to Ars Technica in the linked article : "In the longer term, the deal also means that OpenAI can openly and officially utilize Condé Nast articles to train future AI language models, which includes successors to GPT-4o. In this case, "training" means feeding content into an AI model's neural network so the AI model can better process conceptual relationships.
AI training is an expensive and computationally intense process that happens rarely, usually prior to the launch of a major new AI model, although a secondary process called "fine-tuning" can continue over time. Having access to high-quality training data, such as vetted journalism, improves AI language models' ability to provide accurate answers to user questions."
> [...] are they just very inefficient in how they learn?
I'd challenge: inefficient compared to what? For many tasks, the only more capable systems we know of are biological ones that evolved on 4 billion years' worth of data followed by years of constant input from all senses. Gradient descent is incredibly fast and data-efficient by comparison (hence use of mutation + natural selection for machine learning is mostly limtied to tiny toy examples).
> It's worth noting that Condé Nast internal policy still forbids its publications from using text created by generative AI, which is consistent with its AI rules before the deal.
This is a key point. OpenAI is getting guaranteed non-AI generated text data; which is slowly becoming a very valuable resource. You can expect that future LLMs may not require trillions of tokens of text data to generalize if generalizability is picked up better by future model techniques (which IMO would involve picking up general pattern recognition via non-textual data too (i.e. other modals of the current multi-modal and beyond such as puzzles)), and thus even just millions to billions of tokens via high quality sources like news publications, books, and research papers, will be highly worth it.
Also, the SearchGPT integration for latest news is also a plus point. Think of all of those all-you-can-read news subscriptions such as Scroll, but that instead of paying for it that instead paying for SearchGPT covers not just a high quality search engine but also the dues to the news publishers via such payment (via these partnerships) from OpenAI to the news publishers.
7 comments
[ 3.1 ms ] story [ 22.3 ms ] threadplus there's a high likelihood they were scraping and cleaning this data anyway. a formal agreement puts that effort on the publisher's side, letting them cut cost, complexity, and the potential for bad data.
AI training is an expensive and computationally intense process that happens rarely, usually prior to the launch of a major new AI model, although a secondary process called "fine-tuning" can continue over time. Having access to high-quality training data, such as vetted journalism, improves AI language models' ability to provide accurate answers to user questions."
I'd challenge: inefficient compared to what? For many tasks, the only more capable systems we know of are biological ones that evolved on 4 billion years' worth of data followed by years of constant input from all senses. Gradient descent is incredibly fast and data-efficient by comparison (hence use of mutation + natural selection for machine learning is mostly limtied to tiny toy examples).
This is a key point. OpenAI is getting guaranteed non-AI generated text data; which is slowly becoming a very valuable resource. You can expect that future LLMs may not require trillions of tokens of text data to generalize if generalizability is picked up better by future model techniques (which IMO would involve picking up general pattern recognition via non-textual data too (i.e. other modals of the current multi-modal and beyond such as puzzles)), and thus even just millions to billions of tokens via high quality sources like news publications, books, and research papers, will be highly worth it.
Also, the SearchGPT integration for latest news is also a plus point. Think of all of those all-you-can-read news subscriptions such as Scroll, but that instead of paying for it that instead paying for SearchGPT covers not just a high quality search engine but also the dues to the news publishers via such payment (via these partnerships) from OpenAI to the news publishers.