It's similar to us. We can reason about certain things but we often have to look up additional information in a book or ask someone. In the same way I see LLMs being augmented with traditional code. It doesn't really matter if the result is called AI or not.
> It doesn't really matter if the result is called AI or not
I strongly disagree. Wildly overvalued companies disrupt the economy and present huge risks - see Tesla or Nvidia (currently valued at $100m per employee - $3T market cap / 30k employees)
The AGI party might be cancelled. But the current strengths of LLMs still are summarization, analogy, templating and search. Which can solve a lot of business problems already.
My hot take is that UX is the biggest limiter for current LLM products other than chatGPT. Chat has too much friction for many use cases, we need to find better interfaces that are more visual and faster to interact with. Spending $100M to gain .1% on MMLU is a waste of time in comparison.
It absolutely can be. Prompt construction, as well as learning what sort of things the LLM just isn't good at doing, takes work. I've had multiple times where I've asked for something, gotten complete trash back from the LLM, complained to a friend who's more into LLMs about it, and had them go "Oh, you just have to add this extra set of words at the end/phrase it this other way and it'll work." Or they'll give me some massive multi-paragraph "seed" prompt I need to add in to make it not behave stupidly sometimes, that they evidently have to update every few days or weeks to adapt to some new behavior.
Until Google decided to trash their search with their own sub-par LLM, my results were often regularly better just hitting the first link on Google Search like I had for the last decade.
When you go to a sit-down restaurant, you aren't given a list of the ingredients that they have out back in the kitchen and get asked how you want to assemble them. Someone's done work to take those ingredients and form them into menu items for you to pick from.
Given infinite possibilities to enter into a text box is more friction than picking from a menu in front of you.
There's a lot of use for LLMs, and it's tossing the ingredients in a restaurant's kitchen together but sometimes people want a menu.
I like that analogy, but I'd almost take it a little further. It's a restaurant with no menu and claims it has every ingredient you could ever want in the kitchen. If you ask for a sandwich, you'll get a sandwich, and it'll probably be edible. But if you want anything approaching the quality of even a mid-tier restaurant, you almost need to know how to cook it yourself in order to order it, or at least have a very broad idea of what you're willing to accept.
Just saying you want a chicken florentine isn't enough. You may also have to specify that the chicken is fresh but also not still alive, and that kale isn't the same as spinach, and confirm that they have the correct recipe for mornay sauce even though every chef should already know that. And even if you do all that, you may still get a chicken cordon bleu for reasons you cannot explain and neither can the kitchen.
When it's free/cheap, and I'm up for an adventure and sending stuff back to try again a few times, it's not bad. If I'm having to pay a pretty high amount for it and still treat learning the system like a second job, I start to have questions about why I'm bothering.
Yeah. But then an eccentric celebrity comes into your restaurant. Them eating there would be great for your restaurant, so you seat them. They ask for Bubble and Squeak, which isn't on the menu and you've never made it before, though you do happen to have all the ingredients for it. It's a busy night but you've never done it before so you ask your Large Language Mise-en-place-cook to whip up one and it's... okay. You've done a lot of cooking before, so you can spot some of the problems with how the LLM whipped up the dish, and you take the one they made, fix up a few things, and ship it out the door in less time than if you didn't have your Large Language Mise-en-place-cook helping you. It was never going to get it perfectly right, but you're a busy, but expert cook, and you know its shortcomings. Rather than try to get it to work perfectly on its own, you pair with it and are able to get more dishes out than you would be without its help. You're not convinced that it's progress, but it's where the industry has moved.
Makes sense, my biggest takeaway from observing folks in many different roles interact with LLMs is this: they simply don't know how to prompt effectively.
There is no problem with the chat UX, if you know the right techniques to get optimal output, but writing effective prompts is a technical skill similar to programming - most folks that interact with computers are not highly technical.
If we design software powered by LLMs understanding that effective prompting is a skill most users don't have, then we should build UX that abstracts prompting away. Of course there are going to be use cases where a user needs to provide input to the LLM, but that UX also can be assistive and require minimal cognitive overhead (e.g less raw text prompt input, more clicks on buttons/forms etc to a backend prompt builder).
Alas, "summarization" is a crapshoot. It sometimes summarizes essential facts out of the original text. I think it's due to limits on output tokens. And chunking cannot solve this because you cannot summarize well if you don't take in the whole thing. Aggregating summarized chunks doesn't produce good results.
It works very very well on Amazon reviews, though, because these are small and thin.
What a weird article with weird statements and conclusions. If GPT-5 or whatever it would be called comes out as a failure, only then you could have a chance at concluding what the title says. Also multimodality, sensor input data, etc besides just text for the same model has a potential to generate a lot more training data. Eventually the model could get the exact same input as we humans get from sight, hearing and other sensors.
To me there is still so much left to do.
And even if we truly did multiple laps over all the data in the World there are still different new architectures and strategies left to try.
Ahh the ole if you can sense every particles position and velocity you can predict the future.
your comment really belies the desperation that exists now, these models are stuck where they are (hint it’s a natural limit), you are talking about exponentiation of cost to get what a 10% improvement? 5%? They are very few places for which it’s net positive to run them now, and most of those are incredibly shitty things like creating trash marketing content to drown us all in average inanity
I really feel bad for this next generation, they will just be constantly inundated with generated crap, so much of the high fidelity of conversation and meaning is and will be lost.
Desperation? We have had huge advances just from 1.5 years ago with things I wouldn't have thought would be possible in near 10 years. After decades of research with far slower progress and all of sudden we now know that we have hit a wall?
And I am not talking about predicting the future, but more predicting the next action to take based on current state, sensor data in a more seamless way. Like a human being reacting to different input, by moving their muscles etc. There would be huge amount of training data from there that could be incorporated into a single model.
> And I am not talking about predicting the future, but more predicting the next action to take based on current state, sensor data in a more seamless way.
Like self-driving cars?
Self-driving cars is an engineering problem, let alone an AI problem, and we still cannot solve it despite trillion dollar economic incentives.
Just putting together some LLMs on a fuckton of data does not work. Tesla tried that, and failed.
Tesla has been using sparse data to train their models, because they needed to prioritize fast on device inference.
Completely different solution applied to a completely different problem with completely different risk and quality tolerances with completely different mitigations.
The point is to try and see if LLM's wide general knowledge can have advantage in something like sensory data + action learning as well. Current self driving models don't have that.
I don't understand this stretch logic. It absolutely depends on the type of problem where they are inaccurate, how well trained they are in it, there is no way you can extrapolate like this.
You can ask them to do math equation which takes steps and if they are trained in that for certain problems they are accurate near 100 percent of the time.
Like ask gpt-4o to solve different variations of
"""What is the answer to 2x + 7 = 31?"""
If the numbers are of similar magnitude and simplicity, it will follow the same steps and be right 99%+ times, and I'm only not saying 100%, because I haven't tried it enough, but I don't see it being wrong.
For example """What is the answer to 2x + 4 = -6?"""
Just run a test yourself. Do random integers within 0 - 20, it will definitely not be incorrect 5% - 10% time. It will be correct 99%+ time.
Where is this number 5% - 10% even coming from? You could also keep asking it "What is the capital of France?" and it's going to be right 99%+ of the time.
My understanding (roughly) is that the way we got here was kind of by surprise. We've had a lot of the fundamental algorithms for a long time, but we ran into sort of a happy surprise when transformer models got scaled up - suddenly they started doing interesting things. Scaling them up even more made them start do do potentially useful things.
That happy discovery was never really a linear improvement path, though. We had an explosion of capability, but all along there have been active questions about how far the improvements would go with the current approach.
I think the point that a lot of researchers are making is that that we're starting to see those limits (with LLMs, at least).
There are also a lot of questions around business model and cost/value prop. Training and running these things at scale is enormously expensive. I'm seeing a lot of FOMO and gold rush mentality in the space, similar to the online streaming wars, and I'm not convinced of the long term viability of a lot of the companies. Especially once open models like llama are "good enough" and become commodities.
Of course, it's still early days and there's a ton of room for discovery, but it looks like we'll hit a limit with the current approach pretty soon.
Personally, I'd be OK with that. With the current state of things we have an interesting toy that can sometimes do useful work. It's an incremental quality of life improvement and another good tool in the chest, but it's not a civilization impacting technology.
My question still stands. How could anyone besides OpenAI be confident of there being a limit if no one has managed to even build as strong model so far as OpenAI has? Only Claude Opus seems close, but still weaker at reasoning than GPT-4o. Better at creative writing though.
And only after 1.5 years? And especially of we just had an happy surprise like you mentioned. How does it make sense to already start claiming that we have hit the limits. How do we know there is no more scaling, optimisations and happy surprises?
> That happy discovery was never really a linear improvement path, though. We had an explosion of capability, but all along there have been active questions about how far the improvements would go with the current approach.
> I think the point that a lot of researchers are making is that that we're starting to see those limits (with LLMs, at least).
The kinds of limitations we're "starting to see" are largely the same as they were a year ago. People were talking about it on here back then, but now it's becoming more apparent to more people as they get used to LLMs.
For those who saw it back then, this does look like we're hitting a limit. For others, not so much.
How do active questions about a technology imply we are approaching a brick wall?
How could researchers without having access to the latest state of the art - by OpenAI or any other unknown companies be able to even test that we could be approaching a brick wall? It seems to me that it would take trillions to find out what the exact limit is.
It's possible that we will get diminishing returns, but I don't see how we can confidently claim or know it?
> The kinds of limitations we're "starting to see" are largely the same as they were a year ago. People were talking about it on here back then, but now it's becoming more apparent to more people as they get used to LLMs.
I don't follow. GPT-3.5 was borderline useless at reasoning.
But it still seemed amazing and what I wouldn't have thought to be possible in any near future.
And then GPT-4 was a crazy advancement over that to me. And I've been using it daily since it was available, for various use-cases. Are you saying we are seeing the limitations of GPT-4 specifically? Because, sure, GPT-4 is far from AGI, but I don't see how this implies that further scaling, optimisation, training data improvements, techniques like multi modality and other potential strategies that I might not be aware of couldn't bring another explosive step?
Also the fact that GPT-4 reasoning skill hasn't been reproduced by anyone else so far seems to leave me thinking that everyone except OpenAI are clueless. Claude Opus is close, like I've said before, but not quite GPT-4 levels in specific reasoning tasks that I'm using the API for.
If you can't reproduce GPT-4, how could we trust the assessment that we have hit a limit?
There weren't really any advancements from around 2018. The majority of the 'advancements' were in the amount of parameters, training data, and its applications. What was the GPT-3 to ChatGPT transition? It involved fine-tuning, using specifically crafted training data. What changed from GPT-3 to GPT-4? It was the increase in the number of parameters, improved training data, and the addition of another modality. From GPT-4 to GPT-40? There was more optimization and the introduction of a new modality. The only thing left that could further improve models is to add one more modality, which could be video or other sensory inputs, along with some optimization and more parameters. We are approaching diminishing returns.
> There weren't really any advancements from around 2018.
Not sure what that means. Why are you marking those as "quotes".
The last actions brought so many returns. And it's unknown what the exact effect would be in adding more modalities, training data, optimisations and even just plain parameters.
Text as training data can only get you so far. Giving real time sensory data from many fields could allow LLM like system to control robots and get even more data from real life. E.g. robot hand movements, object tracking data, all of that to be fed into LLMs, and see how it would work.
- something that can't be modeled because there's no training data
- something that can't be modeled because it's fundamentally stochastic
- something that can't be modeled because the discrepancy in simulating the generating process, for your specific model, can, basically, be made arbitrarily large
Isn’t the statement “being stuck” a bit like an attempt at predicting the future? You don’t know how long something will stay stuck…
I think a very common error when it comes to personal learning or progress is confusing a plateau with a brick wall. The reason is, unless you have already walked the path, it’s not possible to differentiate them. And when it comes to progress, no one has already walked the path, hence no one knows actually.
> If GPT-5 or whatever it would be called comes out as a failure, only then you could have a chance at concluding what the title says.
What if it already did, and it's called GPT4-o? Like sure, OAI realized it was a mostly marginal improvement over 4 after it was finished. But did they know that ahead of training?
It would be odd because GPT-4o is much faster and cheaper so it just seems optimised, pruned gpt-4 with few improvements in certain intelligence areas.
I would have expected gpt-5 at least initially to be much, much slower, and they have only recently talked about starting on it.
What’s wrong with being excited and working hard towards something? Hoping for nearly anyone’s demoralization is not good. I wish you a spirit of adventure that takes you many exciting places!
Also, what specific safety problems do you think need to be solved?
There’s a sudden market for a new flavor of bullshit, and brand-new bullshit providers have sprung up to fill the vacuum. But all the exec-webinar/thought-leader/hypetrain/pr stuff is a sideshow from people building actual value.
> What’s wrong with being excited and working hard towards something?
Ahh, the old "how can cryptocurrency be harmful if people like buying it?" argument.
There are many realistic and amenable goals in life, like mapping the human genome or writing an Open Source microkernel. Declaring that you intend to surpass human intelligence via a statistical text generator is not one of those things. It does not have precedent, it does not have feasibility studies, it does not seemingly indicate in any way shape or form that it is possible. Nobody can even spell-out the intermediate steps to get us to AGI; every single "novel" solution involves scaling up our current, broken, concepts. It's ELIZA versus the Lisp pundits all over again.
Excitement and hard work go a long ways, but you won't know which way until you apply a little logic. The current "AGI" trend is practically non-existent outside the venture-capital sphere and OpenAI employees, both of whom would be bullish on AGI anyways since it's good for business. Once you discard the biased opinions, you're left with legitimately confused investors and nonsense opinions propagated by conspiracy theorists. Much like crypto, the concept of "AGI" is being used to confuse and exploit people who misunderstand technology and finance.
> Also, what specific safety problems do you think need to be solved?
I think you misread their comment. They said "safely incorporated", which is not a specific safety problem for AI but a holistic consideration that stops your product from sucking. Your computer vision model could be statistically perfect, but absolutely useless for self-driving tasks and multimodal robotic agents. It's not taste that separates these good implementations from the bad ones, it's logic. You have to be considerate when implementing AI in traditional systems, because inherently AI can be wrong and you must have a failure-mode for those situations.
Many people reject this idea, because it precludes the idea that someone could sell a cure-all to today's AI ills. But real AI safety cannot be baked-in to a model. It only exists when genuinely thoughtful humans anticipate every single fail-state; if that sounds like hard work, it's because it is. And nobody, nobody, sells it as "AGI".
Calling the guy e/acc is absurd. He’s the polar opposite, an e/altruist.
Not a fan of e/acc but seems like you’re just labeling rather than adding anything important to the conversation, ironic because you seem to stress the importance of calmer and cautious heads.
My point is that you labeled him e/acc when e/acc was created as an ideological protest to effective altruism, which is known for it s safetyist stance in AI.
Oh, I thought this was going to be about AIs with more situational awareness.
Notes:
- What's the difference between "superalignment" and regular "alignment", anyway? OpenAI uses the term, but mostly for marketing purposes.
- Alignment to what?
Select alignment configuration option:
1. Asimov's Three Laws.
2. Friedman's “There is one and only one social responsibility of business—to use its resources and engage in activities designed to increase its profits.”
3. Xi Thought.
4. MAGA.
5. The first pillar of Islam, "There is no god but God, Muhammad is the messenger of God."
6. The Leader is never wrong.
Selection? >
I agree. The idea behind predicting the next token is that you regress to the mean. If humans are mostly idiots, then the mean is idiots. If you filter your data to only include genius humans, then you can train and get an average answer that a genius would write online to a given prompt. But how would you train to generate an order of magnitude of intelligence or insight above a genius if the entire corpus of training data is at most genius-level?
You'd have to hope for some sort of emergent intelligence or knowledge-breadth integration based intelligence I suppose. But I already get annoyed by ChatGPT 4o for having Average Redditor Tier ideas and responses.
I think of it differently. The model becomes a simulator, an actor of a sort. If it has enough of good data to build good patterns or reaches a certain intelligence threshold, the content generated by less than average intelligence becomes an acting material to understand those flawed ideas and how to emulate them, which still requires to build and improve empathy. It can be challenging to exactly predict what a "stupid" person would say next, so you need to develop understanding to human psyche in order to solve for that well.
And then with a prompt you can tweak, "a reddit tier response", "an expert response", etc.
It kind of works already, but the more intelligent it gets the better it should get at acting different roles.
Moreover, you only get an answer for a question which has already been posed and answered, hence is represented by a pattern which the large language model can arrive at --- it's not possible to get an answer which wasn't already present in the training data.
You need a reward system or game. There are lots of professions and areas/topics of debate where one can just make up convincing nonsense. In a competitive area or one with quality feedback there is no need for the blabbering of fuzzy humans.
One of my 1000 business plans I will never execute is to build an automated chemistry lab where one can order different ingredients to be subjected to different processes, treatments and measurements. The researcher/customer would have no influence on the input or output and no humans in the building. I'm clueless about the scale to have a positive cost benefit analysis but if it can be good enough to throw things at the wall and see what sticks it seems an AI could be useful to pick the best ways to blow up the lab.
We might develop a selective corpus with say, the contents of the "Great Books" of western civilization and add the results of scientific history.
But the underlying problem in this is politics: everyone has a different idea of what is appropriate for that corpus (and consequently the resulting AI). Ergo the various brou-ha-ha's about "safety" etc. Indeed one assumption of these discussions may be correct: NN AIS may be as malleable, hard-headed or gullible as any human intelligence [and I don't know whether that is good or bad]. So many questions arise: "Should we let it read Karl Marx?", "What about St. Augustine?", etc.
Presumably we're modeling an intelligence akin to ourselves. We each occupy a single mind but the difference between minds can be great. The most familiar approach is therefore to develop an AI that is as much like us as possible.
We could also model many single minds with different corpuses and let them communicate, discuss et al as humans do. Maybe they would let us interact too.
FWIW I think you should be happy that any "intelligence" shown so far is of "Average Redditor" value. What would you do if you scattered some holy water on a pentagram in your upstairs living room, hurled out a diabolical incantation calling forth spirits and something akin to Satan himself appeared? That's (kind of) where we are with GPT.
The next revolution won't be a bigger model imo. It will be a way to run a gpt (at least 3.5) cheap and fast enough for real time text or vocal conversation on our phones or home devices.
Everyone will have a personal AI concierge, capable of consuming all their media and then re-displaying it for them with just the bits they want (no ads).
Dario Amodei, CEO and co-founder of Anthropic, recently said that AI capability has been following an exponential growth curve for the last ten years, a straight line on a log scale, and he hasn't seen any indication that the slope of that line is changing at all.
I would expect that he's seen capabilities that haven't been released to the public yet, and that those capabilities continue to follow that straight line on a log chart.
My joke since the 90's was that AI will have to wait for someone dumb enough to figure out how we think. I guess that means it is finally time for me to share my thoughts :P
My mind expressed annoyance with how sure LLM's are about everything. This is not a sign of intelligence - quite the opposite! I know people like that. The smartest sounding training data sounds the best I'm sure but intelligence is to be as accurate as possible about how sure you are. You zoom in on the facts, estimate their certainty then prioritize the data that matches the correct level of certainty. Humans hallucinate all the time, we call it imagination, it's great stuff as long as you present it as that.
A lot of people seem convinced otherwise, but to me GPT-4 is only a minor improvement (in best cases) over 3/3.5. Nowhere near the significant improvement after GPT-2. I'm assuming GPT-5 won't be any sort of breakthrough either.
Not that that would be hugely disappointing - all of this has been such a huge improvement in language modeling compared to the junky crap we had before.
> Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out.
I suspect filtering out all the junk will make them smarter and smaller, of course. But then we will take all that saved space and pack it back full of other modalities like audio and images, and the cycle will continue for a long time yet.
Ideally a LLM would have every text in existence in its training set, right?
Only if every text in existence provides good source material to factor into generating the response. Given that half of all texts are even worse than average, this seems unlikely, and Karpathy’s argument seems very reasonable to me.
The final paragraph from the original tweet, after the quote in the GP comment, mentions another interesting aspect. Even if we do have sufficient expert-level source material in a particular field to train a useful generative AI model and that in itself produces more desirable responses than training on a larger but more average-quality data set, is there still potentially useful information that could be extracted from the larger training set as well? How can we classify which aspects of a larger training set are desirable to keep while filtering out noise that competes with higher-quality source material? It feels like the progress of generative AI over the next few years might be defined more by these kinds of questions than just trying to build ever larger models using ever larger sets of training data.
They don't express concerns because they've already gotten their data, and retroactive regulation won't ever take it away from them. You will never be able to train on even a fraction of the corpus OpenAI has culminated.
We will never run out of training data. The idea that we will despite billions in investment is absurd. Simply scraping the web is the lowest level and there are millions of other ways to glean information about the world.
I am always reminded of Severance, the TV show, when I think of LLMs. Data refinement is what is happening massively right now. Basically, take data harvested on the Internet, which may be of poor quality, and use a teacher-student approach to reformulate it in higher quality. Then use that synthetic data to train new models. It's a bit like the biblical multiplication of the bread loaves. How much farther can it get LLMs? I don't know.
During the next turn, though, a lot of the same synthetic goop is going to turn up on the Internet. That's probably why OpenAI is spending the big bucks to secure access to original content producers. Now, here is to hope "original content producers" are not going to depend on LLMs to generate that original content.
I think training data would be plentiful if autonomous drones roamed the face of the earth, constantly recording audio and video. There is nothing as good as reality to learn what real is. The representation of reality on the Internet has become a matter of opinion for a lot of people.
There can be all kinds of strange and possibly perverse incentives at work for the “original content producers”, too.
Forums like LinkedIn have had regular listings in recent times looking for programmers to help train new models for answering programming questions and generating code. However, the rates offered are nowhere near enough to compete with real programming jobs for skilled developers, at least not in the major Western economies.
One wonders what quality level these new models will actually be trained at. Given that these roles are kinda-sorta asking professional programmers to train themselves out of a job, one also wonders whether 100% of those who do contribute will be doing so in good faith. And as you point out yourself, there is also a risk that the “original” content is anything but.
Given the scale of the data involved here, and the sheer amount of skilled human input that would be required to refine that data set manually, it seems unlikely that a Mechanical Turk strategy alone will solve many fundamental problems here.
One limit of LLMs I'm pretty sure about is they'll never be able to predict chaotic systems, i.e. systems with sensitive dependence on initial conditions. This includes things like the weather and the stock market.
They're not LLMs to my knowledge but AI weather forecasting will increasingly be a thing.
It's true that complex systems often can't be predicted at all e.g. you could design a pinball game such that the (deterministic) wiggling was arbitrarily large such that no floating point format could actually express anything, but AI as a tool for massive dimensionality reduction and pattern extraction does mean that they can make some huge gains in the cases where weather is merely difficult to predict using classical means
What is the concern? These businesses didn't exist two years ago. There's nothing to reckon. Google's had a wildly profitable model forever, their "existential threat" is LLMs not petering out.
At some point they're going to realise that they need to make cluster of LLM AI's all trained in specialities.
Then they will pass "thoughts" to each other, back and forth to self check and verify before outputting.
It'll be called something like LPM (Large Philosopher Model). It'll end up chewing up more compute and power than the bloody blockchain, but It'll be really impressive and scare the crap out of Turing testers.
What transformers have taught us is that gobbling up the whole internet can get you pretty far, but it doesn't change the fundamentals. Hallucinations still are trivial [1] to elicit. On the other hand, transformers are a software breakthrough that made training on the whole internet even feasible. Other methods (RNNs and LSTMs) were simply too slow. Transformers are a new hammer for an internet-sized nail.
If I can make an analogy to chess engines, a type of AI so old that people don't call it AI anymore [2], if we had started out by applying transformers to available chess games, we would never be able to create an engine that could reliably beat a grandmaster. Predicting the next chess move based on some representation of the game and its relationship with previously-seen positions only gets you so far. Deep Mind tried [3] this.
One route forward, which doesn't require a fundamental breakthrough, would be synthetic data. To stick with chess, we could create huge synthetic datasets of a huge amount of chess games, and simply continue training as we were. However, this is very expensive, and can be tricky if the synthetic distribution isn't the same as the real one. And the end result is still going to require incredible computation power to use.
What an LLM really lacks is the ability to search. And search is really important [4]: without it you'll probably need orders of magnitude more data and orders of magnitude larger models. At a fundamental level, search means we don't just make a snap judgement about e.g. the likely next token, but, as Daniel Kahneman would say, "thinking slowly" about what to do/say next.
Assuming we will not have search is how you get projections like needing 100 GW power plants to supply dedicated training data centers. The human brain uses 12 watts. 100 GW is the energy requirements of all human brains on the planet. And yet, if we put all human brains on the planet together, our collective capabilities would be outclassed by a chess engine I can run on my phone.
We have barely scratched the surface of the algorithms that can satisfy all the attributes of significant learning. https://www.buffalo.edu/catt/teach/develop/design/learning-o...
Although I believe, the much more serious issues are around the impact of deploying AI systems on the Human agency. These algorithms could potentially make any kind of change harder and could seriously cap human potential.
91 comments
[ 2.7 ms ] story [ 175 ms ] threadI strongly disagree. Wildly overvalued companies disrupt the economy and present huge risks - see Tesla or Nvidia (currently valued at $100m per employee - $3T market cap / 30k employees)
My hot take is that UX is the biggest limiter for current LLM products other than chatGPT. Chat has too much friction for many use cases, we need to find better interfaces that are more visual and faster to interact with. Spending $100M to gain .1% on MMLU is a waste of time in comparison.
Asking for what you want (a.k.a: chat) is too much friction?!?
Until Google decided to trash their search with their own sub-par LLM, my results were often regularly better just hitting the first link on Google Search like I had for the last decade.
Given infinite possibilities to enter into a text box is more friction than picking from a menu in front of you.
There's a lot of use for LLMs, and it's tossing the ingredients in a restaurant's kitchen together but sometimes people want a menu.
Just saying you want a chicken florentine isn't enough. You may also have to specify that the chicken is fresh but also not still alive, and that kale isn't the same as spinach, and confirm that they have the correct recipe for mornay sauce even though every chef should already know that. And even if you do all that, you may still get a chicken cordon bleu for reasons you cannot explain and neither can the kitchen.
When it's free/cheap, and I'm up for an adventure and sending stuff back to try again a few times, it's not bad. If I'm having to pay a pretty high amount for it and still treat learning the system like a second job, I start to have questions about why I'm bothering.
There is no problem with the chat UX, if you know the right techniques to get optimal output, but writing effective prompts is a technical skill similar to programming - most folks that interact with computers are not highly technical.
If we design software powered by LLMs understanding that effective prompting is a skill most users don't have, then we should build UX that abstracts prompting away. Of course there are going to be use cases where a user needs to provide input to the LLM, but that UX also can be assistive and require minimal cognitive overhead (e.g less raw text prompt input, more clicks on buttons/forms etc to a backend prompt builder).
It works very very well on Amazon reviews, though, because these are small and thin.
To me there is still so much left to do.
And even if we truly did multiple laps over all the data in the World there are still different new architectures and strategies left to try.
your comment really belies the desperation that exists now, these models are stuck where they are (hint it’s a natural limit), you are talking about exponentiation of cost to get what a 10% improvement? 5%? They are very few places for which it’s net positive to run them now, and most of those are incredibly shitty things like creating trash marketing content to drown us all in average inanity
I really feel bad for this next generation, they will just be constantly inundated with generated crap, so much of the high fidelity of conversation and meaning is and will be lost.
And I am not talking about predicting the future, but more predicting the next action to take based on current state, sensor data in a more seamless way. Like a human being reacting to different input, by moving their muscles etc. There would be huge amount of training data from there that could be incorporated into a single model.
Like self-driving cars?
Self-driving cars is an engineering problem, let alone an AI problem, and we still cannot solve it despite trillion dollar economic incentives.
Just putting together some LLMs on a fuckton of data does not work. Tesla tried that, and failed.
Completely different solution applied to a completely different problem with completely different risk and quality tolerances with completely different mitigations.
Given that LLMs are inaccurate around 5-10% of the time each step will compound the error rate until you are better off flipping a coin.
You can ask them to do math equation which takes steps and if they are trained in that for certain problems they are accurate near 100 percent of the time.
Like ask gpt-4o to solve different variations of
"""What is the answer to 2x + 7 = 31?"""
If the numbers are of similar magnitude and simplicity, it will follow the same steps and be right 99%+ times, and I'm only not saying 100%, because I haven't tried it enough, but I don't see it being wrong.
For example """What is the answer to 2x + 4 = -6?"""
Just run a test yourself. Do random integers within 0 - 20, it will definitely not be incorrect 5% - 10% time. It will be correct 99%+ time.
Where is this number 5% - 10% even coming from? You could also keep asking it "What is the capital of France?" and it's going to be right 99%+ of the time.
And the 5-10% is on average and gets significantly worse as you expand the context length which is also something you want for an agent.
Based on what you are attempting to do you could get any average in the end.
That happy discovery was never really a linear improvement path, though. We had an explosion of capability, but all along there have been active questions about how far the improvements would go with the current approach.
I think the point that a lot of researchers are making is that that we're starting to see those limits (with LLMs, at least).
There are also a lot of questions around business model and cost/value prop. Training and running these things at scale is enormously expensive. I'm seeing a lot of FOMO and gold rush mentality in the space, similar to the online streaming wars, and I'm not convinced of the long term viability of a lot of the companies. Especially once open models like llama are "good enough" and become commodities.
Of course, it's still early days and there's a ton of room for discovery, but it looks like we'll hit a limit with the current approach pretty soon.
Personally, I'd be OK with that. With the current state of things we have an interesting toy that can sometimes do useful work. It's an incremental quality of life improvement and another good tool in the chest, but it's not a civilization impacting technology.
That's probably for the best.
And only after 1.5 years? And especially of we just had an happy surprise like you mentioned. How does it make sense to already start claiming that we have hit the limits. How do we know there is no more scaling, optimisations and happy surprises?
> That happy discovery was never really a linear improvement path, though. We had an explosion of capability, but all along there have been active questions about how far the improvements would go with the current approach.
> I think the point that a lot of researchers are making is that that we're starting to see those limits (with LLMs, at least).
The kinds of limitations we're "starting to see" are largely the same as they were a year ago. People were talking about it on here back then, but now it's becoming more apparent to more people as they get used to LLMs.
For those who saw it back then, this does look like we're hitting a limit. For others, not so much.
How do active questions about a technology imply we are approaching a brick wall?
How could researchers without having access to the latest state of the art - by OpenAI or any other unknown companies be able to even test that we could be approaching a brick wall? It seems to me that it would take trillions to find out what the exact limit is.
It's possible that we will get diminishing returns, but I don't see how we can confidently claim or know it?
> The kinds of limitations we're "starting to see" are largely the same as they were a year ago. People were talking about it on here back then, but now it's becoming more apparent to more people as they get used to LLMs.
I don't follow. GPT-3.5 was borderline useless at reasoning. But it still seemed amazing and what I wouldn't have thought to be possible in any near future.
And then GPT-4 was a crazy advancement over that to me. And I've been using it daily since it was available, for various use-cases. Are you saying we are seeing the limitations of GPT-4 specifically? Because, sure, GPT-4 is far from AGI, but I don't see how this implies that further scaling, optimisation, training data improvements, techniques like multi modality and other potential strategies that I might not be aware of couldn't bring another explosive step?
Also the fact that GPT-4 reasoning skill hasn't been reproduced by anyone else so far seems to leave me thinking that everyone except OpenAI are clueless. Claude Opus is close, like I've said before, but not quite GPT-4 levels in specific reasoning tasks that I'm using the API for.
If you can't reproduce GPT-4, how could we trust the assessment that we have hit a limit?
Not sure what that means. Why are you marking those as "quotes".
The last actions brought so many returns. And it's unknown what the exact effect would be in adding more modalities, training data, optimisations and even just plain parameters.
Text as training data can only get you so far. Giving real time sensory data from many fields could allow LLM like system to control robots and get even more data from real life. E.g. robot hand movements, object tracking data, all of that to be fed into LLMs, and see how it would work.
- something that can't be modeled because there's no training data
- something that can't be modeled because it's fundamentally stochastic
- something that can't be modeled because the discrepancy in simulating the generating process, for your specific model, can, basically, be made arbitrarily large
I think a very common error when it comes to personal learning or progress is confusing a plateau with a brick wall. The reason is, unless you have already walked the path, it’s not possible to differentiate them. And when it comes to progress, no one has already walked the path, hence no one knows actually.
What if it already did, and it's called GPT4-o? Like sure, OAI realized it was a mostly marginal improvement over 4 after it was finished. But did they know that ahead of training?
I would have expected gpt-5 at least initially to be much, much slower, and they have only recently talked about starting on it.
https://situational-awareness.ai
He did a very long interview with Dwarkesh Patel, too:
https://www.youtube.com/watch?v=zdbVtZIn9IM
We need calmer, more cautious heads to prevail so AI is more thoughtfully and safely incorporated into products.
The whole e/acc, AGI cult is not helping anyone.
Also, what specific safety problems do you think need to be solved?
They are the ones on the sidelines making money selling courses, going on podcasts, getting engagement money from X etc.
Most of the guests on Dwarkesh’s podcast are working full-time in the industry, many of them in extremely senior.
Ahh, the old "how can cryptocurrency be harmful if people like buying it?" argument.
There are many realistic and amenable goals in life, like mapping the human genome or writing an Open Source microkernel. Declaring that you intend to surpass human intelligence via a statistical text generator is not one of those things. It does not have precedent, it does not have feasibility studies, it does not seemingly indicate in any way shape or form that it is possible. Nobody can even spell-out the intermediate steps to get us to AGI; every single "novel" solution involves scaling up our current, broken, concepts. It's ELIZA versus the Lisp pundits all over again.
Excitement and hard work go a long ways, but you won't know which way until you apply a little logic. The current "AGI" trend is practically non-existent outside the venture-capital sphere and OpenAI employees, both of whom would be bullish on AGI anyways since it's good for business. Once you discard the biased opinions, you're left with legitimately confused investors and nonsense opinions propagated by conspiracy theorists. Much like crypto, the concept of "AGI" is being used to confuse and exploit people who misunderstand technology and finance.
> Also, what specific safety problems do you think need to be solved?
I think you misread their comment. They said "safely incorporated", which is not a specific safety problem for AI but a holistic consideration that stops your product from sucking. Your computer vision model could be statistically perfect, but absolutely useless for self-driving tasks and multimodal robotic agents. It's not taste that separates these good implementations from the bad ones, it's logic. You have to be considerate when implementing AI in traditional systems, because inherently AI can be wrong and you must have a failure-mode for those situations.
Many people reject this idea, because it precludes the idea that someone could sell a cure-all to today's AI ills. But real AI safety cannot be baked-in to a model. It only exists when genuinely thoughtful humans anticipate every single fail-state; if that sounds like hard work, it's because it is. And nobody, nobody, sells it as "AGI".
Not a fan of e/acc but seems like you’re just labeling rather than adding anything important to the conversation, ironic because you seem to stress the importance of calmer and cautious heads.
Notes:
- What's the difference between "superalignment" and regular "alignment", anyway? OpenAI uses the term, but mostly for marketing purposes.
- Alignment to what?
- Has "hallucination" been fixed yet?You'd have to hope for some sort of emergent intelligence or knowledge-breadth integration based intelligence I suppose. But I already get annoyed by ChatGPT 4o for having Average Redditor Tier ideas and responses.
And then with a prompt you can tweak, "a reddit tier response", "an expert response", etc.
It kind of works already, but the more intelligent it gets the better it should get at acting different roles.
One of my 1000 business plans I will never execute is to build an automated chemistry lab where one can order different ingredients to be subjected to different processes, treatments and measurements. The researcher/customer would have no influence on the input or output and no humans in the building. I'm clueless about the scale to have a positive cost benefit analysis but if it can be good enough to throw things at the wall and see what sticks it seems an AI could be useful to pick the best ways to blow up the lab.
But the underlying problem in this is politics: everyone has a different idea of what is appropriate for that corpus (and consequently the resulting AI). Ergo the various brou-ha-ha's about "safety" etc. Indeed one assumption of these discussions may be correct: NN AIS may be as malleable, hard-headed or gullible as any human intelligence [and I don't know whether that is good or bad]. So many questions arise: "Should we let it read Karl Marx?", "What about St. Augustine?", etc.
Presumably we're modeling an intelligence akin to ourselves. We each occupy a single mind but the difference between minds can be great. The most familiar approach is therefore to develop an AI that is as much like us as possible.
We could also model many single minds with different corpuses and let them communicate, discuss et al as humans do. Maybe they would let us interact too.
FWIW I think you should be happy that any "intelligence" shown so far is of "Average Redditor" value. What would you do if you scattered some holy water on a pentagram in your upstairs living room, hurled out a diabolical incantation calling forth spirits and something akin to Satan himself appeared? That's (kind of) where we are with GPT.
The current digital ad industry is doomed.
“it is difficult to get a man to understand something, when his salary (or company valuation) depends on his not understanding it.”
- Upton Sinclair (with edits)
Math benchmark:
Minevra, Jun 2022: 50.3%
Opus, Mar 2024: 60.1%
And there is a high chance it leaked to Opus training data, since it is old and now popular benchmark.
My mind expressed annoyance with how sure LLM's are about everything. This is not a sign of intelligence - quite the opposite! I know people like that. The smartest sounding training data sounds the best I'm sure but intelligence is to be as accurate as possible about how sure you are. You zoom in on the facts, estimate their certainty then prioritize the data that matches the correct level of certainty. Humans hallucinate all the time, we call it imagination, it's great stuff as long as you present it as that.
Not that that would be hugely disappointing - all of this has been such a huge improvement in language modeling compared to the junky crap we had before.
Has anyone seen representatives of the cutting-edge AI labs - OpenAI, Anthropic, Meta, Mistral etc - express concern about this?
The impression I've been getting is that quality turns out to be much more important than quality. This tweet from Andrej Karpathy for example: https://twitter.com/karpathy/status/1797313173449764933
> Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out.
Only if every text in existence provides good source material to factor into generating the response. Given that half of all texts are even worse than average, this seems unlikely, and Karpathy’s argument seems very reasonable to me.
The final paragraph from the original tweet, after the quote in the GP comment, mentions another interesting aspect. Even if we do have sufficient expert-level source material in a particular field to train a useful generative AI model and that in itself produces more desirable responses than training on a larger but more average-quality data set, is there still potentially useful information that could be extracted from the larger training set as well? How can we classify which aspects of a larger training set are desirable to keep while filtering out noise that competes with higher-quality source material? It feels like the progress of generative AI over the next few years might be defined more by these kinds of questions than just trying to build ever larger models using ever larger sets of training data.
During the next turn, though, a lot of the same synthetic goop is going to turn up on the Internet. That's probably why OpenAI is spending the big bucks to secure access to original content producers. Now, here is to hope "original content producers" are not going to depend on LLMs to generate that original content.
I think training data would be plentiful if autonomous drones roamed the face of the earth, constantly recording audio and video. There is nothing as good as reality to learn what real is. The representation of reality on the Internet has become a matter of opinion for a lot of people.
Forums like LinkedIn have had regular listings in recent times looking for programmers to help train new models for answering programming questions and generating code. However, the rates offered are nowhere near enough to compete with real programming jobs for skilled developers, at least not in the major Western economies.
One wonders what quality level these new models will actually be trained at. Given that these roles are kinda-sorta asking professional programmers to train themselves out of a job, one also wonders whether 100% of those who do contribute will be doing so in good faith. And as you point out yourself, there is also a risk that the “original” content is anything but.
Given the scale of the data involved here, and the sheer amount of skilled human input that would be required to refine that data set manually, it seems unlikely that a Mechanical Turk strategy alone will solve many fundamental problems here.
It's true that complex systems often can't be predicted at all e.g. you could design a pinball game such that the (deterministic) wiggling was arbitrarily large such that no floating point format could actually express anything, but AI as a tool for massive dimensionality reduction and pattern extraction does mean that they can make some huge gains in the cases where weather is merely difficult to predict using classical means
>GraphCast: AI model for faster and more accurate global weather forecasting https://deepmind.google/discover/blog/graphcast-ai-model-for...
I think you inherently don't want a language model for modeling fluid flow. A vector model perhaps.
Then they will pass "thoughts" to each other, back and forth to self check and verify before outputting.
It'll be called something like LPM (Large Philosopher Model). It'll end up chewing up more compute and power than the bloody blockchain, but It'll be really impressive and scare the crap out of Turing testers.
If I can make an analogy to chess engines, a type of AI so old that people don't call it AI anymore [2], if we had started out by applying transformers to available chess games, we would never be able to create an engine that could reliably beat a grandmaster. Predicting the next chess move based on some representation of the game and its relationship with previously-seen positions only gets you so far. Deep Mind tried [3] this.
One route forward, which doesn't require a fundamental breakthrough, would be synthetic data. To stick with chess, we could create huge synthetic datasets of a huge amount of chess games, and simply continue training as we were. However, this is very expensive, and can be tricky if the synthetic distribution isn't the same as the real one. And the end result is still going to require incredible computation power to use.
What an LLM really lacks is the ability to search. And search is really important [4]: without it you'll probably need orders of magnitude more data and orders of magnitude larger models. At a fundamental level, search means we don't just make a snap judgement about e.g. the likely next token, but, as Daniel Kahneman would say, "thinking slowly" about what to do/say next.
Assuming we will not have search is how you get projections like needing 100 GW power plants to supply dedicated training data centers. The human brain uses 12 watts. 100 GW is the energy requirements of all human brains on the planet. And yet, if we put all human brains on the planet together, our collective capabilities would be outclassed by a chess engine I can run on my phone.
[1] https://arxiv.org/pdf/2406.02061
[2] https://en.wikipedia.org/wiki/AI_effect
[3] https://arxiv.org/pdf/2402.04494
[4] https://www.youtube.com/watch?v=ceCg90Q9N6Y&t=1200s