AI is the cutting edge of CS. When it's well understood it stops being AI. Some day LLM and neural nets will get the same reaction... who calls LLM "AI"? It's just a formula!
Is it really though? I feel like it's mostly non-technical people who think this, because back when I was a CS student, AI was just a big field of study that included many types of problem solving, from simple to complex, including all types of search such as A*.
I first learned it in the famous book "Artificial Intelligence: A Modern Approach" by Peter Norvig. I think 3 chapters are dedicated to search. A lot of AI can be framed as search, and a whole lot of historical AI, or mundane present AI, is straightforward search.
By way of [1], here [2] is a paper about A* also posted under CS/AI. It introduces "Q* search, a search algorithm that uses deep Q-networks to guide search" and uses it "to solve the Rubik's cube", the thing which Gary Marcus complained about OpenAI not having done with a neural network in 2019.
As pointed out by techbro92 [3] the authors do not seem to be affiliated with OpenAI, but AFAIK nobody claimed that OpenAI invented Q*. It's not hard to imagine OpenAI trying it out with their Rubik-solving robot hand and then starting to think about other things it might be applied to.
I still think the "AI effect" is bullshit. A* was/is AI under a different meaning - it's what understood as "AI" in computer games, where the point is for the computer opponent to pretend to be smarter than a brick, and literally any trick, cheat or heuristic used to do that is part of "AI". A*? Obviously AI. Random walk? Yes. Fuzzy logic? Yes. State machines? Yes.
True that A* would later go on to be used in computer game bots (as LLMs may eventually be) where "an AI" doesn't necessarily correspond to the field of AI, but A* was still within what was understood to be the field of AI at the time it was published.
The meaning shifting over time, and the reason for which it shifts, is the AI effect. Heuristic-guided pathfinding is no longer what's meant by AI, IBM Deep Blue's alpha–beta search is no longer what's meant by AI, and I'd claim it's even shifting away from earlier image classifiers in favor of only generative models like GPT4.
Probably because of A*'s popularity for path finding in game development. Everything that imitates human-like behaviour in games is called "AI", even when most of it is just a big messy hairball of adhoc if-else code under the hood ;)
There's no magic in 'AI', all of it can be described as something else. The whole of ML is 'just statistics', especially classification.
(That said I was and I suppose am a bit of a sceptic because of that. But mainly I think because to me it's more useful to call a spade a spade and say we're doing some advanced blah stats to blah. Helps explain, onboard, and motivate the next generation (who likes stats at school? But if that's how AI works, oh suddenly it's cool). Demystify it. If you're just searching over Levenshtein distance [to a technical audience] say that, don't start waffling about 'AI'.)
I'm trying to refresh mind on what Q learning is, if I may think out loud here?
Q is an evaluation of how good a state is assuming the agent acts optimally going forward. In Chess for example, sacrificing a rook to capture a queen would have a high Q value. Sacing a rook is bad, but gaining the queen later is better. Q values are supposed to see past immediate consequences and reflect the long term consequences.
How do we find a Q value though? We can look several steps ahead, or we can look to the end of the game, etc. These aren't real Q values though, because optimal actions probably weren't taken. We can use Bellman equations, which roughly update one Q value by a small amount to the maximum of the next possible states, which results in the highest Q values gradually flowing backward into states that can lead to good outcomes.
I'm trying to think how this would apply to LLMs though. Where do the basic values come from? In Chess the basic values are the number of pieces, or whether or not a side ultimately wins. What are the basic values in a LLM? What is the Q agent aiming to do?
In computer chess, traditionally 'minimax' was the strategy that allowed you to efficiently probe ahead. The only requirement for this is that your evaluation function returns a single (scalar) value.
I don't think your intuition about computer chess is going to help you here with transformer architecture.
Usually in transformer models[1], for each attention head there are 3 vectors of weights, known as q(uery), k(ey) and v(alue). I was assuming that the q in q* applied to the q vector, so q-learning is training this vector. In a transformer model you don't have an objective function for any sort of state evaluation so it can't be the Q you're thinking of.
If they've done something funky with the q vector that could indeed be a breakthrough since a lot of people feel that we are sort of running out of juice with scaling transformers as-is and need a new architecture to really have a step change in capability. That's pure speculation on my part though.
Current architectures have a lot of juice left across several axes, using extraordinarily accurate scaling laws. We're nowhere close to a wall.
The industry has been focusing hard on 'System Two' approaches to augment our largely 'System One'-style models, where optimal decision policies make Q-learning a natural approach. The recent unlock here might be related to using the neural nets' general intelligence/flexibility to better broadly estimate their own states/rewards/actions (like humans). [EDIT: To be clear, the Q-learning stuff is my own speculation, whereas the 'System Two' stuff is well known.]
How do you apply Q-learning to LLMs? They would need to estimate the action-value of each action in a given state. How can you pick the best action when you only have a scoring function that would rate how good each action is, but the action space is infinite. You can't pick the best action.
It is probably something else, like Decision Transformer.
If I remember correctly DQLearning is using discrete action space, limited number of possible actions which makes it possible to simply take the argmax_i(qvalue(state, action_i))
They’re not assuming that the Q refers to the Transformer’s “query”, they’re talking about the reward function in reinforcement learning, using chess as an example.
It's probably something to do with path finding with constraints in high dimensional search spaces I'd guess. Like quake bots used to do in the late 90s
> Q is an evaluation of how good a state is assuming the agent acts optimally going forward.
The Q-value represents the expected utility or the total reward of taking a certain action in a given state, followed by following an optimal policy thereafter. So, yes, it does evaluate how "good" or beneficial a state is, but specifically in conjunction with a chosen action and under the assumption that the agent will act optimally in the future.
So it is conditioned on state and action, not just on state.
I am surprised by how dismissive the whole post sounds. For example:
> OpenAI could in fact have a breakthrough that fundamentally changes the world
Well, it appears to me that OpenAI already has such a breakthrough- it had it roughly 4 years ago with GPT2, and it's still scaling it.
Considering that it's not yet a year since the introduction of the first ChatGPT, and given the pace at which it's evolving, I would say that the current product is already showing great promise to fundamentally change the world. I would not be surprised if just incremental changes were enough to fulfill that prediction. The impact at this point seems more limited by the ability of society to absorb and process the technology rather than intrinsic limits of the technology itself.
Skeptic and completely reactionary. I had to unfollow him on Twitter because he always has to have a "take" on every AI headline, and he's often contradictory between "AI is useless" and "AI is a huge threat".
The people behind LLMs believe in it so thoroughly that they are pushing it to do things that aren’t safe. So it can easily be true that it’s both overhyped and needs better regulation. The fact that LLMs can’t actually solve the problems they are being used for is, in fact, a problem that may require government intervention.
If LLMs aren't fit for the types of problems they're being used for, what would the government be needed for? Users would pretty quickly give up and move on if LLMs continue to suck at solving the problems people want them to solve.
Not if those people are "true believers". Once you get emotionally attached to the idea that something must work, you don't respond to failures rationally.
If we're delving into the realm of belief and therefore religion, pulling in the government to shut it down goes against everything that America was (supposedly) founded on.
>I am surprised by how dismissive the whole post sounds.
I wouldn't be surprised, it's Gary Marcus. He's an academic with a lot of prestige to lose if the LLM approach is actually good/useful/insightful, who's only widely publicly known now because AI has had a backlash and media needed an expert to quote for "the other side". Same as the computational linguistics researchers who always get quoted for the same reason.
In general, academics in competing fields whose funding threatens to get tanked or eclipsed by research approaches that work on principles they have fundamental academic disagreements with are going to talk negatively about the tech, no matter what it is achieving. Where I think it can be valuable to listen to them is when they're giving the technology credit - generally they'll only do that when it's something really undeniable or potentially concerning.
> it's Gary Marcus. He's an academic with a lot of prestige to lose if the LLM approach is actually good/useful/insightful
I wouldn't bet on loss of perceived expert on the other side status or prestige no matter how poorly his gloomy LLM forecasts fare. It hasn't happened from any of his previous consistently pessimistic predictions around deep learning since the beginning of its resurgence (which has clearly far surpassed its state in 2012, even if currently overhyped). To quote Gary Marcus quoting Gary Marcus:
> Yet deep learning may well be approaching a wall, much as I anticipated earlier, at beginning of the resurgence (Marcus, 2012)
This is such a case of a bad-comment that seems clever and insightful. It boils down to saying we don't need to debate or even consider the content of his arguments because we can assume he's only motivated by prestige and money (but without considering the second-order effects on his credibility and funding if he actually turns out to be proven substantially wrong in the future).
I don't know how right or wrong he is - none of us do. That's why it's all still being debated.
The one thing I know is that we can only truly understand a topic by fully understanding arguments for and against all the claims. I also know the pro-LLM set have way more money (double-digit billions as we saw just this week) and credibility to lose over this topic than Gary Marcus does.
I completely understand your comment, it's reasonable to want sources. I didn't have enough time when I made my original comment to do some good sourcing, but I had that time now. In the interest of not making this comment absurdly long and not repeating work that's already been done, I'll try to be brief and link to secondary sources that contain accurate analysis where possible, rather than linking to primary sources with mostly my own analysis.
>but without considering the second-order effects on his credibility and funding if he actually turns out to be proven substantially wrong in the future
He has continuously made predictions about how bad deep learning is and how it would never achieve much, unlike his "much better" neurosymbolic theories that have never shown results anywhere near as significant (there might be good ideas there, doesn't mean that his continuous goalpost shifting is good science). Those predictions have continuously been proven wrong for the last decade, including a number of predictions specifically about LLMs. This is relatively normal in the academy, people have hobby horses, but he's now engaging in public discourse and advocacy about an actually contentious topic and there are different standards here, so I'm going to respond in that context. If you're not familiar with him at all, that's fine, here's some of his track record and explanations of why he's wrong:
For deep learning in general, in case you want an overview of where he's coming from (he stands by everything in this article today, don't let the age inspire sympathy, he's a deep learning skeptic to the core): https://www.newyorker.com/news/news-desk/is-deep-learning-a-...
CTRL-F "Gary Marcus" on this page to find a bunch of sourced, specific claims from Marcus alongside evidence they are wrong: https://gwern.net/gpt-3-nonfiction
He has never stopped, and probably never will. He's the student of the original guy who first buried neural nets back in the sixties or seventies; he's royalty in symbolic AI. The only concession I've ever seen him make to the continued progress of deep learning and LLMs in spite of his continuous early proclamations of death is, bizzarely, that he is concerned about existential risk from current AI research. I don't personally see how he squares "Deep learning and LLMs are all nonsense, have hit a wall, off-ramp on the road to AGI" with "Current AI research [which is overwhelmingly deep learning, even if he doesn't say that part out loud] presents an existential risk", but he does it somehow.
Basically, no matter what new AI research or artifacts come out, if they are made with neural nets he will explain:
1. They are not that important
2. It's a dead end
3. They're harmful because people might wrongly get the impression that they work
4. They're nowhere near as good as his favoured AI architectures are, if only his favoured theories had as much funding as that broken deep learning crap
OK, sure, it's a better answer given that it does actually try and address his arguments and includes sources.
But then you've included some very unfair/invalid lines of attack. You've used quote marks when attributing arguments to him, but these are not actual quotes. You've asserted that he argues that "current AI research ... presents an existential risk" - which he recently said he specifically doesn't believe [1].
For the record, I've been aware of Gary Marcus and his positions on AI research since he was on EconTalk in 2014 [2]. I don't consider him the most compelling skeptic on contemporary AI, but I find it beneficial to hear and consider what he has to say. I'm also very familiar with ‘Mr Wern’
and his approach to discussing this and other topics. I've personally debated him on this website over a different topic, and whilst I benefited by learning new things about that topic, which was good, I was also surprised how much somebody of that persuasion would resort to emotion-charged, flawed or fallacious arguments when pressed.
And this is the main issue I'm wanting to confront: whether it's in this thread, or on LW/OB, or Sam Altman himself (in this tweet [3] that to me was a pretty alarming mask-slip for someone who has worked so hard to portray himself as a good operator over the years), it's striking how quickly the arguments turn personal and nasty rather than just pointing to simple proof that he's wrong. Which is just further evidence that discussion about this topic is so much more driven by emotion and grandiosity than sober examination of evidence.
As for Gary Marcus, sure, for someone who speaks/writes publicly a lot, you can always find specific details on which they might be wrong at a point in time. And where he's clearly factually wrong he should be corrected and he should welcome that. But I'm yet to see any slam-dunk evidence that the central position that he's continued to argue and explain for years is wrong.
Indeed, this quote from a post of his from 2018 [4] seems to be identical in substance to what Sam Altman said at Cambridge this very month [5]:
"Possibly. I do think that deep learning might play an important role in getting us to AGI, if some key things (many not yet discovered) are added in first."
I don't know, it is an opinion and it seems kind of well-founded (i.e. there's no evidence for groundbreaking research on OpenAI part except for scaling things up).
It seems to me that the breakthrough claim was a desperate attempt of OpenAI staffs to get their beloved boss back to OpenAI. If anything it most probably will be incremental things rather than a breakthrough but definitely not a bad things but just call a spade, a spade [1]. Ironically the breakthrough happened somewhere else in Google and for some unknown reasons as of now, Google has missed the boat of commercializing its very own invention.
Personally I have been recently have been using the ChatGPT-4 on a daily basis, and I considered myself a long time and ardent user of Google products (mainly search) for the past two decades. However it becomes increasingly frustrating for the past several years since it is getting more difficult to perform "search that matters" (going to trademark this motto). What frequently irked me the most is not when doing random or targetted search with Google, is when looking for something that you discovered previously but it's forgotten and apparently it does really matter now. It was quoted that about 10% of our time looking for items that we lost or misplaced, and the same can be said to our acquired knowledge and information. Some times we crave so much for the knowledge or the info we had but very difficult or impossible to recall. ChatGPT-4, in particular the online search with Bing feature, is extremely useful in this regard but at the same time is very limited since it is not a well supported features with many failed attempts, perhaps due to limited online data scrapping capability, and thus sub-par results. This specific feature, I call it search that matters, is like Google search with steroid and the fact that Google, with its Deepmind subsidiary, failed to utilize and monetize on this opportunity until now is just beyond me. If ChatGPT, call it ChatGPT-5 if you like, can perform this operation intuitively and seamlessly it will be a game changer but not a breakthrough. Apparently according to ChatGPT-4 you can have a game changer not a breakthrough.
The breakthrough, however, is to fundamentally improve AI or LLM itself as mentioned by Stephen Wolfram in his tutorial article on ChatGPT, not merely enhancing its existing operations [2]:
When it comes to training (AKA learning) the different “hardware” of the brain and of current computers (as well as, perhaps, some undeveloped algorithmic ideas) forces ChatGPT to use a strategy that’s probably rather different (and in some ways much less efficient) than the brain. And there’s something else as well: unlike even in typical algorithmic computation, ChatGPT doesn’t internally “have loops” or “recompute on data”. And that inevitably limits its computational capability - even with respect to current computers, but definitely with respect to the brain.
[1] Eight Things to Know about Large Language Models:
Must agree with you. Even when GPT-2 not impressed me much, but with GPT-3 and GPT-4 we constantly see significant progress, not seen on near every other technologies.
Even if GPT-5 will be fatally expensive, but still show such leap as GPT-4, it will be enough to consider OpenAI most successful AI research in human history.
And for about expenses, we just need to wait some 10 years, and GPT-3 class tech will be in every smartphone, and in 30 years, GPT-4 will become also affordable for everyone, GPT-5 will be just ordinary business workhorse.
One thing OpenAI has now that it didn't have 4 years ago is a lot more compute power at its disposal. Sam Altman has already said "I think we're at the end of the era where it's going to be these, like, giant, giant models." If that's actually true, then the GPTX tech has largely hit a wall in which throwing more compute at it won't get the same increase in capability. Bill Gates predicted that GTP5 won't be much better than GPT4. So, any "breakthrough" really could be incremental rather than game changing.
I think there are two main source of learning for AI - the web scrape datasets, which contain our historical experiences and communications, and AI feedback generated from deployed agents. The web text is almost exhausted, or we can't scale it 100x more, but the feedback is just starting to ramp up.
Every day millions of chat sessions are recorded, and they are exceptional training examples. They would contain the kind of errors LLMs do, and the kind of demands people have, and include a human reaction to each LLM message.
The OpenAI move to create "GPTs" is showing they are actively working on improving the feedback signals by empowering the LLM with RAG, code execution and API access. In such a setup it is possible to use a model at level N to generate data at level N+1.
The keyword here is learning from feedback, which aligns with recent talk of using RL methods like AlphaZero with LLMs. A RL agent would create its own data as it goes. I think progress will be gradual, as we need to wait for the world to produce the learning feedback signal. Of course in domains where we can speed that up, AI will progress faster.
Interesting thought: by making LLMs available to the public, they are going to assist people in many ways and create effects that will percolate in the next training set: LLM inference -> text -> effects in the world -> text -> LLM training. So there is already an implicit feedback loop when we retrain the base models. GPT-5 will train on data from a world influenced and shaped with GPT-4.
The corollary of this is optimistic though. If increasingly large models don't make a big difference, this is hinting at the fact that data quality is a lot more important than the raw quantity of data. This is good news for open source models, because it would be possible to run or train viable models on less expensive hardware, and having billions of dollars in GPUs isn't as much of a moat as they are suggesting.
I used to be friends with some guys who made a rubiks cube solving robot at college years ago. It was a cool novelty back then, but is unimpressive for OpenAI to boast about now. That robot could solve a cube that wasn't specially instrumented like openai's is here as well.
Just as in my parent post. Q learning is kinda old tech for AI. It works rather well though and is also computationally cheap to run. An old laptop could do it with ease.
What problem is AI going to solve? This question is being carefully avoided for some reason, and many seem to believe that AI is like air or water, which is good by itself.
Automated software writing from a codebase and a requirement description, with the model asking clarifying questions where necessary, would be an immensely valuable application. We're quite far away from that, but GPT-4 is already rather helpful for some coding tasks.
In this specific use case though it sucks the most since one will be answering more questions than code needed in sheer volume. Because the code is already straightmost clarification to the initial question. We will never be closer than looking up what you don't understand to solve your problem in the first place and it will always take more time than directly solving it.
I had a good example recently and I'll quote it again:
People seriously using the "Please write an expression for me which adds 2 and 2" prompt in order to get the "2+2" expression they need – advocating that they got it with magical efficiency
And at the same time most manual forms of human labor.
Object recognition is AI. A decade ago we were piss poor at Object recog, today labeling AI is at human level. Being able to recognize your environment is the first step in navigating it effectively.
I have a hard time being happy about it. A lot of people - including me - find pride in solving problems, and selling this as a valuable skill. When it's no longer the case, when the only valuable skill becomes just being a presence in the physical world an act on it (but without the problem solving part), it will really feel like we are enslaved by the AIs.
In a sense, we kind of already are already, since out internet feed is (for most apps with recommendation algorithms) just spoonfed to us, rather than the product of our own search.
To be honest, I wish the progress will stop and that AGI won't happen. That possibility scares me.
Don't you ever get up at 8AM and think "ugh not again"? Surely you're not a fan of the 40h work week? I totally agree with everything else and also find it scary, but defending work for the sake of it seems like the wrong angle to take. There will always be opportunities for productivity, regardless of whether that productivity is a requirement for food & shelter or not.
Of course I do, but at least I kind of chose a path that I kind of like. You are assuming that with a powerful AGI, we won't work 40h a week, but what if you'd still work as much, just to other positions that you don't like ? Do you think people in charge will give everyone leisure time, or do you think they'll want to keep extracting as much value as possible from people ? I think the latter ;).
I am a creative independent and I certainly don't work 40h a week...more like 10, but I'm very happy with creating things and I would be very sad if AI took my job away. I hope AI burns.
Just curious who you think is avoiding the question? As if it’s a conspiracy? Every single leader I know that is involved with AI talks about it all the time.
Superintelligence can solve all the things we’re struggling to solve. All human diseases, Earth being destroyed, fundamental questions about the universe, etc., etc. etc., etc. Worth pursuing?
There is no reason to assume superintelligence can solve all our problems. The trade offs required are not going away and disagreements on those are not necessarily rational.
This is not a question that is being avoided. It is a question that has already been answered because AI already solves or helps to solve miriad problems in the real world. You could search "applications of AI" and enlighten yourself, but here are a tiny sample off the top of my head:
Fraud detection and prevention
Personalization and recommendation systems
Autonomous vehicles
Spam filters
Smart navigation systems
Inventory management
System control in industrial processes
Increasing crop yields in agriculture
All sorts of use cases in finance
Diagnosis in healthcare
Image and sound intelligent upscaling
etc etc etc
These are all real use cases of AI which are in place already.
I mean... What kind of answer are you expecting? Like, specific companies that are making money off of AI already? It's kinda early days. Describing discussions of the future as "hype" isn't really helpful either way, since that word has no meaningful definition besides "bad predictions" I guess.
If you want companies that have already chosen to apply AI to their work, consider Google, Facebook, and Tesla.
Updating the internal world model of its host to maximize its future probability of existence.
Initially, of course, that of the host, but this can change over time.
If you look at the host's modus operandi so far, you couldn't blame it.
It seems to me that you used to have to work for years to create a product and prove it’s worth in the market place and only then could you convince some wealthy person to back you to make it into a product and eventually a valuable company because the commercial viability was so glaringly obvious. Thomas Edison, Chester Carlson, etc.
But now it seems that has been turned on its head. Assemble a group of brilliant people, back them with a giant company with a highly respected CEO, and wait for success and product market fit to inevitably arrive.
I hate to sound like a downer but it just seems very odd for so many people to talk about how transformative a company is without much evidence that the products are really being used at scale. And the way opportunists like Benioff have piled on to enhance the perceived value of their own products with their own spins feels off. In fact many have openly discussed that Nvidia’s future growth relies on GPT finding product market fit at scale or they will end up with massive supply overhang which would actually cause costs of AI research to plummet, opening the door for many others to work on it. So many are already seemingly aware that commercial viability is still TBD.
I believe all this is likely a product of a period of a string of successful products starting with the iPhone, then Amazon’s transformation of shipping then AWS, etc. Whereas everyone used to be skeptical by nature, it seems now we’re in an age of extreme credulity. Everyone wants to believe in every new storyline.
And I want to be clear that I’m not suggesting that there aren’t some useful applications of GPT. But the value of products should be self evident and consumption of them should flow naturally from demand. This all just feels backwards to me. Success is assumed straight away and historically that tends to coincide with bubbles and precede a pretty nasty crash in sentiment. I’d love to be proven wrong.
I think its about efficiency. thats why you may not be seeing "new problems" being solved.
like people used to be able to travel before cars as well, cars didn't solve a new problem and the world didn't have any demand for them too. but once they became available, regardless of whether someone knows it or not, the whole world needed cars. now the car builders control the demand because they can't instantly scale up the supply
AI is similar to that but people can scale it pretty quickly to the whole world, there are a number of efficiency problems being solved in the world using AI. a couple of years ago I had to research and read atleast 1 hour worth of content to find out the information that I needed, google was good but still it required 1 hour. now its hardly a couple of minutes with chatgpt in many cases.
efficiency problems usually don't manifest in demand, people adapt to the status quo
I’d question the car analogy. For instance if tomorrow someone introduced a new method of transportation that would be quasi affordable and available to a middle class family and transport you distances in 1/100th the time required by a car there would almost certainly be instant adoption even regardless of economic conditions.
Today, though, I see companies like Tesla and OpenAI and cryptocurrencies gain gigantic enthusiastic followings without a whole lot of evidence of massive adoption and product/market fit. It’s pretty clear that underlying the enthusiasm is an economic interest in their success. That doesn’t alone negate the validity but without the adoption being self evident it’s very difficult to set aside my skepticism.
I remember walking down the magnificent mile in Chicago and going into an Apple Store for the first time and seeing and touching an early iPod nano. I knew right away that the device was an amazing leap forward. I of course didn’t predict the iPhone and Apple becoming today’s apple but I remember the “wow” factor that so many of these newer products lack. Again, I sincerely hope to be proven wrong here as I want better products!
Wait, does that mean Q* is just tree-of-thought reasoning? Like are we at the point where OpenAI is so closed that they NIH everyone else's ideas with different names?
The Q function is a fundamental part of reinforcement learning. It's a function that gives the expected reward of an agent given a specific action and state. Deep Q Learning, that is using a neural network to estimate the Q function has been around for a while now.
I remember every technology they said was going to revolutionize the world. But the only revolution I have seen is in entertainment and warfare, and I guess turn by turn navigation. Everything else is the same as it has been for over half a century. Putting lipstick on a pig doesn't change what it is.
Funny the author cherry picks the rubiks cube "breakthrough" but not AlphaGo or from GPT2 to ChatGPT. I don't think anyone could've predicted the massive jump from pre-instruct GPTs, heck even going 3.5 to 4 is a clear step difference.
If Altman was referring to Q* in his "pushing the veil of ignorance" comment then Q* is in OAI's top four breakthroughs
I think he’s using that to show that there is a precedent with this company to make misleading announcements that at first sound exciting but under the surface turn out to be overstated.
It does seem a bit odd to make a grandiose, vague announcement about a breakthrough outside of a major Jobs-esque unveiling. If you had some amazing new capability, what’s the motivation to hint at it publicly versus unveil it with a major announcement/demo? It makes me wonder if it’s an attempt to keep the public in awe with a dangling carrot. I’m also skeptical when companies’ and founders’ reputations seem to greatly outpace actual product deliveries.
AlphaGo and ChatGPT were groundbreaking and are more recent, if anything those should form a new precendent, he is deliberately ignoring them and dug out the most underwhelming one. Q* was a leak when the company was without a CEO, their DevDay was 2 week ago
That said, based on many comments here the author is a skeptic to the point that people don't take him seriously
Gary - his posts go from -“oh my god! AGI is 6 months away, Take all GPUs, stop all AI” to “AGI will take 600 years, what a bunch of horseshit, idonbelieveit”.
I wish he actually put out research saying how we solve some of the shortcomings of current techniques instead of pointing fingers.
I do agree the media loves to hype for clicks.
I loved his book “Rebooting AI”.
I think openAI is overvalued and overhyped. But such is the train of progress.
110 comments
[ 2.9 ms ] story [ 175 ms ] threadThe "AI effect"[1] describes how the field tends to get left with only the cutting-edge research.
[0]: http://ai.stanford.edu/~nilsson/OnlinePubs-Nils/PublishedPap...
[1]: https://en.wikipedia.org/wiki/AI_effect
As pointed out by techbro92 [3] the authors do not seem to be affiliated with OpenAI, but AFAIK nobody claimed that OpenAI invented Q*. It's not hard to imagine OpenAI trying it out with their Rubik-solving robot hand and then starting to think about other things it might be applied to.
[1] https://news.ycombinator.com/item?id=38395959
[2] https://arxiv.org/abs/2102.04518
[3] https://news.ycombinator.com/item?id=38398243
The meaning shifting over time, and the reason for which it shifts, is the AI effect. Heuristic-guided pathfinding is no longer what's meant by AI, IBM Deep Blue's alpha–beta search is no longer what's meant by AI, and I'd claim it's even shifting away from earlier image classifiers in favor of only generative models like GPT4.
(That said I was and I suppose am a bit of a sceptic because of that. But mainly I think because to me it's more useful to call a spade a spade and say we're doing some advanced blah stats to blah. Helps explain, onboard, and motivate the next generation (who likes stats at school? But if that's how AI works, oh suddenly it's cool). Demystify it. If you're just searching over Levenshtein distance [to a technical audience] say that, don't start waffling about 'AI'.)
Q is an evaluation of how good a state is assuming the agent acts optimally going forward. In Chess for example, sacrificing a rook to capture a queen would have a high Q value. Sacing a rook is bad, but gaining the queen later is better. Q values are supposed to see past immediate consequences and reflect the long term consequences.
How do we find a Q value though? We can look several steps ahead, or we can look to the end of the game, etc. These aren't real Q values though, because optimal actions probably weren't taken. We can use Bellman equations, which roughly update one Q value by a small amount to the maximum of the next possible states, which results in the highest Q values gradually flowing backward into states that can lead to good outcomes.
I'm trying to think how this would apply to LLMs though. Where do the basic values come from? In Chess the basic values are the number of pieces, or whether or not a side ultimately wins. What are the basic values in a LLM? What is the Q agent aiming to do?
Usually in transformer models[1], for each attention head there are 3 vectors of weights, known as q(uery), k(ey) and v(alue). I was assuming that the q in q* applied to the q vector, so q-learning is training this vector. In a transformer model you don't have an objective function for any sort of state evaluation so it can't be the Q you're thinking of.
If they've done something funky with the q vector that could indeed be a breakthrough since a lot of people feel that we are sort of running out of juice with scaling transformers as-is and need a new architecture to really have a step change in capability. That's pure speculation on my part though.
[1] Here's Vaswani et al, the paper that first set out the transformer architecture https://arxiv.org/abs/1706.03762
The industry has been focusing hard on 'System Two' approaches to augment our largely 'System One'-style models, where optimal decision policies make Q-learning a natural approach. The recent unlock here might be related to using the neural nets' general intelligence/flexibility to better broadly estimate their own states/rewards/actions (like humans). [EDIT: To be clear, the Q-learning stuff is my own speculation, whereas the 'System Two' stuff is well known.]
Serendipitously, Karpathy broadly discussed these two issues yesterday! Toward the lecture's end: https://www.youtube.com/watch?v=zjkBMFhNj_g
It is probably something else, like Decision Transformer.
Basically, instead of computing every state value like in normal Q learning. You use a Neural Network to estimate the best value.
I think Theta* is the ticket: https://en.wikipedia.org/wiki/Theta*
HN’s software doesn’t recognise an asterisk in a URL as part of the URL
The Q-value represents the expected utility or the total reward of taking a certain action in a given state, followed by following an optimal policy thereafter. So, yes, it does evaluate how "good" or beneficial a state is, but specifically in conjunction with a chosen action and under the assumption that the agent will act optimally in the future.
So it is conditioned on state and action, not just on state.
> OpenAI could in fact have a breakthrough that fundamentally changes the world
Well, it appears to me that OpenAI already has such a breakthrough- it had it roughly 4 years ago with GPT2, and it's still scaling it.
Considering that it's not yet a year since the introduction of the first ChatGPT, and given the pace at which it's evolving, I would say that the current product is already showing great promise to fundamentally change the world. I would not be surprised if just incremental changes were enough to fulfill that prediction. The impact at this point seems more limited by the ability of society to absorb and process the technology rather than intrinsic limits of the technology itself.
I wouldn't be surprised, it's Gary Marcus. He's an academic with a lot of prestige to lose if the LLM approach is actually good/useful/insightful, who's only widely publicly known now because AI has had a backlash and media needed an expert to quote for "the other side". Same as the computational linguistics researchers who always get quoted for the same reason.
In general, academics in competing fields whose funding threatens to get tanked or eclipsed by research approaches that work on principles they have fundamental academic disagreements with are going to talk negatively about the tech, no matter what it is achieving. Where I think it can be valuable to listen to them is when they're giving the technology credit - generally they'll only do that when it's something really undeniable or potentially concerning.
I wouldn't bet on loss of perceived expert on the other side status or prestige no matter how poorly his gloomy LLM forecasts fare. It hasn't happened from any of his previous consistently pessimistic predictions around deep learning since the beginning of its resurgence (which has clearly far surpassed its state in 2012, even if currently overhyped). To quote Gary Marcus quoting Gary Marcus:
> Yet deep learning may well be approaching a wall, much as I anticipated earlier, at beginning of the resurgence (Marcus, 2012)
I don't know how right or wrong he is - none of us do. That's why it's all still being debated.
The one thing I know is that we can only truly understand a topic by fully understanding arguments for and against all the claims. I also know the pro-LLM set have way more money (double-digit billions as we saw just this week) and credibility to lose over this topic than Gary Marcus does.
>but without considering the second-order effects on his credibility and funding if he actually turns out to be proven substantially wrong in the future
He has continuously made predictions about how bad deep learning is and how it would never achieve much, unlike his "much better" neurosymbolic theories that have never shown results anywhere near as significant (there might be good ideas there, doesn't mean that his continuous goalpost shifting is good science). Those predictions have continuously been proven wrong for the last decade, including a number of predictions specifically about LLMs. This is relatively normal in the academy, people have hobby horses, but he's now engaging in public discourse and advocacy about an actually contentious topic and there are different standards here, so I'm going to respond in that context. If you're not familiar with him at all, that's fine, here's some of his track record and explanations of why he's wrong:
For deep learning in general, in case you want an overview of where he's coming from (he stands by everything in this article today, don't let the age inspire sympathy, he's a deep learning skeptic to the core): https://www.newyorker.com/news/news-desk/is-deep-learning-a-...
For GPT-2 (a moderately important retrospective is in the comments): https://www.lesswrong.com/posts/ZFtesgbY9XwtqqyZ5/human-psyc...
For GPT-3: https://www.tumblr.com/nostalgebraist/628024664310136832/gar...
CTRL-F "Gary Marcus" on this page to find a bunch of sourced, specific claims from Marcus alongside evidence they are wrong: https://gwern.net/gpt-3-nonfiction
He has never stopped, and probably never will. He's the student of the original guy who first buried neural nets back in the sixties or seventies; he's royalty in symbolic AI. The only concession I've ever seen him make to the continued progress of deep learning and LLMs in spite of his continuous early proclamations of death is, bizzarely, that he is concerned about existential risk from current AI research. I don't personally see how he squares "Deep learning and LLMs are all nonsense, have hit a wall, off-ramp on the road to AGI" with "Current AI research [which is overwhelmingly deep learning, even if he doesn't say that part out loud] presents an existential risk", but he does it somehow.
Basically, no matter what new AI research or artifacts come out, if they are made with neural nets he will explain:
1. They are not that important
2. It's a dead end
3. They're harmful because people might wrongly get the impression that they work
4. They're nowhere near as good as his favoured AI architectures are, if only his favoured theories had as much funding as that broken deep learning crap
But then you've included some very unfair/invalid lines of attack. You've used quote marks when attributing arguments to him, but these are not actual quotes. You've asserted that he argues that "current AI research ... presents an existential risk" - which he recently said he specifically doesn't believe [1].
For the record, I've been aware of Gary Marcus and his positions on AI research since he was on EconTalk in 2014 [2]. I don't consider him the most compelling skeptic on contemporary AI, but I find it beneficial to hear and consider what he has to say. I'm also very familiar with ‘Mr Wern’ and his approach to discussing this and other topics. I've personally debated him on this website over a different topic, and whilst I benefited by learning new things about that topic, which was good, I was also surprised how much somebody of that persuasion would resort to emotion-charged, flawed or fallacious arguments when pressed.
And this is the main issue I'm wanting to confront: whether it's in this thread, or on LW/OB, or Sam Altman himself (in this tweet [3] that to me was a pretty alarming mask-slip for someone who has worked so hard to portray himself as a good operator over the years), it's striking how quickly the arguments turn personal and nasty rather than just pointing to simple proof that he's wrong. Which is just further evidence that discussion about this topic is so much more driven by emotion and grandiosity than sober examination of evidence.
As for Gary Marcus, sure, for someone who speaks/writes publicly a lot, you can always find specific details on which they might be wrong at a point in time. And where he's clearly factually wrong he should be corrected and he should welcome that. But I'm yet to see any slam-dunk evidence that the central position that he's continued to argue and explain for years is wrong.
Indeed, this quote from a post of his from 2018 [4] seems to be identical in substance to what Sam Altman said at Cambridge this very month [5]:
"Possibly. I do think that deep learning might play an important role in getting us to AGI, if some key things (many not yet discovered) are added in first."
[1] https://www.france24.com/en/live-news/20230604-human-extinct...
[2] https://www.econtalk.org/gary-marcus-on-the-future-of-artifi...
[3] https://x.com/sama/status/1512471289545383940
[4] https://medium.com/@GaryMarcus/in-defense-of-skepticism-abou...
[5] https://www.reddit.com/r/singularity/comments/17wknc5/altman...
Personally I have been recently have been using the ChatGPT-4 on a daily basis, and I considered myself a long time and ardent user of Google products (mainly search) for the past two decades. However it becomes increasingly frustrating for the past several years since it is getting more difficult to perform "search that matters" (going to trademark this motto). What frequently irked me the most is not when doing random or targetted search with Google, is when looking for something that you discovered previously but it's forgotten and apparently it does really matter now. It was quoted that about 10% of our time looking for items that we lost or misplaced, and the same can be said to our acquired knowledge and information. Some times we crave so much for the knowledge or the info we had but very difficult or impossible to recall. ChatGPT-4, in particular the online search with Bing feature, is extremely useful in this regard but at the same time is very limited since it is not a well supported features with many failed attempts, perhaps due to limited online data scrapping capability, and thus sub-par results. This specific feature, I call it search that matters, is like Google search with steroid and the fact that Google, with its Deepmind subsidiary, failed to utilize and monetize on this opportunity until now is just beyond me. If ChatGPT, call it ChatGPT-5 if you like, can perform this operation intuitively and seamlessly it will be a game changer but not a breakthrough. Apparently according to ChatGPT-4 you can have a game changer not a breakthrough.
The breakthrough, however, is to fundamentally improve AI or LLM itself as mentioned by Stephen Wolfram in his tutorial article on ChatGPT, not merely enhancing its existing operations [2]:
When it comes to training (AKA learning) the different “hardware” of the brain and of current computers (as well as, perhaps, some undeveloped algorithmic ideas) forces ChatGPT to use a strategy that’s probably rather different (and in some ways much less efficient) than the brain. And there’s something else as well: unlike even in typical algorithmic computation, ChatGPT doesn’t internally “have loops” or “recompute on data”. And that inevitably limits its computational capability - even with respect to current computers, but definitely with respect to the brain.
[1] Eight Things to Know about Large Language Models:
https://wp.nyu.edu/arg/eight-things-to-know-about-large-lang...
[2] What Is ChatGPT Doing and Why Does It Work:
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
Even if GPT-5 will be fatally expensive, but still show such leap as GPT-4, it will be enough to consider OpenAI most successful AI research in human history.
And for about expenses, we just need to wait some 10 years, and GPT-3 class tech will be in every smartphone, and in 30 years, GPT-4 will become also affordable for everyone, GPT-5 will be just ordinary business workhorse.
Edit: Oh I see: https://azure.microsoft.com/en-us/pricing/details/cognitive-...
Every day millions of chat sessions are recorded, and they are exceptional training examples. They would contain the kind of errors LLMs do, and the kind of demands people have, and include a human reaction to each LLM message.
The OpenAI move to create "GPTs" is showing they are actively working on improving the feedback signals by empowering the LLM with RAG, code execution and API access. In such a setup it is possible to use a model at level N to generate data at level N+1.
The keyword here is learning from feedback, which aligns with recent talk of using RL methods like AlphaZero with LLMs. A RL agent would create its own data as it goes. I think progress will be gradual, as we need to wait for the world to produce the learning feedback signal. Of course in domains where we can speed that up, AI will progress faster.
Interesting thought: by making LLMs available to the public, they are going to assist people in many ways and create effects that will percolate in the next training set: LLM inference -> text -> effects in the world -> text -> LLM training. So there is already an implicit feedback loop when we retrain the base models. GPT-5 will train on data from a world influenced and shaped with GPT-4.
https://qtransformer.github.io/
I had a good example recently and I'll quote it again:
People seriously using the "Please write an expression for me which adds 2 and 2" prompt in order to get the "2+2" expression they need – advocating that they got it with magical efficiency
Object recognition is AI. A decade ago we were piss poor at Object recog, today labeling AI is at human level. Being able to recognize your environment is the first step in navigating it effectively.
In a sense, we kind of already are already, since out internet feed is (for most apps with recommendation algorithms) just spoonfed to us, rather than the product of our own search.
To be honest, I wish the progress will stop and that AGI won't happen. That possibility scares me.
Engineering is about applying the science to real world problems.
We’re mostly in the science phase right now.
Superintelligence can solve all the things we’re struggling to solve. All human diseases, Earth being destroyed, fundamental questions about the universe, etc., etc. etc., etc. Worth pursuing?
Fraud detection and prevention
Personalization and recommendation systems
Autonomous vehicles
Spam filters
Smart navigation systems
Inventory management
System control in industrial processes
Increasing crop yields in agriculture
All sorts of use cases in finance
Diagnosis in healthcare
Image and sound intelligent upscaling
etc etc etc
These are all real use cases of AI which are in place already.
If you want companies that have already chosen to apply AI to their work, consider Google, Facebook, and Tesla.
Those are all real applications now. For example you might not think some autonomous vehicles are very good but they do exist and are in production.
It seems to me that you used to have to work for years to create a product and prove it’s worth in the market place and only then could you convince some wealthy person to back you to make it into a product and eventually a valuable company because the commercial viability was so glaringly obvious. Thomas Edison, Chester Carlson, etc.
But now it seems that has been turned on its head. Assemble a group of brilliant people, back them with a giant company with a highly respected CEO, and wait for success and product market fit to inevitably arrive.
I hate to sound like a downer but it just seems very odd for so many people to talk about how transformative a company is without much evidence that the products are really being used at scale. And the way opportunists like Benioff have piled on to enhance the perceived value of their own products with their own spins feels off. In fact many have openly discussed that Nvidia’s future growth relies on GPT finding product market fit at scale or they will end up with massive supply overhang which would actually cause costs of AI research to plummet, opening the door for many others to work on it. So many are already seemingly aware that commercial viability is still TBD.
I believe all this is likely a product of a period of a string of successful products starting with the iPhone, then Amazon’s transformation of shipping then AWS, etc. Whereas everyone used to be skeptical by nature, it seems now we’re in an age of extreme credulity. Everyone wants to believe in every new storyline.
And I want to be clear that I’m not suggesting that there aren’t some useful applications of GPT. But the value of products should be self evident and consumption of them should flow naturally from demand. This all just feels backwards to me. Success is assumed straight away and historically that tends to coincide with bubbles and precede a pretty nasty crash in sentiment. I’d love to be proven wrong.
like people used to be able to travel before cars as well, cars didn't solve a new problem and the world didn't have any demand for them too. but once they became available, regardless of whether someone knows it or not, the whole world needed cars. now the car builders control the demand because they can't instantly scale up the supply
AI is similar to that but people can scale it pretty quickly to the whole world, there are a number of efficiency problems being solved in the world using AI. a couple of years ago I had to research and read atleast 1 hour worth of content to find out the information that I needed, google was good but still it required 1 hour. now its hardly a couple of minutes with chatgpt in many cases.
efficiency problems usually don't manifest in demand, people adapt to the status quo
Today, though, I see companies like Tesla and OpenAI and cryptocurrencies gain gigantic enthusiastic followings without a whole lot of evidence of massive adoption and product/market fit. It’s pretty clear that underlying the enthusiasm is an economic interest in their success. That doesn’t alone negate the validity but without the adoption being self evident it’s very difficult to set aside my skepticism.
I remember walking down the magnificent mile in Chicago and going into an Apple Store for the first time and seeing and touching an early iPod nano. I knew right away that the device was an amazing leap forward. I of course didn’t predict the iPhone and Apple becoming today’s apple but I remember the “wow” factor that so many of these newer products lack. Again, I sincerely hope to be proven wrong here as I want better products!
If Altman was referring to Q* in his "pushing the veil of ignorance" comment then Q* is in OAI's top four breakthroughs
It does seem a bit odd to make a grandiose, vague announcement about a breakthrough outside of a major Jobs-esque unveiling. If you had some amazing new capability, what’s the motivation to hint at it publicly versus unveil it with a major announcement/demo? It makes me wonder if it’s an attempt to keep the public in awe with a dangling carrot. I’m also skeptical when companies’ and founders’ reputations seem to greatly outpace actual product deliveries.
That said, based on many comments here the author is a skeptic to the point that people don't take him seriously
I wish he actually put out research saying how we solve some of the shortcomings of current techniques instead of pointing fingers.
I do agree the media loves to hype for clicks.
I loved his book “Rebooting AI”.
I think openAI is overvalued and overhyped. But such is the train of progress.