>Encoded in the large, highly evolved sensory and motor portions of the human brain is a billion years of experience about the nature of the world and how to survive in it. The deliberate process we call reasoning is, I believe, the thinnest veneer of human thought, effective only because it is supported by this much older and much more powerful, though usually unconscious, sensorimotor knowledge. We are all prodigious olympians in perceptual and motor areas, so good that we make the difficult look easy. Abstract thought, though, is a new trick, perhaps less than 100 thousand years old. We have not yet mastered it. It is not all that intrinsically difficult; it just seems so when we do it.
They should probably optimize for minimal friction force. If you're pulling the handle not entirely in the direction of movement, you're not doing it right.
This came out a while ago, but I can see why it's been reposted. It's another approach to searching the solution space with rewards and transformer language models. Like XoT and others as I pointed out in my last comment: https://news.ycombinator.com/item?id=38471388
It's Noam Brown's (former Deep Mind) "I'm joining OpenAI" thread from Back in June, and he talks about how he wants to bring some of the work from AlphaGoZero etc into OpenAI's AGI architecture.
It looks like this Q-transformer work shares a lot of the same lineage of what Q* is supposed to be. This is Google's attempt at mushing Q-learning into transformer land, which is probably what Q* is as well. It's not the same thing, for sure, but they are at least sibling bodies of work.
Apparently this guy like a bunch of others like https://github.com/ggerganov/ggml are implementing transformers from papers for people that want them. Pretty cool.
I’m glad to see people in the open source community taking the initiative to implement these tools for the general public. It’s one thing to understand the high-level concepts in a paper, but another entirely to implement them efficiently. Some people just really have a knack for reading academic papers and implementing those ideas in software.
The additional link is indeed very useful, however, with one exception, it is limited to inference, ie code for evaluating a model using somebody else’s trained model weights. The original link has code for training (which can also be used for inference, of course.) Inference is super important, of course, so it makes a lot of sense to put effort to make it work well on all kinds of hardware; the papers typically focus on training and only then evaluating/validating a new model.
It's clear we're searching for the god algorithm of AI, just like physicists are searching for theory of everything. Are transformers the answer though?
It seems more likely that there will be multiple avenues to AGI, all with their strengths and weaknesses. But perhaps the "God AI" will be a multifaceted model composed of many different models acting in unison.
On the other hand, most of the original gods were parts of polytheistic pantheons. Maybe a bunch of models that represent identities and biases could be more useful, they could argue amongst themselves, presenting a more full point of view, users could become familiar with the particular perspectives.
I could see something like the "modularity of mind" model of human consciousness, where multiple approaches are working on "subconscious" solutions to a given problem in parallel, with a top layer deciding which is appropriate at the moment.
My human brain doesn't use the same algorithm for learning to play a song on a piano as learning to play a new board game. I'm not an AI person, but it seems reasonable to imagine we'd have different "modules" to apply as needed.
AlphaGo probably sucks at conversation. ChatGPT can't play Go. The part of my brain writing this couldn't throw a baseball. The physics engine that lets me throw a baseball couldn't write this. Is there a reason we'd want or need one specific AI approach to be universally applicable?
We're searching for an efficient algorithm that leads to AGI. Given sufficient time and compute, I'm sure that we could get there with existing stuff, by accident, and we wouldn't realize it before moving on to the next thing... and there'd be a poor orphan AGI, lost in a Git repo, waiting for runtime.
"Given sufficient time and compute" covers up a lot, though. The ultimate God AGI that would be created through that sort of process would take the form of a large room filled with a whole lot of monkeys and typewriters.
As it stands, each token processed by transformer requires a constant amount of computation and energy. For an AGI system, this would imply the ability to solve problems of any complexity with a fixed amount of energy. But if this were true, it would essentially mean that P equals NP, a major theoretical breakthrough in computational complexity theory. IMO we are still missing something.
Algorithm is too simple, but yes - what does the system look like
I think the transformers architecture, or something very similar with eventually-on-policy time series forecasting in a markov decision process, is the right answer actually and was what I have been trying to make progress on for a long time[1].
After this past few years of progress on multivariate stream forecasting I really think it’s going to be different implementations of the same basic streaming data forecasting architecture
Theory of AI is just going to be some weird network with a shit-ton of compute power, where the latter is more important to the outcome than the former.
There's this nice new paper ("Simplifying Transformer Blocks" https://arxiv.org/abs/2311.01906) about getting rid of all the unnecessary parts of a transformer. On their dataset, they manage to get rid of the layernorm, the skip connection, the value matrix, and the projection matrix.
No, it’s a much simpler architecture. Anything that evolved can’t require a really complicated niche structure.
The difficulty is the scale. Every synapse of a neuron is effectively a neuron itself, and every synapse acts on the synapses around it. So before you’ve even got to the neuron as a whole you’ve already got the equivalent of thousands of neurons and logic gates. Then the final result gets passed on to thousands more neurons.
I don’t know how you would recreate such complexity in programming. It’s not just the scale, it’s the flexibility of the structure.
Could one of the roomba companies now make those things recognize the objects and avoid them, instead of dead reckoning or RF based obstacle avoidance?
Just to be clear the "scalable" part here is assuming the dataset of human demonstrations is available it can in a closed loop learn better than before?
In other words, am I right in understanding that the mapping from Abstract actions to concrete steps in the State space is provided at a low "enough" cost?
Like in Workflow Guided explorations like: https://arxiv.org/pdf/1802.08802.pdf, for example, if you said "Forward the email" the Bot needs to sample from a set of actions which in this case would be DOM elements and the JS API to call to click the right button to forward the email. This would be a harder/interesting to test out. Can it learn to forward an email on firefox desktop browser and then do it on an iphone?
It feels like opening a drawer once learn't, the bot has a (world?) model of what all drawers might look like, so it can open different drawers in any world. But this might not generalize to web interfaces and more specifically, how to do those actions on those interfaces?
Not to take away from what this paper's scope and achievements.
Looking at the figures and videos it seems... worse? Sort of surprised they didn't compare it but I guess they're trying to limit the discussion to purely reinforcement learning methods.
EDIT: Ah, I see this is an older result and was published concurrently to the diffusion policy paper, so it's likely the authors didn't know about it in time to add the extra comparisons.
To save you guys some time, this is a 2 month old paper and not made by OpenAI. It could be related to Q* (I don't know enough to determine that) but it's not OpenAI's Q*.
To make this strand of discussion slightly more interesting, I hereby present you a page [1] with the history of Q-tips. Apparently, the "Q" stands for Quality :)
73 comments
[ 2.5 ms ] story [ 140 ms ] threadhttps://www.youtube.com/watch?v=UAG_FBZJVJ8
>Encoded in the large, highly evolved sensory and motor portions of the human brain is a billion years of experience about the nature of the world and how to survive in it. The deliberate process we call reasoning is, I believe, the thinnest veneer of human thought, effective only because it is supported by this much older and much more powerful, though usually unconscious, sensorimotor knowledge. We are all prodigious olympians in perceptual and motor areas, so good that we make the difficult look easy. Abstract thought, though, is a new trick, perhaps less than 100 thousand years old. We have not yet mastered it. It is not all that intrinsically difficult; it just seems so when we do it.
Still working on closing the drawers afterwards, though...
[0] https://www.theverge.com/2023/11/29/23982046/sam-altman-inte...
https://aws.amazon.com/de/q/
It's Noam Brown's (former Deep Mind) "I'm joining OpenAI" thread from Back in June, and he talks about how he wants to bring some of the work from AlphaGoZero etc into OpenAI's AGI architecture.
It looks like this Q-transformer work shares a lot of the same lineage of what Q* is supposed to be. This is Google's attempt at mushing Q-learning into transformer land, which is probably what Q* is as well. It's not the same thing, for sure, but they are at least sibling bodies of work.
What's the letter?
What's the letter?
The letter of the day is...
Q!
[1]: https://github.com/lucidrains/q-transformer
[0] https://en.wikipedia.org/wiki/Themes_in_A_Song_of_Ice_and_Fi...
My human brain doesn't use the same algorithm for learning to play a song on a piano as learning to play a new board game. I'm not an AI person, but it seems reasonable to imagine we'd have different "modules" to apply as needed.
AlphaGo probably sucks at conversation. ChatGPT can't play Go. The part of my brain writing this couldn't throw a baseball. The physics engine that lets me throw a baseball couldn't write this. Is there a reason we'd want or need one specific AI approach to be universally applicable?
I think the transformers architecture, or something very similar with eventually-on-policy time series forecasting in a markov decision process, is the right answer actually and was what I have been trying to make progress on for a long time[1].
[1]https://kemendo.com/research/streaminference.html
Theory of AI is just going to be some weird network with a shit-ton of compute power, where the latter is more important to the outcome than the former.
The difficulty is the scale. Every synapse of a neuron is effectively a neuron itself, and every synapse acts on the synapses around it. So before you’ve even got to the neuron as a whole you’ve already got the equivalent of thousands of neurons and logic gates. Then the final result gets passed on to thousands more neurons.
I don’t know how you would recreate such complexity in programming. It’s not just the scale, it’s the flexibility of the structure.
It feels like opening a drawer once learn't, the bot has a (world?) model of what all drawers might look like, so it can open different drawers in any world. But this might not generalize to web interfaces and more specifically, how to do those actions on those interfaces?
Not to take away from what this paper's scope and achievements.
https://diffusion-policy.cs.columbia.edu/
Looking at the figures and videos it seems... worse? Sort of surprised they didn't compare it but I guess they're trying to limit the discussion to purely reinforcement learning methods.
EDIT: Ah, I see this is an older result and was published concurrently to the diffusion policy paper, so it's likely the authors didn't know about it in time to add the extra comparisons.
[1] https://www.qtips.com/about/
First it was the LK-99 hype, now it is Q. Chasing hype like flies to a light bulb.