They did, in 1984 and 1991. It's now been decades. It's even almost two decades since people felt it was now ok to call a robotics company Cyberdyne[0].
Decades have passed, and with pretty much no actual surprises on the way, we're steadily marching towards creating robots with the power to destroy humanity. We haven't made much progress on time travel though, so whatever happens, we probably won't have a chance at a do-over.
Robotics is such a lovely world. For the hobbyists, I'm really curious about how to get the home-brew robotics training lab working at home? The last time I reviewd this, you'd spend thousands of dollars just to get a reasonable robotic arm. Right now, if work is relegated to the rich research departments of mega corporations, this certainly doesn't seem more interesting than a corporate press release..
Depends on what part of robotics you want to explore. You can get a quadruped with an arm for around 400-1000$ if you don't mind it being small, and with a limited set of sensors.
It becomes expensive when you want an robot arm with high dexterity and torque-sensors, which is often a requirement for some tasks, but position-based control can be done on cheaper models.
With emphasis on "small" I got one called XGO mini 2 (and 1) that I sponsored on Kickstarter, which had it at half the price (400$ ish) but it's around $800 here[1].
It runs on Raspberry Pi which is a big +, as version 1 was running on custom hardware. People had to plug a Jetson Nano or other device to run their code (such as this [2]). It comes with their own App, connected via bluetooth, for simple commands too.
I (and most kickstart backers) funded/bought it for either hobby or research purpose (some backers were from labs) so if you're looking for *practical* application this one might be a bit too weak.
But version 1&2 did come with a processing power that could run YOLO detection so there's enough power to test things out and learn.
I thought by now there'd be kits like plastic airplane models. Injection-mold a bunch of parts, connected by sprues, and throw in the odd bits of (metal) hardware where they are critical to the whole.
Someone else mentioned sims - Mujoco is pretty common and you'll want to learn ROS anyway. Also robotics doesn't mean just arms, you can play with navigation and perception algorithms with a single camera.
The Portal-themed animation really sells it as a step in the right direction. ;)
Seriously though, I’m excited to see where this goes. AI is progressing so fast now that newcomers loudly proclaim “AI is dead” and “going nowhere” when they haven’t seen SotA beaten for a whole month, because they don’t remember the times when a small improvement in a decade was big news.
You know where it's going dude...to the military and law enforcement. It's completely naive to think otherwise. Advanced robotics might be used to unpack your dishwasher, or save kittens stuck in trees, but we know where the $$$ lies...
Depends on who sparks the hard takeoff. There's a whole bunch of interesting times coming up and I don't think our human power structures are going to have any control over it. The next five years are going to be less about status quo and more about... good parenting?
This is a step towards what I think AGI would be: a model trained on visual and language data from the Internet used as a prior for an action model (finetuned perhaps with reinforcement learning) that would be able to learn how to use the frozen prior to make useful actions; I would separate the prior from the action model so there is no possibility of catastrophic forgetting (this is dealt by co-fine-tuning the model in the RT-2 paper), and because a more advanced robot would need to control more actions quickly, so it would be expensive to run the entire prior just to do basic movements in real time.
Edit: also, the model should be smart about how to use its "context window", when the robot is taught how to do a task and it needs to do that task, it must retain the knowledge.
That seems like a real step in the direction of AGI but not anywhere near the full solution. The current context windows are far too small to replicate human intelligence. I can’t quite put my finger on it but it feels like we are missing a sort of bridge between learning done in-context and offline training. A true AGI would be able to learn in-context and then quickly apply those lessons to the base model. If in context learning is analogous to a person’s short term memory, we need a mechanism to move short term memory to long term memory.
In the near term I expect we will see much more general robotics that know how to do lots of tasks and can follow basic instructions, but lack the ability to develop complex new skills over time. Robots doing dishes and laundry will soon be feasible, just don’t expect unbounded self improvement.
I agree that it would be much better if a model could move the knowledge to the weights, but I don't think it's necessary. Models like RNN have practically infinite lossy "context window", and I think it would be much more reasonable to find an RNN architecture that would scale like a transformer than what you propose, while still achieving our goal. An alternative analogy you can think of is that the frozen weights in the model are our DNA, while the state or "context window" of the model is our state.
If it was tethered to the floor, I would trust a good LLM in 2023 with folding laundry, but very little else. No dishes. No letting kids or pets bear it. No plumbing. I wouldn't even let it wander around cleaning stuff, because I'd expect it to knock over things.
I personally am less concerned about knocking things over, which is a thing we already trust vacuum robots with; rather I'd be much more concerned about the robots accidentally (or intentionally?) "folding" my kids or pets.
Why does Google actually build stuff like this? What is their actual end game and how does it relate to what they actually do as a business?
They're an advertising company with a mission to "organize the worlds information", pivoting to robots? Do they need robots to organize the worlds information?
No idea how big of a breakthrough this is, but it is absolutely undeniable that Googles PR team is going wild on anything AI since OpenAI provided them with their first real existential crisis / code red.
> Do they need robots to organize the worlds information?
Well, my bookshelves are a mess and I'd be ok with getting a robot to help organize my information, if I had some assurances that it won't kill me, or at the very least a "Don't be evil" clause.
google runs on ad money but they're also in the business of acquiring elite ML talent and publishing sota research is one of the main drivers in accomplishing that.
Google, unfortunately, does not "get" robotics. They are great at big ML training infra, but they need better robotics fundamentals that involve the real world.
This seem like an cool upgrade from RT1, judging from the result. It seems to now also output the delta of the end-effector pose, which was previously handled by a different motion planner.
It does seem like this work (and a lot of robot learning works) are still stuck on position/velocity control and not impedance control. Which is essentially output where to go, either closed-loop with a controller or open-loop with a motion planner. This seem to dramatically lower the data requirement but it feel like a fundamental limit to what task we can accomplish.
The reason robot manipulation is hard is because we need to take in to the account not just what's happening in the world but also how our interaction alters it and how we need to react to that.
Robot Learning right now is either Reinforcement Learning or Imitation Learning (often latter) and I'm not sure how one would collect data that capture this.
Edit: I'm surprised Google still do robotics work, I thought Everyday Robotics was shut down [1]
Seems to me that the way to architect this is to have multiple quasi independent embedded controllers at different levels. For example, you might have 3 independent finger controllers, managed by a hand controller, and so on, on up to the highest level LLM that drives everything. So you have an LLM that just says, pick up the green can, issues whatever structured data needs to go to the arm controller and on down the line, going to down to more real-time and less high level processing as you go.
As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when I want it to happen. I don't think you need to design a robotic AI that understands how every part of it's mechanics work. The fingers don't need to know how the feet work, for example. There should be semi-autonomous "intelligence" embedded throughout the system, with only necessary feedback being fed back up.
I agree. We need feedback loops on each joint that keep doing what they are doing until a higher feedback loop decides to change things.
For example, as I enter this text, most of my fingers are holding the phone and do not need to be told to keep holding it.
The low level controllers should also be able to respond to simple things.
It's like when I get hurt, my hand yanks back before my brain realizes what has happened. Or when I almost trip and my legs correct before I realized it happened.
Low level tasks should be at a low level and not bother the higher professors.
LLMs have terrible spatial awareness. This probably comes from ONLY being trained on text.
I wonder if it would help a LLM running a robot to have a separate controller calculating it's position and what is around it, and feed that into the LLM constantly.
It would be like when a video game has a radar display or map to let you know where you are in relation to other things.
I actually tried that a while back, giving 3.5-turbo a multishot prompt that consisted of distance readings for ahead, left, right and back in an array, as extracted from lidar data, then giving it movement instructions. It performed rather terribly.
You've very much correct that their spatial awareness is terrible. Something as simple as drive forward, then back, turn left, etc. works just fine and they can generally translate it to a specified message format reasonably reliably, but give them something more complex to execute, like drive a robot in a square pattern (an example answer would be go forward, turn right, go forward, turn right, etc.) they start to generate nonsense.
I also tested it with the 30B WizardLM at the time which performed almost as well in terms of message format but had even worse awareness.
Part of the problem is that the training data contains next to no examples that would teach it how 3D space works. I considered making a dataset of driving a robot around with human movement commands and then logging the aggregated sensor data and commands for fine tuning so the prompt format would be pre-learned, but I'm not entirely sure how much it would help.
Hi @moffkalast, I am interested in doing something similar (building a larger dataset of robot movements) and fine-tuning an LLM with it. Would love to have a quick call and chat further. You can reach me at "hi [at] adeelzaman [dot] me".
LLMs have a firm grip on common sense. It's because it allows them to deal with the utterly unexpected they are deemed useful in robotics. Not to perform delicate movements, but stop doing so when police enters the room.
That is indeed what a lot of Machine Learning turned Robotics researchers/enthusiasts are banking on. A counter argument to that is what Dhruv Batra responded to the question "lol why not use LLM for everything" [1].
>As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when I want it to happen. I don't think you need to design a robotic AI that understands how every part of it's mechanics work.
This is true for us ofc, and is encompassed in what is called the "Moravec's paradox" [2]. You don't understand it because it's unconscious process, and It's harder to reverse-engineer an unconscious process (motor movement) than conscious ones (calculating math, playing game, writing text, reading).
But the thing is that in the real world we do need to take in to account everything, including noise and time-delay. Evolution gave rise to complex language in the last 100k years compared to millions of year for motor movement. I do agree that there must be some "hierarchical" structure for complex motion, but we currently don't know where and how that hierarchy is.
Boston dynamics uses Model Predictive Control for complex movement, which means that at least some model of the world is required, for dexterous motion. Now if that model is part of LLM or not is a hard guess.
But if we don't know this it's hard to say what kind of data we need to collect to train a LLM-model applied to embodiment (robotics).
Researchers in the past have already made the mistaken assumption of "oh cognition and language is the hard part of intelligence. Perception and motion is easy" [3] and then their work amounted to nothing because turns out the latter was way, wayyyyyy harder.
There's an implicit bias in us that think Language, puzzles and logic are harder [2] and therefore models that accomplish this can just be rammed in to the "easier" issues.
Edit: I too would like "LLM models will solve these" attitude because otherwise the research I'm doing right now is a dead-end, but the more I try (with my limited compute) the less I'm sure
>It's harder to reverse-engineer an unconscious process
Aside from some basic life support systems, don't almost all movements start with conscious effort? Whether you are deliberate about the exercise or not, you practice and practice until you develop 'muscle memory' where it becomes unconscious: walking, dribbling a basketball, holding a G chord on a guitar, etc.
Maybe the octopus is a more tractable model for robotic control. I understand that octopus neurons are not as concentrated in a central brain, but spread throughout its limbs that are autonomous compared to humans.
For example, when you reach out to pick up a green can, your brain makes the decision to do the task but it's your spinal cord and peripheral nerves that carry out the detailed work – orienting the hand, managing grasp strength, controlling the arm movements etc. This process is mostly unconscious – you don't need to actively think about how to tense each muscle in the same way that an embedded controller wouldn't need to understanding the working of the entire robotic system to carry out its specific task.
Much like the model suggested, the human body communicates feedback across layers — this process is crucial to maintaining balance, coordination and effectively reacting to the environment. For instance, if your fingers touch a hot stove, the sensory receptors in your skin will immediately send a signal to your spinal cord and a reflex action will make you pull your hand back even before you consciously perceive that the stove is hot.
They collect data for RL/IL in simulation which _can_ generalize to the real world. Also, being Google, they have the resources to collect data by brute force i.e. scientists manually collecting that data. The paper says - one main source of the data is internet scale vision/llm data. The second source is 6k trial runs.
A principal idea behind this work is that you can collect data in one domain to avoid collecting data in another. Training an LLM can train the 'reasoning' portion of the robot so that it can perform real-world skills with less training.
They use the RT1 data which is (if I remember correctly) 11 month of data using like 6 research engineers sitting everyday, remote controlling the robot. They might even use ROSIE data which is data-augmentation using Stable-diffusion inpainting.
And for that they can do mobile manipulation, which have been possible for a long while (as long as we know ground-truth of object location, and have a some-what consistent map. Although the latter is solved with SLAM). it is way, way more versatile (if their claims are correct) so this is cool result.
as for sim2real, that's a whole different set of issue.... Simulator have come a long way, but from last time I talked with team-lead of Mujoco sim, it seems like completely realistic ones (at a reasonable compute time) is still far away. Deform-able objects are hard to simulate
My point is just that the "reasoning" you mention have a limit;[1] there's no amount of reasoning with language that can zero-shot a robot to do gymnastics or ride a bide. I'd be more than happy to be proven wrong on this point though, then my line of research is not dead-end =)
Sure, I wouldn't say reasoning can guide a robot to do gymnastics zero-shot either. But it is a substantiated approach to generalizing existing capabilities - just like in humans. You or a robot can't reason your way to bike-riding but if you know how to bake cookies or brush your teeth, you can perform much better zero-shot on related tasks like baking a chicken or brushing a mirror.
There under "demo" are a few videos at 2x and 4x speed. It's slow. None of the videos include audio of the verbal commands or the latency between commands and action.
I'm actually pretty bullish on humanoid robots like the Tesla bot - combination of LLMs, cheap batteries/motors/controllers from cars and vision research should be able to come together in useful and cheap ways in a few years, say 2030.
$35K for a robot that can putter around the house doing basic stuff is just not that high of a bar. With a 10 year life span, that's $3.5k/year, or $10/day. Doing 1 hour of useful minimum wage work around the house is just not that high of a bar - doing laundry, cleaning, tidying up, weeding, wiping surfaces down, taking out the garbage etc. If it can do some combination of those, it would make sense for basically every household. And it doesn't need to be able to do the crazy parkour of Boston Dynamics to achieve this. Our world is generally designed to be operable by all sorts of people - disabled, old etc. Crazy athleticism isn't required to do useful work.
Agreed on the big value of robotics in the near term, but Tesla won't be the company to do it. Tesla will make an "affordable" humanoid but will struggle to get useful and robust autonomy out the door for the entirety of its offering.
They thing to keep in mind about the "Boston Dynamics approach" is that in order to solve the hardest real world problems, you need to do one of two things:
1) Overshoot the capabilities of the system so that it is robust when actually deployed subject to the uncertainty of the real world (e.g. motivating athletic intelligence).
2.) Grossly constrain your environment so that uncertainty is not a factor. This is (and has been) happening in warehouses and factories for decades.
There's a cool part where they ask the robot to pick up a lion from a group of toy figures it hasn't seen before. After it does it correctly the NYT reporter asks it to pick up the extinct animal and the robot picks up the dinosaur toy.
It would be interesting to see how it solve AI Planning Problems like BlocksWorld[1]. I've read that in the past with these things[2][3] when multiple goals needed to be met at once and there was interaction between them, it just falls over itself. Being able to generate coherent plans and execute them, as I understand, an important aspect of generating Action Sequences given a State and thus for planning in robotics. How are these overcome in RT2?
P.S.: I also was told that the key here is that in automated planning you can't have a human in the loop doing the actual learning. If you are going to prompt engineer or get a human in the loop to a degree that you effectively fool yourself that the robot is solving a problem then its not planning.
Related work is the planning paper by Valmeekan et al [1]. The gist is that LLMs are incapable of planning, which is due to their autoregressive nature. METAs Head of AI Yann Lecun also talks about this topic in a talk [2]. As RT2 is based on a similar architecture, I think the results will be similar.
I never see mention of another type of integration that seems to me to be necessary for the improvement of AI and for any attempt to establish general artificial intelligence: integration with the senses. Chains of thought are born in the senses and are constantly updated in the background with information from them.
To fully match the human model, AGI based on neural networks will need to dynamically receive information from sensors of all kinds [to be able to sample at least five major categories of physical stimuli, like us]. If we want to reach the level of the neural networks that we carry in our heads, we have to learn more about the role of other brain structures, such as glial cells, which are known to influence the activation and moderation of synapses in the human brain [which correspond to the 'weights' in artificial neural networks], and then apply them to our technological endeavors.
Without it two things will happen
1. We will never have strong AI
2. Strong AI that works virtually only, aware of the world and material reality, but denied access to them by its creators.
Imagine that it is possible to take a snapshot of a human mental state [an analogue of .h5 weights] and then run it on hardware as we do with our neural networks. It would certainly be a nightmare for a mind endowed with "qualia" to find itself enclosed in a metal box [brain in a vat] with no access to the senses. There are clinical cases analogous to this condition, for example in states where the patient is aware of everything around him but is unable to respond.
These issues highlight the ever-present ethical boundaries of IA research. Without integration with real-time sensors AI will always be incomplete and/or inhumane.
An immediately useful application of this is in roombas that can not just clean the floor but effectively avoid/move obstacles. All the vaccums i have gotten generally get stuck or suck in paper/usb cables and get stuck in corners. There are two planning aspects here
1. Choosing a workflow(i.e. a series of general steps that can achieve a goal)
2. Generating low level policy actions given a state(i.e. sensor data).
Like lets say the Workflow is to clean the room. Now there is a big chair in the way and a USB wire there too, the robot could just decide to move around the chair but it could pick up the wire and place it on the table and clean the area. Generating policy actions this way seems to be very much in line with simple VQA based things proposed in the paper, or so it would seem...
I don't think its even necessary to pick up the wire. The newest vacuums use cameras to try to avoid common obstacles, but the image classification is so rudimentary they get stuck anyway. Having the common sense to stay away from the end of the cable is all the robot needs to not get stuck.
74 comments
[ 2.0 ms ] story [ 140 ms ] threadEdit. So many downvotes -- here have a /s (sarcasm)
Decades have passed, and with pretty much no actual surprises on the way, we're steadily marching towards creating robots with the power to destroy humanity. We haven't made much progress on time travel though, so whatever happens, we probably won't have a chance at a do-over.
[0] https://en.wikipedia.org/wiki/Cyberdyne_Inc.
Since it didn't happen, maybe a Terminator Genisys type reset of the timeline has occurred. So we have a few more decades buffer again, to maybe 2050?
EDIT: And here's the paper pdf - https://robotics-transformer2.github.io/assets/rt2.pdf
(preferably a physics simulator library, not an entire framework)
idk there is plenty of universities that have robotics labs.. Probably the best value is to apply for masters etc if you are interested in that..
It becomes expensive when you want an robot arm with high dexterity and torque-sensors, which is often a requirement for some tasks, but position-based control can be done on cheaper models.
Quadruped with an arm for <$1000 would be a steal. Got any recommendations?
It runs on Raspberry Pi which is a big +, as version 1 was running on custom hardware. People had to plug a Jetson Nano or other device to run their code (such as this [2]). It comes with their own App, connected via bluetooth, for simple commands too.
I (and most kickstart backers) funded/bought it for either hobby or research purpose (some backers were from labs) so if you're looking for *practical* application this one might be a bit too weak.
But version 1&2 did come with a processing power that could run YOLO detection so there's enough power to test things out and learn.
[1] https://www.robotshop.com/products/xgo-mini-2-quadruped-robo... [2] https://www.youtube.com/watch?v=xqfwCwu4-Rc
Does it has servos feedback too ?
Good off-the-shelf arms are low thousands last time I looked. You can DIY for $1-2k https://www.anninrobotics.com/robot-kits
https://www.youtube.com/watch?v=6zPvT0ig1VM
Seriously though, I’m excited to see where this goes. AI is progressing so fast now that newcomers loudly proclaim “AI is dead” and “going nowhere” when they haven’t seen SotA beaten for a whole month, because they don’t remember the times when a small improvement in a decade was big news.
Edit: also, the model should be smart about how to use its "context window", when the robot is taught how to do a task and it needs to do that task, it must retain the knowledge.
In the near term I expect we will see much more general robotics that know how to do lots of tasks and can follow basic instructions, but lack the ability to develop complex new skills over time. Robots doing dishes and laundry will soon be feasible, just don’t expect unbounded self improvement.
I would then have one cat and one messed up LLM robot.
Progress!
They're an advertising company with a mission to "organize the worlds information", pivoting to robots? Do they need robots to organize the worlds information?
No idea how big of a breakthrough this is, but it is absolutely undeniable that Googles PR team is going wild on anything AI since OpenAI provided them with their first real existential crisis / code red.
Well, my bookshelves are a mess and I'd be ok with getting a robot to help organize my information, if I had some assurances that it won't kill me, or at the very least a "Don't be evil" clause.
I'm so sick of Google I just paid for Kagi.
It does seem like this work (and a lot of robot learning works) are still stuck on position/velocity control and not impedance control. Which is essentially output where to go, either closed-loop with a controller or open-loop with a motion planner. This seem to dramatically lower the data requirement but it feel like a fundamental limit to what task we can accomplish.
The reason robot manipulation is hard is because we need to take in to the account not just what's happening in the world but also how our interaction alters it and how we need to react to that.
Robot Learning right now is either Reinforcement Learning or Imitation Learning (often latter) and I'm not sure how one would collect data that capture this.
Edit: I'm surprised Google still do robotics work, I thought Everyday Robotics was shut down [1]
[1] https://www.therobotreport.com/alphabet-closes-everyday-robo...
As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when I want it to happen. I don't think you need to design a robotic AI that understands how every part of it's mechanics work. The fingers don't need to know how the feet work, for example. There should be semi-autonomous "intelligence" embedded throughout the system, with only necessary feedback being fed back up.
For example, as I enter this text, most of my fingers are holding the phone and do not need to be told to keep holding it.
The low level controllers should also be able to respond to simple things.
It's like when I get hurt, my hand yanks back before my brain realizes what has happened. Or when I almost trip and my legs correct before I realized it happened.
Low level tasks should be at a low level and not bother the higher professors.
I wonder if it would help a LLM running a robot to have a separate controller calculating it's position and what is around it, and feed that into the LLM constantly.
It would be like when a video game has a radar display or map to let you know where you are in relation to other things.
You've very much correct that their spatial awareness is terrible. Something as simple as drive forward, then back, turn left, etc. works just fine and they can generally translate it to a specified message format reasonably reliably, but give them something more complex to execute, like drive a robot in a square pattern (an example answer would be go forward, turn right, go forward, turn right, etc.) they start to generate nonsense.
I also tested it with the 30B WizardLM at the time which performed almost as well in terms of message format but had even worse awareness.
Part of the problem is that the training data contains next to no examples that would teach it how 3D space works. I considered making a dataset of driving a robot around with human movement commands and then logging the aggregated sensor data and commands for fine tuning so the prompt format would be pre-learned, but I'm not entirely sure how much it would help.
>As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when I want it to happen. I don't think you need to design a robotic AI that understands how every part of it's mechanics work.
This is true for us ofc, and is encompassed in what is called the "Moravec's paradox" [2]. You don't understand it because it's unconscious process, and It's harder to reverse-engineer an unconscious process (motor movement) than conscious ones (calculating math, playing game, writing text, reading).
But the thing is that in the real world we do need to take in to account everything, including noise and time-delay. Evolution gave rise to complex language in the last 100k years compared to millions of year for motor movement. I do agree that there must be some "hierarchical" structure for complex motion, but we currently don't know where and how that hierarchy is. Boston dynamics uses Model Predictive Control for complex movement, which means that at least some model of the world is required, for dexterous motion. Now if that model is part of LLM or not is a hard guess.
But if we don't know this it's hard to say what kind of data we need to collect to train a LLM-model applied to embodiment (robotics).
Researchers in the past have already made the mistaken assumption of "oh cognition and language is the hard part of intelligence. Perception and motion is easy" [3] and then their work amounted to nothing because turns out the latter was way, wayyyyyy harder.
There's an implicit bias in us that think Language, puzzles and logic are harder [2] and therefore models that accomplish this can just be rammed in to the "easier" issues.
Edit: I too would like "LLM models will solve these" attitude because otherwise the research I'm doing right now is a dead-end, but the more I try (with my limited compute) the less I'm sure
[1] https://imgur.com/eWsH5ui originally https://twitter.com/DhruvBatraDB/status/1641871357020614656
[2] https://en.wikipedia.org/wiki/Moravec%27s_paradox
[3] https://youtu.be/x10964w00zk?list=PLSQhB89mdG7PsZsDz2_5hZL8C...
Aside from some basic life support systems, don't almost all movements start with conscious effort? Whether you are deliberate about the exercise or not, you practice and practice until you develop 'muscle memory' where it becomes unconscious: walking, dribbling a basketball, holding a G chord on a guitar, etc.
For example, when you reach out to pick up a green can, your brain makes the decision to do the task but it's your spinal cord and peripheral nerves that carry out the detailed work – orienting the hand, managing grasp strength, controlling the arm movements etc. This process is mostly unconscious – you don't need to actively think about how to tense each muscle in the same way that an embedded controller wouldn't need to understanding the working of the entire robotic system to carry out its specific task.
Much like the model suggested, the human body communicates feedback across layers — this process is crucial to maintaining balance, coordination and effectively reacting to the environment. For instance, if your fingers touch a hot stove, the sensory receptors in your skin will immediately send a signal to your spinal cord and a reflex action will make you pull your hand back even before you consciously perceive that the stove is hot.
A principal idea behind this work is that you can collect data in one domain to avoid collecting data in another. Training an LLM can train the 'reasoning' portion of the robot so that it can perform real-world skills with less training.
And for that they can do mobile manipulation, which have been possible for a long while (as long as we know ground-truth of object location, and have a some-what consistent map. Although the latter is solved with SLAM). it is way, way more versatile (if their claims are correct) so this is cool result.
as for sim2real, that's a whole different set of issue.... Simulator have come a long way, but from last time I talked with team-lead of Mujoco sim, it seems like completely realistic ones (at a reasonable compute time) is still far away. Deform-able objects are hard to simulate
My point is just that the "reasoning" you mention have a limit;[1] there's no amount of reasoning with language that can zero-shot a robot to do gymnastics or ride a bide. I'd be more than happy to be proven wrong on this point though, then my line of research is not dead-end =)
[1] https://youtu.be/x10964w00zk
There under "demo" are a few videos at 2x and 4x speed. It's slow. None of the videos include audio of the verbal commands or the latency between commands and action.
$35K for a robot that can putter around the house doing basic stuff is just not that high of a bar. With a 10 year life span, that's $3.5k/year, or $10/day. Doing 1 hour of useful minimum wage work around the house is just not that high of a bar - doing laundry, cleaning, tidying up, weeding, wiping surfaces down, taking out the garbage etc. If it can do some combination of those, it would make sense for basically every household. And it doesn't need to be able to do the crazy parkour of Boston Dynamics to achieve this. Our world is generally designed to be operable by all sorts of people - disabled, old etc. Crazy athleticism isn't required to do useful work.
They thing to keep in mind about the "Boston Dynamics approach" is that in order to solve the hardest real world problems, you need to do one of two things: 1) Overshoot the capabilities of the system so that it is robust when actually deployed subject to the uncertainty of the real world (e.g. motivating athletic intelligence). 2.) Grossly constrain your environment so that uncertainty is not a factor. This is (and has been) happening in warehouses and factories for decades.
There's a cool part where they ask the robot to pick up a lion from a group of toy figures it hasn't seen before. After it does it correctly the NYT reporter asks it to pick up the extinct animal and the robot picks up the dinosaur toy.
P.S.: I also was told that the key here is that in automated planning you can't have a human in the loop doing the actual learning. If you are going to prompt engineer or get a human in the loop to a degree that you effectively fool yourself that the robot is solving a problem then its not planning.
[1] https://en.wikipedia.org/wiki/Sussman_anomaly [2] https://chat.openai.com/share/16a8a0e9-7422-41da-a192-6393cc... [3] https://twitter.com/rao2z/status/1599462959788744704?s=20
[1] https://arxiv.org/abs/2305.15771 [2] https://youtu.be/x10964w00zk
To fully match the human model, AGI based on neural networks will need to dynamically receive information from sensors of all kinds [to be able to sample at least five major categories of physical stimuli, like us]. If we want to reach the level of the neural networks that we carry in our heads, we have to learn more about the role of other brain structures, such as glial cells, which are known to influence the activation and moderation of synapses in the human brain [which correspond to the 'weights' in artificial neural networks], and then apply them to our technological endeavors.
Without it two things will happen
1. We will never have strong AI 2. Strong AI that works virtually only, aware of the world and material reality, but denied access to them by its creators.
Imagine that it is possible to take a snapshot of a human mental state [an analogue of .h5 weights] and then run it on hardware as we do with our neural networks. It would certainly be a nightmare for a mind endowed with "qualia" to find itself enclosed in a metal box [brain in a vat] with no access to the senses. There are clinical cases analogous to this condition, for example in states where the patient is aware of everything around him but is unable to respond.
These issues highlight the ever-present ethical boundaries of IA research. Without integration with real-time sensors AI will always be incomplete and/or inhumane.
1. Choosing a workflow(i.e. a series of general steps that can achieve a goal) 2. Generating low level policy actions given a state(i.e. sensor data).
Like lets say the Workflow is to clean the room. Now there is a big chair in the way and a USB wire there too, the robot could just decide to move around the chair but it could pick up the wire and place it on the table and clean the area. Generating policy actions this way seems to be very much in line with simple VQA based things proposed in the paper, or so it would seem...