something requiring navigating stairs while holding something full like a laundry basket. bronze - straight stairs, silver - one 90deg turn. gold - spiral.
something requiring co-ordination between 2 robots. think relay race which the olympics has. So say, moving a couch together.
btw love the idea and the silver body suit. good stuff.
Maybe careful application of large amounts of force? Opening a jar, peeling garlic, splitting a squash, opening a soda can. This category seems like a good test of "grip" strength + force feedback + sense of touch.
Consider examples using building tools like screwing in a drywall screw, or hammering a nail, using a paint roller, caulking a sink, minor plumbing repair with a torch and solder. These differ enough in terms of forces, state changes, and combined dexterity/acuity (two-handed proprioception) from the windex, sandwich and key examples
I'd go for something in the manipulation of ropes or wires.
State of the art seems to be that they can untangle a loosely knotted cord.
Untying a short rope with a tightly pulled overhand knot in the middle seems like it's decades away. You have to be able to grip it well enough, then twist the rope and push (even though every physicist says pushing a rope is impossible).
Standard evening family home tidy/reset - toys, books, clothes, shoes away in their places. All over the house.
Oh, and load/unload dishwasher. Same with laundry machines. Along with folding laundry, these are the domestic robot equivalents of 'de-mining' and 'search and rescue': the classic motivating use cases for mobile autonomous robots.
I love your list and it makes me think we are so far away from these things ever being feasible/cost effective compared to just hiring a poor person to do it. And the world is making a lot of poor people right now.
(HN link on Substack points at empty page instead of this one, at least before I made this comment.)
What I think is missing is marathon events. Biathalons and Triathalons.
We all know LLMs have a rather limited context window. Thus seeing robots do longer chains of events would be interesting to see that they're capable than a possibly rigged demo.
Something like: move a stack of boxes from one room to another. The boxes at the end also need to be stacked up. or how about pick up a box, go up some stairs, open a door, and put the box on a shelf on the other side.
Also, the real world is sloppy and messy and dirty and, to be real, kinda janky sometimes. Gold for unlocking a door with a key at a well-maintained office complex, (and opening it, and walking through it) is one thing, because facilities is going to replace the lock before it gets old and needs replacing, and we can assume the door fits in the frame properly so it doesn't need to be shoved or lifted up or yanked in order to be opened is easier than. But the real world is messy and sloppy and you gotta jiggle the key in just the right way in order to get it to work.
Closing the door (assuming the robots weren't raised in a robot barn) is also harder than it looks if the door is shitty and needs a proper slam in order to be fully closed. Also, the robot locking the door behind itself after it comes in.
Scanning a key card and opening a door, but the first try fails.
We're a long way from a general robot that can screw a simple screw together like you would to assemble Ikea furniture.
Object recognition.
Gather only the dishes from a messy coffee table and put them in the dish bin.
Pick up only the clothes from a messy floor and bed, and put them in the hamper.
Dump a hamper of clothes onto a table, and sort out stuff that doesn't want to go into the washing machine.
Terrain traversal.
Just walk 500 ft, but theres increasing levels of obstacles in the way.
We all saw Boston dynamics robot parkour videos, but what I want to see is a robot make it from the front door of Simpsons house to the kitchen in the back, but it's got to go through the living room, but it's hella messy, with Maggie and Bart and Lisa’s crap strewn all over, Homer’s got some beer bottles, some empty, some full, all over the floor and on the table, and all the robot has to do is walk from one side of the room to the far side of the room without stepping on anything, or knocking anything over. (Simpsons merely being a home layout that's familiar to most people. Doesn't need to actually be them.)
Ducking under a low ceiling. Climb over a barrier,
of varying shapes and sizes.
Other loocomotion. how much weight in its arms in front of it, holding a 5-lb briefcase with one hand while walking. Can it carry something on its back? What's the limit? Can it give piggyback rides?
A category for simulated. Let companies show off their robot's kinematics control systems, so have something on the level of CoppeliaSim, so the motors and the gears and the actuators are themselves simulated, vs a simple 3d video game where they are not. Plug their model into the simulated robot and see how well it just walks. If we remember QWOP, it's harder than it looks!
Obviously it's not going to be totally 100% accurate to the real world. The benefit of this is it lets people complete from all over world without having to replicate a very specific setup in the physical world, and compete from wherever they live am not have to fly to your facility to test, opening up a whole new world of contestants because they can now compete because they can afford it now.
At the end of the day, the most important challenge is, can it pick up a battery from the shelf, swap it with one of the two in its chassis, and put the dead one it just pulled out onto the charger?
You need to manipulate a large sheet, and you probably need to move around, bend down and lean over to reach all the corners. Bonus points for neat hospital corners on a flat sheet.
Putting pillows in pillowcases is another fun one. Usually pretty easy, probably a bronze medal.
Gold medal: put a UK super king size duvet inside a duvet cover. It's huge and awkward, there are buttons, and it's almost but not quite square (why??) so there's a good chance you'll get it round the wrong way and have to rotate it 90 degrees.
Thanks for putting this together. This IMO is 1000x better than any other AI challenges to date. ARC-AGI is bullshit and has nothing to do with reality.
I’ll likely use some of your tests for our robotics (not just humanoid) testing in the future at least as some baselines.
Also I really liked you dressing up as a robot - that’s very fun and really reflects the point of robotics: replace human action for all tasks.
My suggestion: Identify a collapsed person in the home and render first aid (I need this because I have epilepsy and live alone)
Bronze: ID collapse and call emergency services
Silver: Bronze tasks + manipulate person into appropriate recovery position
Gold: Communicate details to emergency personnel and playback previous hours of interactions
Tie your shoe laces? Pet a cat. Open one of those scissors that are sold in those fused plastic boxes, requiring a scissor to open. Opening a packet of tissues (wet ones for extra challenge). Cook rice. Throw out the trash.
Interesting post and great reference to [1] about why laundry hits a sweet spot of capability.
Interestingly the repeated critiques in the article are about sensor richness: primarily force feedback and tactility, which indicates lacking hardware. Software only robotics has a long and fraught history, but it really feels to me that current industrial hardware could be driven more intelligently without much change. No doubt the "ideal" robot for any given task requires developments in both.
I'm also curious about safety, since generally capable mechanisms need a multilayered safety stack that includes semantics, and cobot certification is likely not enough anymore. Examples: feeding someone the wrong pill, pouring a glass of water into electronics, cutting vegetables vs fingers.
Here's a use case that seems more science fictional to me (as the parent of a 2yo) than warp drive: a robot that can gently restrain an uncooperative human baby while changing its diaper, with everything that entails: identifying and eliminating all traces of waste from all crevices, applying diaper cream as necessary, unfolding and positioning the new diaper correctly and quickly, always using enough but never too much force... not to mention the nightmare of providing any guarantees about safety at mass-market scale. Even one maimed baby, or even just a baby some robot neglects to prevent from falling off the changing table, is game over for that line of robots.
Is there any research program that could claim to tackle this? It's so far beyond folding laundry and doing dishes, which are already quite difficult.
I wouldn't bet my life on this tech _never_ materializing, but I would mistrust anyone who claimed it was feasible with today's tech. It calls for an entirely different kind of robotic perception, feedback, and control.
Why? There's nothing particularly special about this problem. I would bet a year for an alpha version, and production version in 5 years. We are not exactly limited by mechanical engineering here, there's nothing particularly unique about the human hand that can't be replicated. Tele operated surgical robotics have been a thing for decades. Give it a few months for the multimodal robotic VLM/LAMs to catch up. In many ways this particular problem is a lot more well defined than e.g. self driving cars.
I don't see why that would be so hard. This is potentially easier than reliably shooting guns at people.
That machine will look like a bean bag couch in rough shape of a giant human hand, with few of cooperative work robotic arms. The couch part hugs and secures all limbs of the baby to into the party escort submission position, then the cobots move in to find the disassembly markers on the diaper to tear it open to remove it. Then a showerhead, then a hair dryer, then baby powder sprayer can be brought out and ran to clean any residues and take care of rashes. Finally, the new diaper can be brought in, baby wrapped, and the double sided tapes on it lightly pressed on to secure it.
The entire machine would probably cost less than 10 million USD per unit if mass produced at reasonable scales, and most technological elements needed in such machines would be readily available.
This should be achievable once hand job robots are widely deployed and proven to be safe. Men are usually sacrificed first and I'm sure there will be volunteers.
> It calls for an entirely different kind of robotic perception, feedback, and control.
Nearly every surgeon used advanced robots to assist with surgeries. My uncle does kidney transplants and he uses robots, so do most surgeons.
For robots to be developed, someone must pay for it. Doctors get skill++ with the surgery robots. There may be vasectomy robots that assist with circumcision, or some actively being developed that doctors would pay for. That would be a far more interesting development than changing diapers. Unless extremely competent and skilled people ( +1000/hour) need them for their work, they won't be developed, so I don't think a diaper changing robot is being developed.
Think about how fast progress is being made now. When I was a kid in the early 2000s we would see some basic robot progress on movements (almost always from Boston Dynamics or sometimes China) and we thought it was incredible. Robot dogs running was amazing and five or so years later a backflip blew our minds. Those robots were specially designed and didn't look humanoid. Now we have bi-pedal humanoid robots and they walk and move fairly capably - even able to get up after falls. Now within the last year I have seen them learn Kung Fu, become really fast at getting up, become resistant to being knocked down by quite a lot of force, and now even doing tasks like those shown here.
Just imagine 2050 if the progress continues at this rate. I am both excited and really scared.
Great post! Now somebody with the connections just needs to make it happen. For event five, slippery when wet, you should definitely include drying your hands on a towel, as it serves an important hygienic function.
Platinum Medal: Complete a gold medal task in the presence of multiple children between the ages of 3 and 8. The children must remain safe throughout (and after) the entire process.
Can I say something: could you all please stop inserting contaminated tool back into a jar of food? You use a clean tool to take out the amount you eat, and that's it.
You can put it back if the tool had touched nothing but: air, the food in the jar, and your hand at the operating end. Otherwise, that butter knife stays on your dish for the rest of the meal. The exception would be if you cleaned the tool, like bare minimum by wiping with a brand new piece of tissue paper(but that's kind of wasteful).
Is that an outrageous ask? I know it's probably not a huge deal, like free water and such, and my techniques are that of total amateur being never professionally involved in medical and/or bio science fields, but just, how can you stand possible breadcrumbs IN THE JAR!?
A wonderful article, truly insightful. But it feels more like a series of challenges than the Olympics.
This is the Humanoid Robot Olympics held in China earlier this year. https://www.youtube.com/watch?v=5Y-tElcmJVE Instead of focusing on individual technical, it requires completing comprehensive movement tasks. For example, long-distance running tests battery managment and durable of joint, while boxing tests flexibility and coordination.
Many participating teams or manufacturers, after failing, return to tackle the specific issues they encountered. This process drives the progress of the entire industry. In my view, this is worthy of being called an Olympic Game—in both form and spirit.
44 comments
[ 1.5 ms ] story [ 58.6 ms ] threadsomething requiring co-ordination between 2 robots. think relay race which the olympics has. So say, moving a couch together.
btw love the idea and the silver body suit. good stuff.
Consider examples using building tools like screwing in a drywall screw, or hammering a nail, using a paint roller, caulking a sink, minor plumbing repair with a torch and solder. These differ enough in terms of forces, state changes, and combined dexterity/acuity (two-handed proprioception) from the windex, sandwich and key examples
Ikea product assembly for gold medal.
State of the art seems to be that they can untangle a loosely knotted cord.
Untying a short rope with a tightly pulled overhand knot in the middle seems like it's decades away. You have to be able to grip it well enough, then twist the rope and push (even though every physicist says pushing a rope is impossible).
Oh, and load/unload dishwasher. Same with laundry machines. Along with folding laundry, these are the domestic robot equivalents of 'de-mining' and 'search and rescue': the classic motivating use cases for mobile autonomous robots.
What I think is missing is marathon events. Biathalons and Triathalons.
We all know LLMs have a rather limited context window. Thus seeing robots do longer chains of events would be interesting to see that they're capable than a possibly rigged demo.
Something like: move a stack of boxes from one room to another. The boxes at the end also need to be stacked up. or how about pick up a box, go up some stairs, open a door, and put the box on a shelf on the other side.
Also, the real world is sloppy and messy and dirty and, to be real, kinda janky sometimes. Gold for unlocking a door with a key at a well-maintained office complex, (and opening it, and walking through it) is one thing, because facilities is going to replace the lock before it gets old and needs replacing, and we can assume the door fits in the frame properly so it doesn't need to be shoved or lifted up or yanked in order to be opened is easier than. But the real world is messy and sloppy and you gotta jiggle the key in just the right way in order to get it to work.
Closing the door (assuming the robots weren't raised in a robot barn) is also harder than it looks if the door is shitty and needs a proper slam in order to be fully closed. Also, the robot locking the door behind itself after it comes in. Scanning a key card and opening a door, but the first try fails.
We're a long way from a general robot that can screw a simple screw together like you would to assemble Ikea furniture.
Object recognition.
Gather only the dishes from a messy coffee table and put them in the dish bin.
Pick up only the clothes from a messy floor and bed, and put them in the hamper.
Dump a hamper of clothes onto a table, and sort out stuff that doesn't want to go into the washing machine.
Terrain traversal.
Just walk 500 ft, but theres increasing levels of obstacles in the way.
We all saw Boston dynamics robot parkour videos, but what I want to see is a robot make it from the front door of Simpsons house to the kitchen in the back, but it's got to go through the living room, but it's hella messy, with Maggie and Bart and Lisa’s crap strewn all over, Homer’s got some beer bottles, some empty, some full, all over the floor and on the table, and all the robot has to do is walk from one side of the room to the far side of the room without stepping on anything, or knocking anything over. (Simpsons merely being a home layout that's familiar to most people. Doesn't need to actually be them.)
Ducking under a low ceiling. Climb over a barrier, of varying shapes and sizes.
Other loocomotion. how much weight in its arms in front of it, holding a 5-lb briefcase with one hand while walking. Can it carry something on its back? What's the limit? Can it give piggyback rides?
A category for simulated. Let companies show off their robot's kinematics control systems, so have something on the level of CoppeliaSim, so the motors and the gears and the actuators are themselves simulated, vs a simple 3d video game where they are not. Plug their model into the simulated robot and see how well it just walks. If we remember QWOP, it's harder than it looks!
Obviously it's not going to be totally 100% accurate to the real world. The benefit of this is it lets people complete from all over world without having to replicate a very specific setup in the physical world, and compete from wherever they live am not have to fly to your facility to test, opening up a whole new world of contestants because they can now compete because they can afford it now.
At the end of the day, the most important challenge is, can it pick up a battery from the shelf, swap it with one of the two in its chassis, and put the dead one it just pulled out onto the charger?
You need to manipulate a large sheet, and you probably need to move around, bend down and lean over to reach all the corners. Bonus points for neat hospital corners on a flat sheet.
Putting pillows in pillowcases is another fun one. Usually pretty easy, probably a bronze medal.
Gold medal: put a UK super king size duvet inside a duvet cover. It's huge and awkward, there are buttons, and it's almost but not quite square (why??) so there's a good chance you'll get it round the wrong way and have to rotate it 90 degrees.
Gold Medal: Separate the egg white from the yolk!
I’ll likely use some of your tests for our robotics (not just humanoid) testing in the future at least as some baselines.
Also I really liked you dressing up as a robot - that’s very fun and really reflects the point of robotics: replace human action for all tasks.
My suggestion: Identify a collapsed person in the home and render first aid (I need this because I have epilepsy and live alone)
Bronze: ID collapse and call emergency services
Silver: Bronze tasks + manipulate person into appropriate recovery position
Gold: Communicate details to emergency personnel and playback previous hours of interactions
Interestingly the repeated critiques in the article are about sensor richness: primarily force feedback and tactility, which indicates lacking hardware. Software only robotics has a long and fraught history, but it really feels to me that current industrial hardware could be driven more intelligently without much change. No doubt the "ideal" robot for any given task requires developments in both.
I'm also curious about safety, since generally capable mechanisms need a multilayered safety stack that includes semantics, and cobot certification is likely not enough anymore. Examples: feeding someone the wrong pill, pouring a glass of water into electronics, cutting vegetables vs fingers.
[1] https://substack.com/redirect/82d94852-76b6-4b0d-8595-86e46a...
https://substack.com/@isitpropaganda/note/c-167073531?utm_so...
Is there any research program that could claim to tackle this? It's so far beyond folding laundry and doing dishes, which are already quite difficult.
I wouldn't bet my life on this tech _never_ materializing, but I would mistrust anyone who claimed it was feasible with today's tech. It calls for an entirely different kind of robotic perception, feedback, and control.
That machine will look like a bean bag couch in rough shape of a giant human hand, with few of cooperative work robotic arms. The couch part hugs and secures all limbs of the baby to into the party escort submission position, then the cobots move in to find the disassembly markers on the diaper to tear it open to remove it. Then a showerhead, then a hair dryer, then baby powder sprayer can be brought out and ran to clean any residues and take care of rashes. Finally, the new diaper can be brought in, baby wrapped, and the double sided tapes on it lightly pressed on to secure it.
The entire machine would probably cost less than 10 million USD per unit if mass produced at reasonable scales, and most technological elements needed in such machines would be readily available.
Nearly every surgeon used advanced robots to assist with surgeries. My uncle does kidney transplants and he uses robots, so do most surgeons.
For robots to be developed, someone must pay for it. Doctors get skill++ with the surgery robots. There may be vasectomy robots that assist with circumcision, or some actively being developed that doctors would pay for. That would be a far more interesting development than changing diapers. Unless extremely competent and skilled people ( +1000/hour) need them for their work, they won't be developed, so I don't think a diaper changing robot is being developed.
Just imagine 2050 if the progress continues at this rate. I am both excited and really scared.
You can put it back if the tool had touched nothing but: air, the food in the jar, and your hand at the operating end. Otherwise, that butter knife stays on your dish for the rest of the meal. The exception would be if you cleaned the tool, like bare minimum by wiping with a brand new piece of tissue paper(but that's kind of wasteful).
Is that an outrageous ask? I know it's probably not a huge deal, like free water and such, and my techniques are that of total amateur being never professionally involved in medical and/or bio science fields, but just, how can you stand possible breadcrumbs IN THE JAR!?
yes, but i thought this wasn't that kind of article.
Like, just a piece of silicone with a few sensors, pressure, heat, etc. that you can "glove" around a robot.