If they think this plan of "connecting" word to actions is promising then they should ask it to tie a shoelace based on "natural language" description. I'm shocked. I really did not expect that MIT would foster such naivety. They completely disregard the fact that natural language is a very crude description of physical actions. Perhaps next they will include in the instructions numerical values, and rediscover CNC.
Indeed, I wonder if natural language is the right interface to this problem.
Reading earlier mathematical texts (eg, Euclid's Elements) and their "natural language" proofs reminds me that a picture is truly worth a thousand words!
It seems more like they used a classifier to evaluate the situation and decide what state the system should be in. Not sure that the LLM contributed that much. LLMs are mediocre planners at this point.
Yes and no. LLMs are definitely struggling with long horizon planning in complex situations, and can hallucinate/misunderstand environmental states such that they can recommend nonsensical or impossible steps. But we are starting to see more research that "grounds" LLM answers in a set state. Where they shine in the planning stage is they provide a sense of contextual heuristics we didn't have available before without a lot of expert domain programming.
LLMS in their current state are certainly not the answer, but point to some exciting possibilities that we couldn't do before because of sheer complexity.
They're trying to get it to do hierarchical planning. People (like Yann LeCun) have complained that nobody knows how to do that in AI, only when they say "AI" they mean "neural nets" and ignore planning -as in Automated Planning and Scheduling, the discipline that has its own conference and journal, and international competition and thousands of people working at it, and dozens of systems, some commercial, some free, and all; but I guess that actually works so it's not AI.
Less sarcastically, automated planners need domain knowledge to break up a task into sub-tasks, and the neural net folks don't like that because they are taught at ML 101 that everything shall be a) end-to-end differentiable and b) have no human in the loop. LLMs seem to encode some kind of domain knowledge, the "common sense" that the article's title mentions, and so there's various teams trying to get robots to follow instructions with LLM-in-the-loop. A year ago I think it was LLMs generating PDDL, now it's natural language labels that obviously need a translation.
This won't work though a) because it's trained by imitation learning and nobody cares about a robot that can scoop marbles from bowl to bowl but would need re-training to do the same thing with e.g. strawberries and b) the LLM itself isn't grounded anywhere and has no way to self-correct its self-correction, once it falls into error. This, too, shall pass.
Btw, are those people mad? Marbles? Couldn't they find something less painful? What happens when your robot overturns the bowl and there's marbles all over the floor of your lab?
I gave up on my Electrolux Ipure 9 since I spent more time babysitting it than it would take to use a normal vacuum. The dumb ones that just bounced and loled around worked better in practice since they eventually cleaned everywhere, like lawn movers, and didn't give up randomly becouse they were lost.
Jeez I am bitter I bought that one. It is up there with my HTC Vive of worst expensive tech gadgets.
Thankfully I already have common sense enough not to put a household robot in my home. It will certainly be used to collect as much data on what goes on inside my house as possible and send that back to at least one company who will leverage that data against me.
No way in hell would I bring a robot butler into my home unless I can be certain that it won't be collecting and leaking sensitive information about my life/household to others. We have too much of that going on as it is.
>> The researchers illustrate their new approach with a simple chore: scooping marbles from one bowl and pouring them into another.
I would love to have a robot that can carry out this common household task. I have to do this every week, on Saturdays, when the bowl holding the marbles overflows with the offerings deposited by the Roc birds at the top of the windcatcher in my desert redoubt. The marbles are sometimes made of quartz, sometimes nephrite, rarely ruby, emerald, sapphire or diamond and so they have to be disposed off, or the sand worms will come for them.
More seriously, training a classifier to map LLM instructions to robot arm actions, as the researchers in the article did is smart, but their motivation was error correction and that, while intersting, is not the big problem. The big er limitation of imitation learning is that if you teach a robot to scoop marbles from bowl A and pour them into bowl B by imitation learning ... that's the only thing it will be able to do. Scoop marbles from bowl A to bowl B. The same bowls A and B, placed in the same positions, with only a small error tolerance (centimeters) and the marbles better be the same colour as the ones seen in training, the spoon better be the same shape, or else!
Or else you're going to be stepping on lots of marbles. My common sense tells me that can be very painful.
13 comments
[ 2.9 ms ] story [ 47.9 ms ] threadReading earlier mathematical texts (eg, Euclid's Elements) and their "natural language" proofs reminds me that a picture is truly worth a thousand words!
LLMS in their current state are certainly not the answer, but point to some exciting possibilities that we couldn't do before because of sheer complexity.
Less sarcastically, automated planners need domain knowledge to break up a task into sub-tasks, and the neural net folks don't like that because they are taught at ML 101 that everything shall be a) end-to-end differentiable and b) have no human in the loop. LLMs seem to encode some kind of domain knowledge, the "common sense" that the article's title mentions, and so there's various teams trying to get robots to follow instructions with LLM-in-the-loop. A year ago I think it was LLMs generating PDDL, now it's natural language labels that obviously need a translation.
This won't work though a) because it's trained by imitation learning and nobody cares about a robot that can scoop marbles from bowl to bowl but would need re-training to do the same thing with e.g. strawberries and b) the LLM itself isn't grounded anywhere and has no way to self-correct its self-correction, once it falls into error. This, too, shall pass.
Btw, are those people mad? Marbles? Couldn't they find something less painful? What happens when your robot overturns the bowl and there's marbles all over the floor of your lab?
Jeez I am bitter I bought that one. It is up there with my HTC Vive of worst expensive tech gadgets.
Even the humble Roomba has already been looked at as a spy for selling maps of people's houses (https://gizmodo.com/roombas-next-big-step-is-selling-maps-of...) and took photos of a woman on the toilet which were leaked online (https://www.businessinsider.com/roomba-photos-recorded-bathr...)
No way in hell would I bring a robot butler into my home unless I can be certain that it won't be collecting and leaking sensitive information about my life/household to others. We have too much of that going on as it is.
I would love to have a robot that can carry out this common household task. I have to do this every week, on Saturdays, when the bowl holding the marbles overflows with the offerings deposited by the Roc birds at the top of the windcatcher in my desert redoubt. The marbles are sometimes made of quartz, sometimes nephrite, rarely ruby, emerald, sapphire or diamond and so they have to be disposed off, or the sand worms will come for them.
More seriously, training a classifier to map LLM instructions to robot arm actions, as the researchers in the article did is smart, but their motivation was error correction and that, while intersting, is not the big problem. The big er limitation of imitation learning is that if you teach a robot to scoop marbles from bowl A and pour them into bowl B by imitation learning ... that's the only thing it will be able to do. Scoop marbles from bowl A to bowl B. The same bowls A and B, placed in the same positions, with only a small error tolerance (centimeters) and the marbles better be the same colour as the ones seen in training, the spoon better be the same shape, or else!
Or else you're going to be stepping on lots of marbles. My common sense tells me that can be very painful.