Show HN: OK-Robot: open, modular home robot framework for pick-and-drop anywhere (ok-robot.github.io)
We based everything off of the current best machine learning models, and so things don't quite work perfectly all the time, so we are hoping to build it together with the community! Our code is open: https://github.com/ok-robot/ok-robot and we have a Discord server for discussion and support: https://discord.gg/wzzZJxqKYC If you are curious what works and what doesn't work, take a quick look at https://ok-robot.github.io/#analysis or read our paper for a detailed analysis: https://arxiv.org/abs/2401.12202
P.S.: while the code is open the project unfortunately isn't fully open source since one of our dependencies, AnyGrasp, has a closed-source, educational license. Apologize in advance, but we used it since that was the best grasping model we could have access to!
Would love to hear more thoughts and feedback on this project!
118 comments
[ 2.4 ms ] story [ 206 ms ] thread- How does it know what objects are? Does it use some sort of realtime object classifier neural net? What limitations are there here?
- Does the robot know when it can't perform a request? I.e. if you ask it to move a large box or very heavy kettlebell?
- How well does it do if the object is hidden or obscured? Does it go looking for it? What if it must move another object to get access to the requested one?
You basically hit the nail on the head with these questions. This work is super cool, but you named a lot of the limitations with contemporary robot learning systems.
1. It's using an object classifier. It's described here (https://github.com/ok-robot/ok-robot/tree/main/ok-robot-navi...), but if I understanding it correctly basically they are using a ViT model (basically an image classification model) to do some labeling of images and projecting them onto a voxel grid. Then they are using language embeddings from CLIP to pair the language with the voxel grid. The limitations of this are that if they want this to run on the robot, they can't use the super huge versions of these models. While they could use a huge model on the cloud, that would introduce a lot of latency.
2. It almost certainly cannot identify invalid requests. There may be requests that are not covered by their language embeddings, in which case the robot would maybe do nothing. But it doesn't appear that this system has any knowledge of physics, other than the hardware limitations of the physical controller.
3. Hidden? Almost certainly wouldn't work. The voxel labeling relies on a module that labels the voxels and without visual info, it can't label them. Also, as far as I can tell, it doesn't appear to have very complex higher-order reasoning about, say, that a fork is in a drawer, which is in a kitchen, which is often in the back of a house. Partially obscured? That would be subject to the limitations of the visual classifier, so it might work. ViT is very good, but it probably depends on how obscured the object is.
Will all the recent work to make gen. AI faster (see groq for LLM & fal.ai for stable diffusion), I wonder if the latency will become low enough to make this a non-issue or at least good enough
Like something that sits next to your router (or more likely, routers that come stock with it).
1. what am i picking up? - this can be AI in the cloud as it does not need to be real time
2. how do i pick it up? - this can be AI in the cloud as it does not need to be real time - the robot can take its time picking the object up
3. after pickup, where do i put the object? localization while moving probably needs to be done locally but identifying where to put down can be done via cloud, again, no rush
4. how do put the object down? again, the robot can take its time
You can see in the video the robot pauses before performing the actions after finding the object in its POV, so real time isn't a hard req for a lot of these
> - How does it know what objects are? Does it use some sort of realtime object classifier neural net? What limitations are there here?
We use Lang-SAM (https://github.com/luca-medeiros/lang-segment-anything) to do most of this, with CLIP embeddings (https://openai.com/research/clip) doing most of the heavy lifting of connecting image and text. One of the nice properties of using CLIP-like models is that you don't have to specify the classes you may want to query later, you can just come up with them during runtime.
> - Does the robot know when it can't perform a request? I.e. if you ask it to move a large box or very heavy kettlebell?
Nope! As it is right now, the models are very simple and they don't try to do anything fancy. However, that's why we open up our code! So the community can build smarter robots on top of this project that can use even more visual cues about the environment.
> - How well does it do if the object is hidden or obscured? Does it go looking for it? What if it must move another object to get access to the requested one?
It fails when the object is hidden or obscured in the initial scan, but once again we think it could be a great starting point for further research :) One of the nice things, however, is that we take full 3D information in consideration, and so even if some object is visible from only some of the angles, the robot has a chance to find it.
Labor isn't free. Building custom PCBs and hardware in low quantity isn't cheap. Building, calibrating, and testing robots isn't cheap.
If you need one gas line, you can get a swivel for a normal shop hose reel for $15. If you need two gas lines on the same axis, the part is similar but way lower volume, so you have to go to a specialty supplier and the price is $350.
The business that makes hose reel swivels makes lots of high volume parts, has lots of competition, and needs to charge close to cost to sell them. The business that makes specialty gas swivels for industry that offers multiple gas lines in one swivel, lots of different options, and makes them higher quality needs to charge a lot more to keep their business operational.
Thank you
I want mini robots cleaning dust and debris, silently and out of my way. I don’t want macro bots getting in my way
Laundry, cooking, dishes, sweeping, vacuuming, and other constantly recurring tasks are what I would love to see automated not just a “robot that sweeps” like the market has been trying to sell me.
Ever since I read the second shift book about the unpaid extra 40 hour week women work doing domestic tasks I’ve dreamed of robots replacing that for humanity. It’s a massive cost to people individually and humanity overall, and kind of a silent epidemic.
It’s crazy but freeing up half of humanity from the drudge work of daily chores is one of the most obvious disruptive technology plays. I rarely hear people put the robot revolution in this context, but I very much think we should start doing so.
Here’s a good overview for the uninitiated:
https://www.americanprogress.org/article/unequal-division-la...
But I worry very much that tools like this will be used primarily to increase corporate profits and reduce money spent on humans, rather than remove drudgery from people’s lives and allow them to do things more aligned with their goals and natures.
E.g., if we make a cleaning robot, hotels will replace half their staff — what will these people do for a living? Work in an AI sweatshop, categorizing images of child abuse?
Old-school science fiction often proposed that we’d be entering a new age of art and leisure, as robots and AI take over menial tasks. In fact today I think we’re seeing AI and robots — in part — taking jobs from humans, and in order to provide entertainment and economic leverage to richer humans.
It’s making me reevaluate all that old science fiction, as it seemed to require an invisible 90% of the population basically working for the AIs so that the AIs can curate a great life for a stratospherically-wealthy minority.
It was also predicted in the mid 20th century that rising productivity would create a shorter work-week; instead we have figured out how to prevent workers from being compensated for higher productivity.
https://www.epi.org/productivity-pay-gap/
For golden age sci fi theories of human work vs leisure to actually take hold, we need universal basic income, or some other monetary theory that allows us to value other people for being alive rather than solely for being feudal slaves of deranged billionaires.
"Hotel maid" as a job really shouldn't exist when robots can do it better and more consistently (which isn't true yet). At that point, not before, should be considered beneath human dignity. But we definitely need an answer for what happens to the newly undignified human.
I think chores aren't necessarily the terrible boredom. But having a robot as an option, you can do them as a sort of hobby if and when you want. That seems nice.
I think we also will need to develop maturity to deal with our free time, but it's probably not the disaster I've seem many claim (that we lose meaning) -- maybe their way to cope with an unfair world? or my way to cope with laziness.
The main thing is how to protect ourselves from rulers when we aren't necessary for labor. It seems like a difficult but solvable problem. Being able to choose how much to work (and play) is the dream!
[1] https://pantor.github.io/speedfolding/
[0] https://www.youtube.com/watch?v=wOpdDxJzNkw
Everything has its limits. Many years ago I was involved in building a series of staircases in a rock climbing area inside a park. There were about a hundred steps in a handful of orientations to get from the parking lot over a rocky hill to the small valleys behind. The project was primarily to prevent trail erosion and falls. These steps weren't going to even have handrails. (Think 2x6 framed boxes filled with dirt and bolted to the rock.) Then someone in government said if we wanted to use donated money inside a park we would have to somehow make the project wheelchair accessible. All stop. Project over. No stairs were built. Access trail remained a mess.
We were going to replicate these stairs from another climbing area in BC. There is no way to make such a thing wheelchair accessible.
https://sonnybou.ca/ssbou2001/skaha01.jpg
Rock climbing areas tend to be inaccessible or at least very rough terrain. Ironically, a vertical rock surface can be made accessible. There are actually many disabled climbers out there. But with a mixed dirt/rock/scree slope you basically need to install a mile-long ramp.
Just because we have Linux, Python and JavaScript doesn't mean nobody's playing with assembly and experimenting with new ideas at a more fundamental level.
I have a suspicion that the sort of people who think they wouldn't do something hard just because other people have it too easy weren't going to do it in the first place.
"improving things" and "mandatory requirements that, in some cases, can go against common sense" are not the same things.
1. https://en.wikipedia.org/wiki/Curb_cut_effect
2. https://99percentinvisible.org/episode/curb-cuts/
So you have to do it right to keep the potential harm as low as possible and not forget about security in the face of rewarding improvements. And watch your step, of course.
Might also be applicable in the context of self-learning household robots and their potential to burn down that house :)
…to (very liberally) paraphrase Rory Sutherland.
Since motors are capable of very precise movements and errors accumulate, this is a best practice when starting new movements. Humans instead have a complex hand-eye coordination system that trained all of our lives (and some people are better at it than others).
Ideally the "zeroing" should be done once when the robot "wakes up" or only once in a while, and there should be digital encoders on all motors, the position should always be known within a tiny margin of error, and not enough to cause a problem for positioning. At least that's how I'd do it, I'm not sure how they built this thing.
I envision an integration with a mobility aid (eg, a wheelchair) for someone with limited control over their limbs. Imagine a "smart" exoskeleton that can help with otherwise impossible tasks -- it could be a game-changer for so many people.
So, a gyro-stabilized platform like a segway that I can send back and forth from point A to point B on a not-terrible-but-rough (walking path) route.
I have tried to stay abreast of the options in the past and have never seen anything that matches this ... does anyone know if there is anything new that matches this use-case ?
(the use-case is a tray of drinks and hors d'oeuvres that needs to go from one part of a property to another without spilling ... needs to be minimally all-terrain)
Not sure I’ve seen them take drinks though, but definitely food.
The technology's out there, just need to glue it all together.
https://dobb-e.com/
The primary difference is that this is zero-shot (meaning the robot needs 0 (zero!) new data in a new home) but has only two skills (pick and drop); where Dobb-E can have many skills but will need you to give some demonstrations in a new home.
https://hello-robot.com/stretch-embodied-ai
You are standing in a kitchen. Ahead of you to your right there is a large refrigerator with the handle on the right side. There is a set of cabinets to your left with a plate sitting on the counter above them.
> get beer
You don't see any beer here.
<< COT: I know that beer is often found in the fridge. I should try opening the refrigerator
> open fridge
Opening the refrigerator reveals 4 cans of beer.
> get beer
taken
Obviously we're still several years from this working, but it's very exciting to consider. Interactive Fiction narrative fed by real sensors plus chain-of-thought blocks as internal monologue.
It would be really cool if the robot could just know where your keys are by attaching some kind of tile-type thing to it. If it already has a scan of your home, theoretically it could show a photo. But I have no idea if it’s possible to pinpoint an object via rfid.
You could take video data and have fuzzy identification of objects moving around, then throw away the video and keep track of the objects, the blue floppy thing (gloves) and the metal shiny deforming things (keys) then have a more constructive dialog about the keys. A voice responding, what do the keys look like? Is there a blue square thing on the key ring? The less identifiable the object the funnier the discussion. What shirt? You have many shirts! Oh, the blue one, you have 4 of those, one in the sink, one behind the bed, one in the laundry basket, one in the closet. Oh the one with stripes! Why didn't you say so, it's behind the bed bro.
It could also ask you if they are suppose to be on the outside in the front door after you close it.
I use this quite a lot actually. Being lazy I take photographs of components and boards and ask it how to wire them to my esp32. It’s able to distinguish the board, chip, etc, as well as the pinouts from a set of photos and tell me what wires to where and anything of note. It’ll often even suggest helpful libraries for the parts. It’s essentially magic.