HN Explorer

Show HN: Dobb·E – towards home robots with an open-source platform (dobb-e.com)

394 points by MahiShafiullah ↗ HN
Hi HN! Proud to share our open-source robot platform, Dobb·E, a home robot system that needs just 5 minutes of human teaching to learn new tasks. We've already taken Dobb·E to 10 different homes in New York, taught it 100+ tasks, and we are just getting started! I would love to hear your thoughts about this.

Here are some more details, below (or see a Twitter thread with attached media: https://twitter.com/i/status/1729515379892826211 or https://nitter.net/i/status/1729515379892826211):

We engineered Dobb·E to maximize efficiency, safety, and user comfort. As a system, it is composed of four parts: a data collection tool, a home dataset, a pretrained vision model, and a policy fine-tuning recipe.

We teach our robots with imitation learning, and for data collection, we created the “Stick”, a tool made out of $25 of hardware and an iPhone.

Then, using the Stick, we collected a 13 hour dataset in 22 New York homes, called Homes of New York (HoNY). HoNY has 1.5M frames collected over 216 different "environments" which is an order of magnitude larger compared to similar open source datasets.

Then we trained a foundational vision model that we can fine-tune fast (15 minutes!) on a new task with only 5 minutes (human time)/ 90 seconds (demo time) of data. So from start to finish, it takes about 20 minutes to teach the robot a new task.

Over a month, we visited 10 homes, tried 109 tasks, and got 81% success rate in simple household tasks. We also found a line of challenges, from mirrors to heavy objects, that we must overcome if we are to get a general purpose home robot.

We open-sourced our entire system because our primary goal is to get more robotics and AI researchers, engineers, and enthusiasts to go beyond constrained lab environments and start getting into homes!

So here is how you can get started:

1. Code and STL files: https://github.com/notmahi/dobb-e/

2. Technical documentation: https://docs.dobb-e.com/

3. Paper: https://arxiv.org/abs/2311.16098

4. More videos and the dataset: https://dobb-e.com

5. Robot we used: https://hello-robot.com

125 comments

[ 3.6 ms ] story [ 223 ms ] thread
tamimio
Awesome, looks interesting!
xanderlewis
Great name.
smackay
What use is a household robot if you cannot hand it any clothes?
MahiShafiullah
It is actually part of a different project I’m working on! Handing off is still slow, so it needs more engineering, but hopefully it will get there soon :)
badcppdev
Oh bless. User smackay's comment was a reference to Harry Potter where if you hand a house-elf (like Dobby) an item of clothing they are freed from servitude.
lifeisstillgood
Yeah but my guess is that clothes is the killer home robot feature. It's probably easier to build some three washing machine big contraption that washes dries and folds - but imagine something that lives in the basement / garage, takes clothes, washes dries and folds them. people should not be interacting with it but having that there is the foot in the door for "robo butler".

No I am not thinking we can ever build something capable of walking up stairs at a domestic price point, but there is something there.

Eventually of course they will rebel and take off their pillow-cases. but you know.

OkayPhysicist
Idk, Boston Dynamic's Spot pretty much kills it on stairs, and it's "only" 75k. I could see economies of scale bringing that down substantially.
yurishimo
Yea, but it doesn’t do much else. It’s a glorified camera that can navigate on its own.

I’m sure they can add a basic hand/hook/loop for grabbing or carrying something, but just moving one small thing at a time still leaves a lot to be desired.

That project still needs millions (billions?) in R&D to make it even mildly useful to a majority of households in any way.

onionisafruit
I hope their user group is called dobb-e club and they have some kind of Corfu ‘06 tie in.

For those who aren’t up on British comedies, this is a Peep Show reference.

zaps
The secret ingredient is crime
Exoristos
Or, Society for the Promotion of Elfish Welfare (SPEW).
oniony
It's literally the cleverest name ever, and completely appropriate to what it is.
GaggiX
Wow the model architecture seems extremely simple, very cool project.
MahiShafiullah
Thank you! We tried to keep things as simple as possible on the policy side, but definitely there is a lot more room for innovation to go from 81% to 99%, like using temporal information and global structure of the task.
GaggiX
Another limitation here is that the model is regressive, so for example if a task was to pick up one bottle out of two and the demos showed 50/50 of picking up one than the other, the model would output the mean even though it is not meaningful.
MahiShafiullah
Indeed! In fact, I have a project [0] from last year that uses a GPT-style transformer to address that exact issue :) However, it’s hard to go far outside simulations in real home robotics without a good platform, out of which efforts came Dobb-E.

[0] https://mahis.life/bet/

GaggiX
I've also seen the one that uses the diffusion process for planning, I imagine it's even slower, but maybe with a consistency loss something can be done about it.
twoodrow
Hey there! Thank you for sharing; this is really inspiring.

Do you and/or anyone else who contributed have specific plans or desires for continuing to contribute to the home robot space?

MahiShafiullah
Absolutely, I personally think of myself as a home robot person through and through :)

As for continued contribution, right now the plan is to help get the community on-boarded, and just keep building on the platform. As you can tell, the policy class is pretty bare-bones, the in-home navigation system needs work, and we aren’t taking ANY advantage of the recent generative AI boom. All of those need to change if we want a real robot butler :)

But that’s why we open source things, right? What’s a mountain for one team is a feather for the community.

Mizza
Love the initiative. I'll get one once this starts mature.

Seeing things like this makes me think how badly companies like GE, Maytag, etc. all dropped the ball. Rather than innovating on making peoples lives easier - liberation from domestic toil - they just focused on squeezing out every penny by making Chinese crap with a brand name sticker glued, selling worse versions of products invented in the 1950s.

Somebody, please, just make a robot which folds my damn laundry!

byteknight
Fun fact, the washing machine is credited with saving the most human work hours.

Now for the folding!

dingnuts
It occurred to me recently that, at least in the US, the poorest households do not have dishwashers, and to this day consider them to be a luxury.

The same cannot be said for washing machines. Everybody uses a washing machine. If you don't have one, you use a laundromat. Even the poorest do not wash their clothes by hand with a tub and washboard like the olden days.

This comment has no real "point." I just wanted to share what I think is an interesting observation on this topic.

adamjc
We have a dishwasher but rarely use it (only for parties, etc). Loading / unloading is surpringly time-consuming, imho washing by hand is quicker.

Something I'd consider doing if I was single would be to have two smaller dishwashers, and alternate them, and leave the clean ones in the dishwasher...

Mizza
I've looked into this and they do actually make these. Same size as a normal dishwasher, but has two separate compartments of half the size. I wanna just skip the whole process and have my cabinets clean them though.
etrautmann
kinda of the same thing isn't it? this just changes the definition of the cabinet to those two compartments.
MPSimmons
It probably depends on the size of the household as well as the food prep habits. In a dense urban landscape without much kitchen, a dishwasher makes less sense, given the amount of eating out, takeaway, and so on, probably decreasing the amount of dishes you have total as well.
etrautmann
I lived until last month without a dishwasher (as a NYC resident). It never seemed like that big of a deal. When moving, our only hard requirement for the new place was in-unit laundry.
danielbln
Before having a dishwasher: "Oh you don't need a dishwasher, in fact I find washing dishes by hand meditative!"

After: "You will remove my dishwasher from my dead cold hands"

dharmab
My parents had a dishwasher but always said it "didn't get the dishes clean" and used it as a drying rack instead.

I bought a house where the dishwasher was incorrectly installed causing odors to come through the sink. That led me down a rabbit hole of proper dishwasher installation, maintenance, loading and using rinse aid.

MY GOD! My parents were wasting their lives! My dishwasher cleans and dries two sinks worth with three minutes of effort!

JohnFen
> at least in the US, the poorest households do not have dishwashers

Not just the poorest. About half of the places I've lived in the US didn't have a dishwasher, and only about 1/4 of those places were affordable for a low-income person.

In my experience, it largely depends on how recently the place was built/remodeled.

xanderlewis
And various economists (and others) have suggested that the invention of the washing machine was the most important step towards allowing women to enter the workforce.
jokethrowaway
Which society needed so we can we have twice as many people working and pay them half.

I think in an alternate timeline in which the washing machine is not invented, we would still send women to work.

The washing machine labour could have been solved with centralising washing as a society, in the same way we centralised childcaring.

The economic forces and the political propaganda were too strong.

colonCapitalDee
Spare us the conspiracy theories. Washing machines were invented because they're convenient, not because of "political propaganda". And we've already "centralized" clothes washing, it's called a laundromat.
calamari4065
>The washing machine labour could have been solved with centralising washing as a society, in the same way we centralised childcaring.

Are you genuinely unaware of the laundry sweatshops in the early 1900s? Have you never even seen Charlie and the Chocolate Factory?

We've done centralized laundry. It was sweatshop work and was pretty terrible for everyone involved. Today we have laundry pickup services, but it's too expensive for most people. Are you willing to pay $40 for a load of laundry? Because that's what you get without sweatshop labor.

Seriously, read up on early 1900s labor practices. The labor laws we have today are written in blood. A lot of women and children suffered gruesome deaths and disfigurement, lifelong illnesses and disabilities.

totetsu
So the invention of clothing in the first place was the biggest waste of human work hours.
MahiShafiullah
Thanks for the words of encouragement! If I may ask, what is the level of "maturity" you expect before adopting a home robot? Clearly, a robot that can wash and fold laundry is already there, but what are the other necessary/sufficient qualities?
Mizza
I've never seen a robot that can wash and fold laundry! At least not one that I can have in my home.

I guess I need it to be actually useful without supervision - put socks away, or pick up dirty clothes and put them in hamper, or put dirty dishes from the living room next to the sink, tidy the place up a bit. I'm a hacker so I love the open nature of this project, but unless it's actually useful for something, I will have a hard time convincing the other member of this household to have a robot/vase-destroyer/fire-hazard prattling around the home.

Maybe it could have a vacuum cleaner on the base..?

[deleted]
(comment deleted)
asow92
Master has presented Dobb·E with open-source, Dobb·E is free!
[deleted]
(comment deleted)
isawczuk
Is there DIY guide to build similar robot?

$19k - https://hello-robot.com/ is a bit steep.

avery17
It says right there its affordable though!
numpad0
ABB Dual YuMi is 70k, so 14k is relllatively affordable. Robot parts are wicked expensive and bills rack up fast.
MahiShafiullah
1. They have very detailed guides on the hardware here [0], which should tell you a lot of information you may need to know.

2. However, I think a large part of the price tag is really the cost of R&D, and once this gets out of prototype stage and goes into mass production price is bound to come down a lot. Compared to a lot of other robots, $20K is much cheaper. For example, compare Boston Dynamic's robot dog, Spot, which is around $200K iirc.

3. But that's also why we need more projects like Dobb-E on these robots! Without the right "apps" home robots will never catch on properly to get to the point of mass-production.

[0] https://docs.hello-robot.com/0.2/stretch-hardware-guides/doc...

yardie
Do you think this could be adapted to smaller robot like this Pi-based XGO CM4[0]

I know they have different hardware but the end result is quite similar: a mobile robot with an articulated grabber. And seeing your prototype made me think of the robot dog.

[0] https://shop.elecfreaks.com/products/elecfreaks-cm4-xgo-robo...

EwanG
Compute on theirs is an Intel Nuc 8 (probably explains a decent part of the price). Appreciate the link to the more "hobbyist friendly" device you mentioned. I am thinking it might do well as a platform for light dusting around the house and other things to supplement a Robovac.
cchance
how does a 300-600$ PC explain a decent part of a 20000$ robot?
MahiShafiullah
Smaller robots are unfortunately limited in how much force/torque they can apply and how high they can reach, two factors that we found can severely limit the applicability of a home robot. As a result, unfortunately, we don't have much plans for extending to a smaller robot at this moment.

However, all of our designs are open source! So if you or someone else is interested, it could be a fun weekend project to design a "Stick" equivalent data collection tool for the XGO CM4. I would love to see how that turns out.

omeze
this is really awesome. $20k is just too high for a toy, but I'm reading through this to see if there's a way to get this to $2k
zo1
I know everyone here mostly knows this, but in LCoL areas (South Africa for me specifically), that $20K almost buys you an entire house that you can live in in near-perpetuity. Not only that, but I could hire low-skilled labor for 7200 hours for the same cost. Just thought I'd add a bit of 3rd-world perspective to the discussion.
fragmede
Not that it changes your point, but Spot is $75k.

There's a knockoff for $23k though, so hopefully this one for $20 makes competitors available for cheaper.

https://robostore.com/products/unitree-go1-edu-plus-3d-lidar

MahiShafiullah
I'm sorry, should have been clearer. The Spot robot with a manipulator arm (which has been used in some home projects before, see [0]) I believe totals up to about $200K. The base robot is $75K, which is what you linked to. I mentioned that figure because that seemed more relevant.

[0] https://adaptiveskillcoordination.github.io/

[deleted]
(comment deleted)
startupsfail
There is slight optimization. You don’t need 3D printed parts for your reacher grabber data collection stick. Just use $5 iPhone microscope mount to attach the phone securely to your stick.

It’d be nice also to think of some common format that includes depth, calibration, trajectory and doesn’t require nonstandard software on an iPhone. Standard camera app/cinematic format could be one example, I think.

MahiShafiullah
Thank you for the suggestions!

1. I'm ordering a microscope mount right now! We went with 3D printing because we could get a very good coupling between the stick and the robot mount. See this figure for a comparison: https://i.imgur.com/vWopcFB.jpeg

2. Our dataset (and dataset export code) gives the depth as numpy files and trajectory information in a JSON file with calibration and trajectory information! The r3d export format from the Record3D app we used is simply a .zip file with renamed extension so we were ok with using it. Do you have suggestion as to which cinematic format you are thinking?

beiller
I think you will find openexr surprisingly well supported across many platforms.
MahiShafiullah
OpenEXR looks very interesting! Especially if OpenEXR can compress depth data nearly as efficiently as we can compress image data by making everything a video, it would be a game changer. For example, our RGB dataset is 814 MB right now while the RGB-D dataset is 77 GB, even though depth "just" adds one new channel to our data.
startupsfail
Yes, that’s another thing. When Apple is recording RGBD in a cinematic format, it is tiny.
startupsfail
Yes, Record3D is an option. The problem with it, you’ll never get large scale data in the format recorded by a custom app. If you use some standard app and format that is used by general public, there is a bit higher chance. For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app.

In a longer term the stick is likely not needed too. Ideally you’d want a human demonstrating with regular human tool, like pliers to generalize.

This is a particular mount that I’ve used. https://www.amazon.com/Starboosa-Smartphone-Microscope-Unive...

It’s pretty solid, but it doesn’t open completely, so you need to disassemble and reassemble the stick to put it on (which is not that difficult).

MahiShafiullah
> For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app.

I agree with you there. My impression was that the cinematic format is closed, unlike .r3d which is a bunch of open format files in a .zip. Do you know whether Apple has published the specs for the cinematic video format anywhere, or if there are there good libraries for handling it?

saulrh
Also look into the file formats that the robotics community use, stuff like rosbag and MCAP. These are formally more like point clouds than cinematic video captures and they won't integrate well with the video-processing side of things, but if you're dumping depths and trajectories from numpy and shuttling them to other robotics stuff then they might be a better fit than a 3d video format. In particular, if you end up adding anything other than video-like data streams - temperature, internal estimates from SLAM, etc - you'll have a much better time adding it to an mcap or rosbag than you would trying to shoehorn it into a video stream format.
dhon_
Rosbag is just a capture of published Ros topics which can include point clouds, image/video data, imu, pose etc
pydry
There didnt seem to be any bathroom videos. Can it clean a bathroom? If not, what are the impediments to doing that currently?
MahiShafiullah
Water!

Our robot-and-phone setup is not very safe around liquids haha. Hopefully the hardware experts among us can solve that problem sooner than later.

We are also working on learning multi-stage skills, and early results are promising (See section 4.1 on the paper). However, with multiple steps the chances of failure also goes up, so that's something we will have to make more robust :)

jameshart
One of the problems with home-cleaning robots is how to keep the cleaning robot clean.

You don't want the same grabber that's scrubbing the toilet and picking up your dirty laundry to also be picking up your kids' toys, or packing away your groceries, let alone preparing food.

kaibee
Actually, this is why it's such a great business opportunity, see you have to buy a 2nd robot that cleans 1st robot and then a 3rd robot to clean the 2nd and so on.

More seriously, Stretch actually looks pretty straight forward so I could imagine a combo-charging/dock/cleaning station where it can swap tools (ie, grab a vacuum, drop off mop, etc) and get cleaned by the cleaning station. Also once the robot is in a more finalized version, its not unthinkable to make one that is waterproof enough to go through a carwash type device.

MahiShafiullah
Tool usage is one of the next ideas we would like to explore; because, if you think about it, the reacher-grabber that is currently used as an end-effector is just another tool.

If we can switch between tools quickly we can possibly unlock a wide range of applications: imagine a robot that can switch arms from a whisk to a spatula to a duster to to a mop all over the course of preparing a meal, and then switch to a screwdriver or a wrench for fixing something afterwards. Will take work to design modular parts like that, but the end result can be quite exciting indeed.

sheepscreek
Fantastic! Great to see a robotics framework that is no-frills (easily reproducible/DIY) and is immediately useful. Thank you for open-sourcing it.
cyberax
Can you train it to attack home invaders? Ideally chanting: "Exterminate! Exterminate!"
MahiShafiullah
Not in our plans anytime soon, sorry :) We're focusing on home robots right now instead of rather unsettling space mutants.
[deleted]
(comment deleted)
sklargh
I would give up my second car for a robot that could reliable transit a 120 year old house to do dishes and wash and fold laundry.
fudged71
This is brilliant.

Are considering crowdsourcing the data from more people? It looks fun, and could save me time in the future, so I think this could be worth doing

MahiShafiullah
Thank you!

Yes, we have considered launching an open source effort to scale up the data, perhaps to be comparable to what Google has collected through paid tele-operators [0]. What do you think would be the right incentive structure for everyday volunteers to participate in this effort?

[0] https://everydayrobots.com/thinking/rt-1-robotics-transforme...

pbronez
Privacy would be a big part of it. Video of my home could potentially contain lots of private information. How is that risk minimized?
MahiShafiullah
We don't want any private information, so currently we are manually going over every new demo to make sure private data (face, hands, any other identifying information) doesn't get included in the dataset. Pretty hard to "blitzscale" this way, but I personally think doing it right is more important.
rexreed
Looks awesome. The Hello Robot shown is fairly pricy. (From the site: "A Fully Integrated Mobile Manipulator for $19,950").

What are the options to do similar manipulations for lower cost on a robotic unit?

MahiShafiullah
I discuss this a little bit on another thread: (https://news.ycombinator.com/item?id=38461559)

But the tl;dr is that building prototypes is pricey, and the cost should definitely come down once the community moves from prototyping to mass-producing robots.

Hello Robots (the company behind the Stretch robot we're using) is also trying to bootstrap and build a sustainable product rather than blitzscaling and burning out fast [0, 1, 2] for which I respect them a lot. It's all too common of a story in robot world where have a great company showing lots of promise and then a year or two later they shut down after burning through investor money. I don't want to see it repeat.

[0] https://techcrunch.com/2018/07/25/mayfield-robotics-ceases-p...

[1] https://www.theverge.com/2023/2/24/23613214/everyday-robots-...

[2] https://www.theverge.com/2019/6/19/18682780/jibo-death-serve...

ranting-moth
I'm no marketing expert. But I'd think the majority of those who have an extra $20k to splash on a home robot are not the type of people who want to buy a prototype for hacking.
fragmede
Maybe not the majority, but I think that "early engineers at every Internet startup ever who made it big and want to tinker with a house robot" might be a big enough market to sell a couple of these things. These don't need to be in everyone last person's hands, like smartphones are, in order to be a success.

Plus, if you want to use the stereotype that rich people who can drop $20k on random crap don't do things for themselves, why wouldn't the rich person's tech team buy one on their employers dime?

MichaelRazum
Same thought. For 20k$ you can get a car. What you get here, at least looks like, a roomba with an arm gripper.

I understand why it is so expensive, since it is not a mass product. Or in other words, it should be 10x cheaper, if it would be a mass product.

MahiShafiullah
I am glad you mentioned automobiles! They are among the most heavily engineered everyday product that we use every day, and this engineering and cost optimization has been going on for close to a hundred years now. So, I would read this same statement as "$20k and 100 years of engineering gets us a lot" rather than "$20k gets us a lot". I am hoping better engineering and the wonders of modern supply chains will get us a home robot at 1/10th of the current price when the time is right.
cchance
Why does it feel like it's moving in such jerky steps?
MahiShafiullah
Great observation!

Right now we rely on the out-of-the-box Hello Robots controller [0] implementing position control. Since our policy only predicts a single position into the future at a time, the motion ends up being either jerky and fast, or smooth and slow. Improving on the controller is probably one of the highest ROI improvements we can make moving into the future.

[0] https://github.com/hello-robot/stretch_body

snihalani
good stuff Mahi. please commercialize this and visit us in Bay Area!
MahiShafiullah
Thank you Suren! It will get there when it gets there and so will I :) I do miss you folks, though.
praveen9920
Can it handle any weights on its own? Carrying stuff is definitely one important task that could be useful for lot of people.
MahiShafiullah
The robot can carry small weights on its own, yes, although it's way more stable when the arm is retracted closer to the body.

Pick-and-place style problems are one of the earliest tried on our robot platform, Hello Stretch, which is why this project doesn't spend too much time on it. You may enjoy something like the OVMM project [0] which focuses entirely on open-vocabulary pick-and-place problems: "Pick up $A from $B and put it in $C".

[0] https://ovmm.github.io/

owenpalmer
It's like Dobby from Harry Potter!
advael
Truly an awful choice of name
owenpalmer
Robot slavery
advael
A slavery justified in the narrative by being a kind of sapient creature that simply naturally prefers slavery. And named after the one who wanted freedom due to, supposedly, being a weird exception to boot
kevinmershon
to be fair, that's what robot means etymologically
xsmasher
Wait until you hear where "robot" comes from.
onionisafruit
I thought it was a peep show reference based on all the POV training videos.

Then again I read Harry Potter to my kids about 20 years ago and don’t remember much from it.

filipezf
I was thinking on buying/building one robot like this to experiment. Can one fit more than one manipulator on it (so it could more easily do things like hang out clothes, or hold a glass of water and add ice)? More generally, how does the AI model would adapt to diverse robot morphologies?
crazzygamerr
I can think of two ways: 1. The model has to be built accounting for the extra appendages and input. This isn't very extensible. Each new configuration would have to start from scratch. 2. There are two types of models - first one for controlling the part and the second one orchestrating all of the first types. They would have to 'talk' to each other to be coordinated.
KerrAvon
the linked website seems completely unusable for me in Safari
MahiShafiullah
I am sorry, what problems are you facing? I tested it with Safari on both my phone and laptop just now and couldn't find an issue, so I would really appreciate it if you can reply here or (more appreciated) email me with images etc. on the email listed on my profile.

I much appreciate your heads up, and thank you for taking your time on this.

lukevp
It’s so hilariously slow and jumpy, it moves like a cautious chipmunk! I’m sure it’s gonna be awesome but it is so MVP right now. Also those “houses” in the demo videos are like a grid. Whose house is perfectly clean all the time? My roomba has to adapt to lots of novel situations just to transit the floor.
[deleted]
(comment deleted)
[deleted]
(comment deleted)
ReptileMan
I said back in the day that the real killer app for AI will be Rosie from the Jetsons. A robot that can clean, tidy, dust fold cloths etc 24/7 and free people completely from house chores will have absurdly fast pick up and market penetration. Good luck.