Cool project! That validation loss curve screams train set memorization without generalization ability.
Too little train data, and/or data of insufficient quality. Maybe let the robot run autonomously with an (expensive) VLM operating it to bootstrap a larger train dataset without needing to annotate it yourself.
Or maybe the problem itself is poorly specified, or intractable with your chosen network architecture. But if you see that a vision llm can pilot the bot, at least you know you have a fighting chance.
Check out using maybe some kind of monocular depth estimation models, like Apple's Depth Pro (https://github.com/apple/ml-depth-pro) and use the depth map to predict a path?
I don't really see how the vacuum can effectively clean a whole room or flat using only a CNN of the current image in front of the robot. This would help detect obstacles, but a bumper sensor would do that as well.
All but the most basic vacuum robots map their work area and devise plans how to clean them systematically. The others just bump into obstacles, rotate a random amount and continue forward.
Don't get me wrong, I love this project and the idea to build it yourself. I just feel like that (huge) part is missing in the article?
Apart from just detecting obstacles, we wanted to build a robot which is intelligent enough to take in semantic cues like this is a doorway so I can go through it, or this is a kitchen I can clean it this way and so on
Thanks! Our main goal was to build a vacuum which understands semantics inside the house so that it can "clean the kitchen" or "clean the bedroom" so we wanted to do machine learning and since we were doing machine learning we were like why not try to do something E2E instead of first doing SLAM, optical flow etc..
If mass produced, no part of a robot vacuum is expensive. Blower fans are ~$1. Camera is $1. Cheap wifi MCU with a little ML accelerator + 8 Mbytes of ram is $1. Gyro is $1. Drive motors+gearboxes together are $1. AC charger $2. Plastic case $2. Batteries are the most expensive bit (~$3), but you can afford to have a battery life of just 10 mins if you can return to base frequently.
The hard part is the engineering hours to make it all work well. But you can get repaid those as long as you can sell 100 Million units to every nation in the world.
Yeah agreed 100%, might also need to factor in the cost of the charging dock but the overall thesis is still sound.
Do you know any cheap wifi MCU with a little ML accelerator that we can buy off the shelf? The only one we could think of was the Jetson Orin Nano and thats not cheap
Here is thought, this is a fixed 3d environment and you lack training data or at least an algorithm to train. Why not use RL to learn good trajectories?
Like build a 3d environment of your home/room and generate images and trajectories in a game engine to generate image data to pretrain/train it, then for each run hand label only promising trajectories i.e. where the robot actually did better cleaning. That might make it a good RL exercise. You could also place some physical flags in the room that when the camera gets close enough it gets rewarded to automate these trajectory rewards.
Wow okay there is a lot here, just so that I understand this correctly:
1. Make a replica of my home/ room in a game engine or a simulator
2. Generate trajectories with RL where the reward is hand specified by me
3. Automate trajectory rewards using some proximity flags
Some stupid questions:
1. How do I build a replica of my home? Is there an SFM algorithm I could use to do this just from camera images?
2. Would this still work even if things/ furniture move around the house?
3. This data collection strategy will have a distribution shift compared to real data so it might struggle with different lighting conditions and stuff?
Caveat here is you may not be able to use their environments or you may or may not have their kind or robots to train your roomba. But at-least you could get an idea of how RL training is done for robots like yours.
Can you please design a version for kids to ride on?
With a seat and handle similar to "wooden bee ride on" by b. toys?
I want a vacuum that kids can actually drive, ride on, do real vacuuming and has minimal levels so safety, like turning it over halts vacuums, stairs/ledges are avoided, and lack of rollers or items that could snare a kids hair, etc.
There may be benefits of fusion of child input signals with supervisory vacuums route goals. Would be age dependent, older kids would want full manual I think.
Kids like to do real jobs, and as a parent I prefer purchasing real items for my kids rather than toy versions if practical.
It may or may not be useful for you but I've been working for a while on converting ORBSLAM3 into a self contained standalone program, without the need for ROS to be useful.
The "UI" for saving/loading the map and calibrating the camera is exposed through a built-in crude webserver. Visualization is done via threejs instead of having a dependency on pangolin.
If your robot can expose the camera feed as anything opencv can ingest ( i.e. mjpeg via http ) you could just point it there and then receive the pose stream via HTTP/SSE
The whole thing is distributed as an AppImage so you just run it and connect to it
25 comments
[ 2.6 ms ] story [ 48.5 ms ] threadToo little train data, and/or data of insufficient quality. Maybe let the robot run autonomously with an (expensive) VLM operating it to bootstrap a larger train dataset without needing to annotate it yourself.
Or maybe the problem itself is poorly specified, or intractable with your chosen network architecture. But if you see that a vision llm can pilot the bot, at least you know you have a fighting chance.
Thats a cool idea, is there any VLM you would suggest? I can think of Gemini maybe? Or any would do?
Very cool project though!
All but the most basic vacuum robots map their work area and devise plans how to clean them systematically. The others just bump into obstacles, rotate a random amount and continue forward.
Don't get me wrong, I love this project and the idea to build it yourself. I just feel like that (huge) part is missing in the article?
(Lidar can of course also be echolocation).
It could easily understand so much about the environment with even a small multimodal model.
The hard part is the engineering hours to make it all work well. But you can get repaid those as long as you can sell 100 Million units to every nation in the world.
Do you know any cheap wifi MCU with a little ML accelerator that we can buy off the shelf? The only one we could think of was the Jetson Orin Nano and thats not cheap
I would begin in one room to practice this.
1. Make a replica of my home/ room in a game engine or a simulator 2. Generate trajectories with RL where the reward is hand specified by me 3. Automate trajectory rewards using some proximity flags
Some stupid questions: 1. How do I build a replica of my home? Is there an SFM algorithm I could use to do this just from camera images? 2. Would this still work even if things/ furniture move around the house? 3. This data collection strategy will have a distribution shift compared to real data so it might struggle with different lighting conditions and stuff?
Caveat here is you may not be able to use their environments or you may or may not have their kind or robots to train your roomba. But at-least you could get an idea of how RL training is done for robots like yours.
With a seat and handle similar to "wooden bee ride on" by b. toys?
I want a vacuum that kids can actually drive, ride on, do real vacuuming and has minimal levels so safety, like turning it over halts vacuums, stairs/ledges are avoided, and lack of rollers or items that could snare a kids hair, etc.
There may be benefits of fusion of child input signals with supervisory vacuums route goals. Would be age dependent, older kids would want full manual I think.
Kids like to do real jobs, and as a parent I prefer purchasing real items for my kids rather than toy versions if practical.
The "UI" for saving/loading the map and calibrating the camera is exposed through a built-in crude webserver. Visualization is done via threejs instead of having a dependency on pangolin.
If your robot can expose the camera feed as anything opencv can ingest ( i.e. mjpeg via http ) you could just point it there and then receive the pose stream via HTTP/SSE
The whole thing is distributed as an AppImage so you just run it and connect to it
https://github.com/mgschwan/ORBSlammer_LocalizationService
I want to work really hard to be too lazy to bother with the grass.