Building a robotics research setup that lives next to my desk (dfdxlabs.com)

[dead]

Great article. I'll be following along. Would like to learn more about the robotics space.

- I've heard the advantage of ROS besides the architecture is the ecosystem (driver integrations, etc). Is that not an issue because the arm supports a Python SDK OOTB?

- Any issues you've been running into with this setup?

- How do you determine if a session recording is good enough for training? Is 50/100 samples really all you need?

Hey this is cool! I am doing something similar myself with the SO101 arm robot from Robot Studio using a patchwork of my own code and LeRobot. Would love to collaborate with you if you are open to it. You can find me on Discord as `.avilay`. https://www.linkedin.com/posts/avilay_lerobot-huggingface-ro...

Would like to know your reasoning on not going with LeRobot.

Cool stuff. At my previous (startup / research) job I had to set up similar system (but with franka arm and multi view camera) alone because I was the only one with robotics background.

>I do not intend to calibrate the camera’s extrinsics or intrinsics for now.

Sensible choice, although I suggest it's good in the long run to do at early stage in your setup, especially if you intend to collect data for policy learning.

Debugging trained policies for visual manipulation task can be a headache and having as much context on variables that can change is a good practice.

My previous setup was in Japan, a earthquake prone place and I wasted some time after realizing the camera got misaligned due to earthquake. A simple solution is just to place an Aruco marker on the table that tracks the relative extrinsic position of camera, and add it as metadata to collected teleoperation dataset.

This is really exciting. Incredible that you can do this for this budget at home. Unthinkable a couple years ago.

As impressive as this setup may be, I'm still amazed at how slow this type of robot is, whether amateur or professional grade. I have no expertise in this field, but as an observer, the apparent progress in this area seems very limited. I guess my expectations are too high and my understanding of the problems to solve is too low.

How does Lerobot prevent « full control » and « understanding »? I thought this was an open source library.

I am not an official supporter of the library but am asking out of curiosity.

ROS sucks, good move. Too complicated

Love this, I’m playing around with the cheapo esp32+servos version of this, super fun.

Something I’m working on is a hardware CLI for agents to run experiments, with a “CICD” pipeline that validates everything and means I can delegate more of the experiments to the agents. I wonder if you have any thoughts on this?

The idea is to allow the coding agent to run the full loop of experiments and validations, with vision, audio, button pressing, speaking etc to interact in place of the human

Very interesting article. I'm looking to get my first robot arm soon-ish, probably something in the SO-101 category. Can someone get reasonably far using recorded sessions compared to training in a simulated environment¹? Do you have any experience with third-party or DIY attachments for robot arms? I assume it's going to be more difficult for something like the Ufactory arm vs. the open models.

¹https://blog.comma.ai/mlsim/

Ah nice I've tinkered a bit with robotics few years ago, e.g. https://twitter-archive.benetou.fr/utopiah/status/1760260544... and it's pretty empowering... but also showcases how amazing the human body is. Our dexterity, ability to sense, etc.

Reminds me of https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... which is basically a stark warning against the hype.

Nice, I will be following your posts! I just bought a robot arm myself, the seeed studio B601DM (€1500 6+1 axis), it works great and is open source hardware as well and a bit more solid then the so101. I also opted to not use ros, I don't want to give up control by putting another framework in between. Is your plan to see whats possible right now or do you also have ideas on how to improve sota?

I had a very similar setup. Really happy with the xarm 6 lite. I played around with the diffusion policy paper experiments and was thinking to buy a webcam as a top camera as well but I ended up buying two intel realsense ones because of the timestamp drift issues. How did you solve that? Or is camera feed syncing not necessary for your intended projects?

Good move getting a nice robot! I'm doing something similar to you, but I went with a cheap robot, the "HIWONDER 6DOF Robotic Arm Kit". It was only $600, but wow is it bad. The precision and repeatability are both "are you drunk?" level. I can hear the gears grind when it moves. I suppose I should upgrade. But if my system can work with a terrible robot, I assume it would work even better with a nice one!

My project is https://github.com/colinator/Ariel - basically, no VLAs - instead, "just write code". Or have the agents do it.

I don't have a writeup yet about applying Ariel to _this_ robot, but this is for a previous one: https://colinator.github.io/Ariel/post1.html.

Excited to follow your progress!

Fascinating article. Keep up the work, Matthias!

wow good post. Thanks for explaining everything in dam simple words .

I hate whinging but why isn't stuff like this not moved higher on HN's front page? This is a great article yet I keep seeing world politics and other matters rated higher - stuff that (unlike this article) will age like milk.

> And one I'm more confident about but expect disagreement on: not building on ROS 2 / LeRobot,

Tell me more! I am slightly biased in that direction. But can’t fully justify it at this point.

This is so cool.

VR controllers are significantly better than a SpaceMouse for teleoperation. Even if I already had a SpaceMouse I would buy a Quest 3s and switch to it, if I was doing more than a trivial amount of teleoperation. (As is typical with robot teleop setups, I would not wear the headset on my head. It's merely a dongle to track the controllers).

- A single arm is sufficient for validating basic Pick/Place tasks, but more complex scenarios require Bi-arm

- Calibration is not required for VLA models.

- RGB or Stereo RGB inputs are sufficient for ACT, DP, and PI0/PI05.

- ROS2 is not strictly required, but it can be useful for sharing/co-developing codes. For instance, the Stanford team built a custom framework for diffusion policy instead. I also developed similar framework because ROS2 is not optimized for bi-manual manipulation or VLA workloads.

This is so cool and looks so fun. I want to play with robots. I just play with software :(

I wanted to build something making bows automatically and damn, pretty complex and expensive to buy the parts.

I was like, I can probably just buy a 3D printer, print all the parts and buy some motors, but it seems it's way more complicated than that.

Like to play with a single hand robot. It looks like you need 10k+$. I wanted to spend max 1k, 3D print parts, buy motorized parts on Alibaba, and code on my Mac + spare GPUs I have access to. I'll have to save a little :(

Really cool. Good move on writing your own stack. I was weighing using ros2 or building our own, but ended up going the ros2 route to “save time”. We are working with autonomous mobility robots in nature, so we figured the ecosystem around ros2 would be worth the compromise. It was not. All the time we saved setting things up initially, we are paying for now.

There should exist a minimal, clear, robotics library like what you’ve built. The Flask of the robotics world.