Launch HN: Azalea Robotics (YC S24) – Baggage-handling robots for airports
Unedited autonomous ops: https://www.youtube.com/watch?v=DuJ3ZORnO1o
Teleoperated (sped up, so no sound): https://www.youtube.com/watch?v=LeK8NQLnYgA
The marketing version: https://www.youtube.com/watch?v=k0SDPm09U6s
Robotics is in an interesting place right now, with many warehouse automation companies humming along for almost a decade, and a lot of new effort going to full general purpose hardware with humanoids and software via generalist robotics foundation models. We love these efforts (David used to work on one at Google X with Everyday Robots), but we also see a lot of utility in the current wave of robotics planning and perception tech that can enable new use cases today.
Airlines in the US compete primarily on efficiency and customer loyalty, and baggage handling hits both (John B. has first-hand experience from working on baggage optimization projects at United Airlines). 2% of flights are delayed by baggage errors, leading to downstream network delays. Baggage handling is also a major complaint in customer experience—almost everyone has a horror story of a missing bag, and sometimes people vow never to fly an airline again for losing their belongings. Furthermore, it’s a really dangerous job for employees from a repetitive stress standpoint. EU regulation is coming to reflect this, protecting workers with a maximum number of bags transferred per shift to alleviate back and tendon injuries that are inherent to this job.
Unfortunately for airlines, passengers don’t package their luggage in nicely uniform cardboard boxes. If they did, then the airlines could benefit directly from the recent takeoff in manipulator tech for warehouses. But airline luggage is way more wacky and irregular. If robots are going to handle it, they need to reason about how to grasp each item, handle its deformability, stack it in a stable way, and do all of this quickly, safely, and reliably.
This is what we’re tackling at Azalea. We’re bringing our expertise in deformable object manipulation, perception, robot learning, and planning, to this logistical problem.
We have a few strong bets behind what we’re working on: (1) The hardware to solve this problem has been available or manufacturable for decades, what’s been missing is perception, planning, and control. (2) Cobots, robots designed to operate alongside humans, aren’t enough for safety. To do this task efficiently, you need to move up to 50 kg bags very quickly, which can be dangerous no matter how well the cobots are designed. Light curtains (arrays of lasers that stop a machine when interrupted) and machine cages are the current industrial standard and remain the way to go. (3) Software for generalist robots needs more data than most people today believe, and it will be at least 15 years before deployment: we should focus on specialized problems of economic value.
Our core technical developments are in a few areas:
- Grasp synthesis and selection: From visual data only, how can we identify good candidate grasp points and rank them? For this, we use a mix of physical reasoning, heuristics, and a lot of learning from previous data, combined in a single objective function. Furthermore, success must be evaluated as both a successful grasp and continual hold throughout the transfer.
- Placement planning: How do we lay out luggage in the module we’re loading? There’s a nice ramp-up in difficulty for this problem, from open-loop “divide the world into a grid” app...
82 comments
[ 2.0 ms ] story [ 156 ms ] threadIn deployment these things are probably going to be on some kind of cart system, I presume all your algorithms can handle small changes in the XY travel plane (i.e. the robots location w.r.t the end of the belt).
There are a few ways we plan to deploy (some fixed rails, some mobile). Since the carts we're loading aren't placed with much precision, even the fixed deployments need to do serious environmental perception / localization.
That'll probably be a big comfort to those who lose their jobs
I'm not saying this isn't back breaking work, I'm just saying that your logic doesn't make sense.
https://youtube.com/shorts/KMlCCyQPfME?si=BsyZ9QZgZDJWVh4f
That said, the arm itself can move 180deg/s in every joint (roughly 5m/s max at the end effector) - these videos are still very much v1 and we're looking forward to leveraging more of the mechanical capability with a better gripper, better perception, and some new planning techniques we're rolling out in the next few months.
I'm curious if there's any roadmap eventually to get this out to the ramp itself. Most of the back injuries seemed to happen in the bin itself because you have to often hunch/be on your knees in a bin tossing 50+ pound bags. I know airlines would probably be very hesitant to have any new equipment around their planes but just curious if there's any discussion around that.
Other side note, I also used to work in Cargo and always thought there could be way more efficient ways of loading loose packages that are on every flight and this seems to be a great possibility for those as well.
Awesome work & will keep tabs on it!
John B was obviously aware from previous experience what a manual and injury prone process this was, but I've also been really surprised as I've dived deeper into airport operations myself.
Bagroom is definitely what we're targeting first - being indoors (usually) is a huge plus, and lets us focus on the manipulation part of the problem without going fully mobile yet.
That said, we're definitely targeting tarmac/ramp operations, particularly between a TUG/PowerStow and narrow-body bag carts. Inside the bin is much trickier but we agree it's the least ergonomic part of the job, you just can't move a massive industrial arm in and out of a plane very easily. We have it on our longer-term roadmap, though, and intend to leverage the baggage dynamics data we collect everywhere else to give us a head start on the packing and manipulation problems there, just with a different mechanism.
Cargo packing is a huge area of interest for us! Particularly around optimizing weight distribution in loaded planes, or just optimizing packing efficiency in general.
Each plane is different, so the racks either waste space or have to have tons of different ones.
Many planes don't have easy big doors for racks (the ones you've probably seen are for designated cargo jets - like this: https://www.aircargo.ups.com/media/Containers-Pallets/upsair... )
And now even if you solve all the above, you still have to load the bags into the rack.
That's a big hurdle for an airline to start climbing now to someday maybe start saving money on bag loading in many years.
The planes that DO have them are almost always military planes not used for other things, or very large jumbos (search 747 cargo or C-5 galaxy).
Almost all other planes are optimized for passengers (the cargo space in even a largish jet may be too short to stand up in, for example).
Are you considering dynamic trajectory constraints in the planner (e.g. for multiple robots loading simultaneously)? That was a thorny problem back when I worked on arms.
Suction works really well but it's not enough, we're rolling out a new gripper soon that covers more cases mechanically (there's a very long tail in the distribution of what comes through airports).
Suction is great though, and ~75% of bags checked through the US are hardshells, so it's something we're not ready to ignore entirely.
There have been some neat attempts with short conveyor belts as end effector tools [1]! Generally these systems rely on being able to rearchitect a significant amount of the process (building a controllable conveyor belt or rearchitecting part of the bag-room), and we're focused on dropping into existing processes.
[1] https://www.youtube.com/watch?v=n2Wy_tduq5k
These videos are more to set a scene for where we're operating the the general process.
Irregular items are a problem. In the baggage handling video, the last bag, the soft one with straps, is sticking out after being placed in the baggage container. That's the same problem which keeps Amazon from totally automating picking. They keep trying, but nothing works well enough yet.
[1] https://www.youtube.com/watch?v=TN-6QaLd3VY
We think that due to irregularity that it's not an easy tech transfer from the existing logistics world into the aviation world. We're very interesting in looking the other direction, though!
Miss the days when my parents took a kitchen sink from Dubai to India, or when my uncle took coconut saplings and a literal banana tree to the US lol.
- Seemed like the baggage handler was required to do a very fast cycle of <scan, toss, align, repeat next bag>. Automation seems helpful, and certainly it’s hard labor.
- This was also a young woman, in presumably a safe union job, working in a very pricey city (one of the mountain west towns that exploded). Adios union job hello robots.
Tricky ethics! Outside of picking stuff up and putting stuff down, not too many automation union-safe jobs left. Saving them from back pain is also going to be saving them from a job.
Generally, regulators seem to be moving in this direction as well. The EU has introduced new regulations on the total amount of weight someone can move in a shift, and the Dutch government has mandated that baggage handling move away from manual processes like this in the near future.
The regulator focus seems like it'd reduce the max allowed weight of a checked bag, not automate the baggage handler handing the checked bag. So, I don't see the similarity between the regulatory push and your product? Edit - to clarify, beyond what Dutch regulators say about Dutch markets, which are a very small subset of "regulator focus" internationally.
In teleoperated mode, I'm guessing you're using the captured data to train the autonomous mode?
Final note, the robot looks beefy enough to lift an entire airplane, forget luggage -- is it overengineered on purpose?
The teleop data is really useful for training data indeed, and lets us collect data on current failure points (e.g. with suction, just how far can we tilt this fabric bag before it peels away, etc). We're not going full behavior-cloned end-to-end for a lot of reasons (sample complexity, safety, adaptability, etc), but we do a lot of learning in specific parts of the system (particularly around grasping and placement).
The robot is indeed beefy, as many robots rated for 50kg applications are (check them out online). We've accidentally stress tested this unit way beyond 50kg without a hiccup, so we're very interested in figuring out what the right-size unit is for our application. There are a few other great aspects to this unit - it's a 7-DOF arm + 1 more DOF for the linear rail, so we have two extra degrees of freedom to play with for collision avoidance during planning.
Multi-finger grippers rely on "affordances" i.e. space between the boxes to get the fingers in. If the boxes are pushed up against one another, you've got to do something to make space for the fingers first. So to grab a box with a multi-finger gripper you need space at the top, left and right.
Suction cups, on the other hand? They only need access to the top. No need for space at the left and right, no need to slide things to make space, just apply the suction cup and pull upwards.
Suction cups can also be made of soft, flexible rubber so if the planning is a bit inaccurate? No problem, the suction cup just bends. A multi-finger gripper, though? If the fingers are rigid and strong enough to lift a 30kg bag then they're also probably rigid and strong enough to punch a hole in an adjacent bag.
Suction cups do have a bunch of disadvantages, of course. Porous materials you can't get a seal on. Vacuuming up detritus and messing up your vacuum generator. The payload swinging about on the flexible suction cup. Losing vacuum and dropping the load if there's a power cut.
But they're certainly a great starting point in an application like this one.
We've since introduced a message-bus layer that makes it possible to do it all over the internet etc, but adds the associated serialization and transport latency.
I wrote this blog post on that topic a while back after having seen various approaches robotics companies take and their shortcomings: https://transitiverobotics.com/blog/streaming-video-from-rob...
I imagine most of this code being reinvented on a daily basis at countless companies around the world, what a waste of human resources.
Open source works great for shared code that isn't part of the "value added" by a company. So for a modern robotics company, it makes a lot more sense to use Linux for instance rather than rolling their own proprietary OS. And to use an open-source compiler for building the code. They're in the business of providing solutions using robotics, not selling operating systems and compilers, just like countless other companies build their products on top of these infrastructural tools, and sometimes contribute bug fixes and improvements back. But the code that actually makes the robot work (vision, motion planning, etc.) is what they spent most of their funding building, so giving it away makes no business sense.
Basically, you're complaining about all companies having trade secrets, and ultimately, you're complaining that competition exists instead of just having a single company having a monopoly over a whole market.
Curious if you guys have put any thought into seeing if there's an operational change you could introduce to airlines, that would result in the tech side being a lot easier?
Palletizing logistics for consumer airtravel would be interesting...
For widebody planes, bags are already loaded into Unit Load Devices (ULDs), which are large semi-truncated boxes that get loaded directly onto aircraft. Narrow body planes don't use these (apparently) because they impact turn-time and decrease the amount of time a plane can be in the air, and also impacts how quickly bags come out, since it adds an extra step to unloading.
Many airport conveyance systems also load each bag into a bin, but those bins aren't loaded into the airplane because they belong to the airport and waste space/weight.
The best case for us would be a customer process change where everyone loads their luggage into perfectly regular and very sturdy hardshells, but this one's probably out of our hands.
I could see a budget airline cooperating with a luggage/case manufacturer and offering "if your checked bag is EXACTLY a Pelican 1615TRVL, it flies free/cheap" - and then work with them to design a case that is easily automatable.
Also --- I couldn't see any obvious sensors. What sensors are you using to perceive the bag? I am imagining some kind of RGB-D sensor like a Kinect (or its successors like the Orbbec).
For these videos we have lidars and two Intel Realsense depth cameras mounted to the safety cage and on a wall. We're working on moving as many sensor on-robot as possible in the near future to aid with deployability.
Going to be a pest for a minute - have you considered a stack other than ROS2?
Unfortunately this breaks down in a few ways that you're probably familiar with, given that you asked this question:
- A crash in (e.g.) a third party sensor driver can bring down your whole binary, any signal catching here is awkward and you end up wanting process isolation
- Perception is, for better, or worse, easiest to prototype and try off the shelf in Python / Pytorch, so you either end up with pybind11 and driving things in Python, ONNX which is IME brittle for some of the crazier Pytorch modules, or message serialization and process isolation.
ROS/ROS2 does _way_ too much in my opinion - why does it have a build system and a ton of packages? This plus pinning OS versions are huge pain points. Unfortunately I also think many community-contributed ROS/2 packages are fairly low code-quality, with notable exceptions. Overall, I'd prefer to have ROS be a pubsub library with a few extra tools for logging and visualization.
In the end, we're currently using ROS2 for the reasons listed above and for easy prototyping, but I'd like to move to something more like FPrime, Basis, Cerulion, or Copper in the near future. I really want to grow something in-house with Zenoh or IceOryx2, but don't want to waste a lot of time on middleware, since I don't think it's what's kept the problem from being solved.
(At the end of this post I now see you're working on Basis, I apologize that I'm over-explaining to you. I'd love to chat about Basis if you have some time in the next few days!)
Would love to be able to provide your middleware, we've connected on LinkedIn, let's chat.
Say golf club hard shells case where the owner's left a bunch of golf balls in the bottom of one of the pockets. It's difficult to tell that one end's quite a lot heavier then the other until you actually try to pick it up.
At least to me I don't see a good way of handling this edge case. You'd need either a second manipulator, a very fast feedback system, or a way for the weird luggage to be diverted for human intervention.
- Golf clubs specifically (and most sporting equipment) actually goes down a different pipeline in most airports, since generally this stuff doesn't behave well on conveyor belts.
- Data driven approaches can tell you a lot just from visual information, usually about deformability of objects, but also about expected centers of mass, etc
- Part of the reason we're using single big robots is because you can use heavy duty end-effectors - grasp all the way around the object with a grasp that's predicted to be robust to these kinds of perturbances, and then use quick feedback to safely execute a plan to place it
You're absolutely correct though, that there's a long tail of things coming through and that some objects are very, very difficult. Our problem formulation then becomes identifying confidence in graspability, and deciding explicitly that we shouldn't attempt to grasp some object and should instead flag them for human handling.
- How much are you able to use the robot's internals to estimate the gripped bag's inertial properties? If you're trying to put rigid, heavy bags below light, amorphous bags, are you adjusting final placement location on-the-fly?
- How dynamic is your scene beyond what we can observe? If you're using light curtains with a single robot on a track, and you're able to estimate some rough geometry of and track the bags down the conveyance, and you are updating occupancy of the bin as you're going, what else is there?
- Is this just inside for the foreseeable future or are you all going to tackle unpacking outside, as well as all the, ahem, baggage that comes along with operating outside the terminal walls.
Nice and straightforward problem, relatively speaking.