Launch HN: Polymath Robotics (YC S22) – General autonomy for industrial vehicles
I’m Stefan, one of the founders of Polymath Robotics (https://www.polymathrobotics.com) and formerly of Starsky Robotics (YC S16). My co-founder is Ilia Baranov, a career roboticist. We’re building general autonomy software for industrial vehicles.
We’ve just come out of stealth with a full autonomy stack freely available in sim online. Whether it’s a tractor on a farm or a 400t dump truck in a mine, we make it easy for you to make it driverless. Where before you would need an industrial vehicle, $25-150k in sensors + compute, and 6-18 months of building time, now you can start building applications that command autonomous robots right away.
See a demo here: https://www.loom.com/share/852d54744d45452d926b9016f61f5e43.
Two big things are hard about automating industrial vehicles: first, robotics is deceptively hard. But also, industry is nitpicky. These two together are too much for most startups. That is, if you put all your energy into the first—the noble effort of getting the robots to actually work— the second will end up killing you. We learned this the hard way at Starsky, and what we’re currently working on at Polymath is a way out of this dilemma.
At Starsky, we were working on autonomous trucks. We knew that was a hard product to build, but I didn’t really have perspective until I went and hung out with SaaS companies afterward. My SaaS friends use horizontal services at every layer of the stack. Payments? Just plug in Stripe. Messaging? Twilio. Libraries and tools exist for everything. Your thing goes a lot faster when you can just buy the rest of the stack.
Contrast that with Starsky, where we were building our own robotics stack completely from scratch. That meant the autonomy itself (our actual product, and really complicated on its own), but also custom teleop, a drive-by-wire system, hardware abstraction layer, and fleet management system to orchestrate all the trucks via an API. Each of those things could be a company in itself! Imagine trying to debug a stack like that when something’s not working.
That’s sort of just how robotics is today. Like software in the 80s, everyone’s rebuilding the whole stack for each new project. So the tech gets super complex, brittle, and unadaptable. In fact, an open secret in the robotics world is that most demo videos you see are “cooked,” because the robots are so unreliable in the field.
I got to know many of the folks building industrial autonomy vehicles from 2016 onwards, and saw a depressing pattern. Brilliant, mission-driven engineers would get a start with a healthy seed round from investors who thought it was “just” an execution problem. After 18-30 months the teams would have a prototype that somewhat worked, but what they wouldn’t have yet was the commercial traction needed to justify more fundraising.
That brings us to the other half of the dilemma: industrial customers are nitpicky. When it comes to scaling an autonomous vehicle POC (proof of concept) with a large industrial company, they’re surprisingly unimpressed with autonomy. Sure - the first time they see a driverless vehicle their jaws drop and they ask about safety. But like everyone else, they quickly assume it “just works” and start asking whether it will integrate with their Oracle instance. Then they start nitpicking the specific way the tractor drives across the field / the luggage tow stops on the apron / the dump truck pulls up to the processor / etc etc.
In other words, after the initial POC, industrial autonomy stops being sexy-super-hard-technical and starts looking like Enterprise SaaS. Except by then the team has spent 2 years heads down making autonomy (the hard part!) actually work rather than talking to customers. As a result, they’re not up on the top 20 things customers care about and t...
75 comments
[ 3.9 ms ] story [ 125 ms ] threadThe point of Caladan is more for you to be able to build specific autonomy functionality without needing a vehicle, or a large plot of land, or a team to build a basic autonomy stack that works.
Not sure if the question is about the simulator Caladan, or about our autonomy core.
For Caladan, we don't support playback of data currently. Do you have a use-case need for this?
For the autonomy core, the localization, costmap generation, and path planner are all probabilistic to a more or less degree, so we can't guarantee determinism to any highly accurate degree. However, if the same data is played back over and over, the vehicle would react in basically the same way (perhaps not navigating to the exact same spot, but within a small tolerance). Same question here; what is your need/use case to have guaranteed determinism? Is it a safety question? (We can have a much longer chat there)
So to narrow it down a bit more, my question is a bit tied to safety, but also to general development. Given a set of logged inputs from a machine that had a field issue X, am I able to reliably reproduce (hopefully deterministically) what happened on that machine? (and therefore what went wrong)
The approach we are taking to this is two fold: 1) Robust self monitoring and metrics, ensuring we measuring system performance (we are using Prometheus to allow us to scale to (hopefully) millions of devices. 2) Ensuring we tie those metrics to a rolling buffer of the last 90 seconds (or so) of data. When an issue occurs, we automatically save to disk, and allow remote operators to pull the raw sensor data as needed.
With the raw data, we can (and do!) play this back through our stack and step through the behavior tree and control outputs, comparing them to what actually happened on device. The system is certainly deterministic enough to allow this sort of debugging and testing!
We will have a set of articles on this approach in the future!
This could be built, but unsure it'd make sense in terms of required compute on each small robot. What would be the use case you are interested in?
I suppose that trucks serving most open mines work that way: they have to share the common spiral part.
Essentially they have a portion of the mine site designated as the "autonomous pit," which is the only place where autonomous vehicles can operate. Not all vehicles in the autonomous pit are autonomous (basically no one can automate the vast majority of industrial vehicles, which is part of what we want to solve). In order to be able to see these non-autonomous vehicles, this PubCo makes the mine buy a $20-50k transceiver to put on every non-autonomous vehicles (in addition to the $1m auton hw upgrade and $250k/yr in autonomy SaaS).
These autonomous vehicles do have sensing, but they more rely on those transceivers to tell where vehicles they shouldn't hit are.
So ya, sure, it kindof works like how you think it should. But in a worse way.
More seriously - there are a surprisingly large number of groups out there building robots and looking for help. Some groups already have not-built-here syndrome, but our hope is that by being a free thing you can just initially build on people will be "Polymath Native."
Your sim question is especially poignant. Our sales line is that reconciling sim:reality is our problem to solve for you. The more real answer is that we're specifically focused on vehicles in environments that are "relatively clean" - i.e. controlled (even if, in fact, they're a landfill). We're also only really selling to technical users - who are hopefully sophisticated to solve >0% of the problems.
One thing I have to say looking at your website, is that is has serious "Underpants Gnomes" vibes.
1. Build in sim 2. Test in sim ... 3. Deploy on a real robot
Taking something from simulation and having it work "effortlessly" (as your marketing puts it) is sort of the holy grail of robotics. It's something that has confounded researchers and roboticists for a long time, so I'm a little bit skeptical as to whether your product actually lives up to the promises made in your marketing material. It would help if you showed more demos of what this means in a practical sense. In the video on your website it shows a simulated tractor, and then a tractor moving in an open field with no obstacles. This is pretty much the best case scenario, but how does it actually perform in more realistic scenarios?
Going into your FAQ a little, it seems to suggest the process isn't so seamless:
"Necessary-but-minimal" is kind of understating how difficult sensor calibration and controller tuning is. It's all the difference between a simulated world and the real world (and also necessitates the need for controls and perception experts, which your marketing seems to imply are not needed or minimally necessary with your product). I'm curious how your product makes this process easier.It appears like your value-add is that your product simplifies the whole ROS tech stack to some degree, but to me I would imagine transitioning from sim to reality without much work would imply that you have made real advancements in simulation technology. Is that the case? Or does your product mostly address improvements in the development experience?
You are asking about the FAQs here: https://www.polymathrobotics.com/product
We are actually a hardware and real robots company first! Our simulator efforts are for two purposes: 1. Allow more people to try out our autonomy core, and build on top of our API 2. Allow our own developers to run testing and tuning in sim.
To your point, we don't expect tuning to purely happen in sim. However, we did have our senior controls engineer just recently tune up a controller in sim for Caladan, and later deploy practically the same thing to the real vehicle, leading to much smoother steering commands. We'll write about that in detail in the future.
Our work is to ensure that API commands that are sent in sim, behave as close to identically as possible on real vehicles, thereby allowing people who build on top of us to focus on higher level software stack (business logic layer, etc). Hence, users of our API don't need to have any robotics experience, they simply command the vehicle to do something, and we ensure it gets done. (and our team is comprised of perception, controls, ML, etc engineers)
The details of how this is done will be a future writeup, but the summary is that we pass sensors through a Hardware Abstraction Layer (HAL), along with kinematic/dynamic configuration and data about the vehicle. This allows the global and local planners to plan for a more generic vehicle (ex: Ackermann steering vs differential steering), while the HAL ensures that the planners don't generate unfeasible or unsafe commands.
Let me know if the above answers your questions!
Users of our API can send high level commands (ex: go to GPS coordinate X), and our software (on vehicle) will ensure it gets done.
Our software is built by a team of roboticists, including ML, controls, etc. The software we write ensures that the real world vehicle responds almost identically to the simulated one.
There are of course limitations, hence we do a commissioning step where we ensure the work we've done in simulation for a particular vehicle (sensor locations, localization fusion of sensors, controls tuning due to kinematics/dynamics etc) are tested and tuned on the real vehicle. This is done before the real vehicle is put into service, and remotely monitored (collection metrics on performance, status of sensors/actuators, etc).
Cheers!
There is a perception stack and a controls tuning stack within that SaaS that we'd be delivering.
Sim =/= sim =/= sim. By which I mean - when some people say "build autonomy in sim" they mean lots of different things. Sometimes they mean solve all ML and data collection problems in a simulated world. We don't mean that.
When we say build and test in sim, we really mean that for just the earliest phases. Essentially - if you wanted to build Bear Flag Robotics 2.0 you could take our example app (lightweight Python app to tell a tractor to till a field), meaningfully improve it, show it to farmers to get LOIs, raise money, buy + outfit a tractor, and then put our autonomy on that tractor.
In that end state the behaviors you built out in Sim would still mostly work on the real tractor (same API commands both) and you could end up modifying those behaviors the way you would need to make them work in the real world. We would also be working with you closely to make sure our product is working well, solving new problems, and probably doing things that don't scale to help you be successful.
Side note: I'm always happy to learn a new reference, but most of them are heavily US-centric, and for people like me who didn't grow up in the US, and live or lived in the US in their adult life, sometimes it's a bit too frustrating, because there's just too much of it going around.
The worst of all is when the reference is killed by Italian dubbers/translators (in Italy you can rarely watch TV series or movies in their original language, which is very comfortable, but deeply damages your ability to understand cultural references, which most non-US foreigners would get because they watch the original English version of everything).
Anyway, just to give all of you some new perspective of how these things are perceived by some foreigners.
[0]: https://en.wikipedia.org/wiki/Gnomes_(South_Park)
Step 1: Collect Data, do some analysis, train a model or two Step 2: … Step 3: Profit!
Absolutely, there is a ton of really interesting controls, ML and general software quality work for us to do.
Ya - I can't imagine how hectic it was for you guys. I remember hearing Josh Switkes talk about hand building radars for pre-2010 autonomy.
Re:booms/buckets/etc: right now we're just focusing on mobility (making these systems drive from A to B), specifically because those systems are deceptively hard. FWIW I think the guys at Built Robotics have done some cool stuff on it, as have groups like Moog and FP Innovations.
I also appreciated the write up. Rooting for you! I think FAR more startups should be focusing on robotics and basically anything to do with meeting fundamental needs IRL.
You did a great job with the description. Makes me want to get off the sidelines and build robotics on your API. (I have startup hyperactivity lol)
Unrelated: do you have to be a YC company to do a Launch HN?
I was deeply upset to realize I was too broken to do something that "easy." (still, really hard). As a mentor of mine, Boom's Blake Scholl, has said - it's sometimes easier to do "hard" things because it's easier to get the motivation to work on them.
Re:Launch HN - I think so? You can always do a Show HN, though!
I'm personally working on something hard in SaaS (automatic summary video after every Zoom/Teams SaaS demo).
I think truly hard things are in a lot of ways easier for truly inventive engineers because it's an execution challenge but you basically know for sure there's a market after.
Also hard things can be decomposed. If you solve one part (as you are doing) that's enough. So you can really niche down within something hard and still change the world
Completely agree on decomposing hard things.
- You have a simulator, like idk The Sims world
- You train a machine in this world, totally virtual
- You deploy the trained machine on real hardware in the physical world
- The machine works good enough that it can perform the task until it can't anymore and then someone remotes-in to fix it?
Did I get this right or can anyone ELI5 to me? I mean it sounds really cool, in theory...
We don't train in simulation (if you are talking about ML training). Our ML training is done purely using real world data.
The simulator is to allow developers to "test drive" our autonomy core and API for free.
We have real vehicles as well, but you can imagine it's more expensive, time consuming, and generally slower to let people test on real hardware. Hence, if people like what they see in sim, they can get in touch with us to deploy their code to real machines on a case-by-case basis.
The machine will work good enough for basic autonomy, but then the real application-specific work begins. Whether that work is you modifying the behaviors you're having us do, you routing those systems to humans to help out, or you complaining that our stuff sucks (and us trying to rapidly improve it).
(please let me emphasize this "reptile" autonomy is today perhaps the hardest/time-consuming part killing many startups, as you nicely explain in the OP---Moravec's paradox)
No, we haven't looked into formal tools like that, good suggestion! TBD on if these approaches bode well for real world robotics, (due to model complexity and model/real world fit problems) but at least there seems to be some applications to path planning and controls: https://ieeexplore.ieee.org/document/5509686
I'll pass it on to the controls team!
To unfairly simplify that assertion, it reminds me of a founder who once told me that "robotics will never be as easy as software. It's just harder and will always be so."
I don't know if I think each autonomy company needs to figure out vehicle dynamics any more than I think each company needs to figure out their cloud infrastructure. At a certain scale (and amount of success) you will certainly need smart people thinking about it, but you can get pretty far on a 90% standard AWS instance. Things like wheel slippage are challenging, but their challenging in similar ways across vehicle types (but different from other tasks you need to work on when building that higher level autonomy).
Similarly, if you assume you can safely stop when there's danger, there are a relatively finite number of environmental conditions you should operate in - which we can build, maintain, and improve "once" as opposed to making every mining OEM or Ag autonomy shop build from scratch. In time, there might be certain portions of this stack that you can swap out with your own (or with some other company that's better at seeing in the fog, for example); but I fundamentally don't believe that every robotics company needs to solve all of these hard problems well on their own for every application.
Re:Sim - for paying customers we are putting their specific vehicle (and sometimes their environment) as a part of the offering. We're also using sim to help figure out how to position sensors (which will be another cool thing that we want to show at some point).
They have, after all, been running remote and autonomous 100 tonne dump tracks and kilometre+ long trains for seven years now, in addition to being an umbrella across one of the largest mining fleets across the globe.
Perhaps this means there's no reason for Rio Tinto to use us, and that's totally fine. There are 1000s of other, much smaller mining companies that can benefit from the same tech, likely at a cheaper price than what Rio Tinto spent developing their own version.
Also, from our experience, these sort of approaches bake in assumptions on particular vehicle types/characteristics. So even in Rio's case, perhaps they have other smaller models, or other types of equipment they haven't automated. In this case, we'd be happy to integrate with their existing command and control software stack.
[1] https://www.riotinto.com/en/about/innovation/automation
Some of their talent date back to sheep shearing robots in the early 1980s.
In the same geographic region (W.Australia) there are related automata projects such as self recharging drone clouds about tractors, agri-bots (visual identification + weed spraying), integrated geophysical air survey, etc.
I was somewhat curious as to your awareness of all this (eg: Rio's plans for a full automated "bottom up" $64 billion copper mine in the US (should it go ahead)).
Also, while we're here, do you have any in house mining | industry experience driving tractors, dump trucks, ship loading, etc. yourselves?
To my knowledge, the main players automating ultraclass mining trucks are Caterpillar, Komatsu, Hitachi, and ASI. CAT has been working on autonomy for a long time, and to my knowledge only offers full driver-out autonomy on a handful of models. Sandvik also has some really cool see-and-repeat autonomy in underground mining. It doesn't surprise me if Rio Tinto is talking about a full zero entry mine (worth it just because no people to hit = faster driving vehicles = more production/yr), but to my knowledge they buy their autonomy from others and no one I know of (besides us) would automate all the various types of equipment that go into running a mine.
A different large mining co had an initiative to build their own autonomy, but the project got cancelled when it wasn't making fast enough progress and the CEO was replaced with someone more Caterpillar friendly.
We don't have in house mining experience - but again we're not building autonomy just for mining. We're building generalized basic autonomy so the folks starting mining-specific autonomy projects don't get stuck re-building the basic autonomy wheel.
As with any large opportunity, there are multiple players in the space, both new and old. As seen with the "innovator's dilemma", I'm not sure I'd place my bets on tech straight out of the 1980/90s/2000s (sheep or no sheep ;) )
I think the way you're bounding your addressable problem space -- industrial vehicles operating in a closed environment -- makes a lot of sense. I also agree that certain classes of problems (like wheel slippage) would be better solved "once" by an expert team rather than repeatedly by people who are actually trying to solve a different problem of higher level autonomy as you put it.
Some random other thoughts:
- your business model and offering sound kinda similar to [Tangram Vision](https://www.tangramvision.com/about-tangram-vision), though not close enough to where you would be competitors. I'm not affiliated with Tangram at all, I've never even interacted with them, just sharing what I've come across in my research.
- How do you think about how your customers join and potentially leave your platform? In your reply you mention a certain scale and amount of success where companies may decide to start doing this themselves, or even going to other companies that specialize more than you do (e.g. Applied Intuition for sensor simulation). It might feel a bit early to think about on launch day, I'm just curious about how you think about your company's value prop for customers at different stages of growth. Are you potentially limiting your market to only niche segments and smaller players? Who is your key customer you're targetting?
I appreciate your response and candor, I'm very interested in seeing your company take off!
I’m not familiar with Tangram, but thanks for sharing. I’ll take a look!
There is definitely a world where customers “graduate out” of using Polymath, people do the same with Stripe (I know Uber has been at that decision point in the past). There’s probably a reasonable cap on how much we can charge a customer as a result (more likely pegged to the cost of 10-100 FT roboticists) and in time we’ll have to offer enterprise level SLAs.
So far we’ve seen traction with startups, tech consultants who are servicing big industrial co.s, OEMs, and the large industrial co.s themselves.
My hunch is that as we stop “doing things that don’t scale,” we’ll stop serving the big industrial companies themselves (and leave them to our other customer groups).
Good luck with your new company! Watched the demo; pretty cool stuff. Feel free to reach out to me if I can be of any help (starting a new VC fund), or even just as a sounding board ($my-hn-username @ gmail, or @simon on twitter).
Maybe we can play bocce instead of coffee?
I just did a quick follow up with a demo of our API driving our real tractor, and simulated tractor, simultaneously.
The API used is the same, and we are sending the same commands at the same time to each vehicle.
Check it out here: https://youtu.be/86pBFbTHPOA
1. When you say you're built on top of ROS, do you mean the autonomy stack you'd deploy in an actual robot is built on ROS? Are you using ROS or ROS2?
2. What hardware does your current autonomy stack use? For parts of your stack that'd depend on using deep learning based methods (e.g. any image or lidar data), the models that you'd train would have to use a lot of collected and annotated data specific to a particular problem/industry and this would take a non-trivial amount of time, especially since you said the camera/sensor configuration is not fixed can can potentially be decided I the simulator by the user. How do you plan to tackle this?
PS: Any potential internship opportunities at Polymath in the coming few months that I can apply to?
1) We are mostly built on ROS (for better and worse) and are starting to migrate some containers to ROS2. Down the road I see an architecture that has some ROS, some ROS2, and maybe some homemade stuff (or maybe even other frameworks).
2) Ilia should be able to give you a better spec overview, but we we’re generally trying to be as stack agnostic as sanely physically possible. Roboticists have really strong preferences when it comes to hardware (often shaped by which components previously ruined demos in their life), and we want to be able to work with whatever you want to work with.
Reach out re:internships! No promises, but now that we have a stack you can build on for free that will definitely affect our decision process!
We are approaching the ML problem somewhat differently, we try to give the robot an understanding of navigability of a space. This is done with semantically segmented images, overlayed with depth data where needed.
Once we have this, we flatten it into a 2d costmap of areas where kinematically (ie: ground clearance, terrain handling ability, allowable areas, etc) the vehicle is allowed to go. This is fed to our planner, which in turn generates valid paths for the vehicle to take.
The particular cameras and lidars used are abstracted away in a Hardware Abstraction Layer (HAL) that I've described a bit elsewhere on this page.
Cheers!
I'm curious to hear what you think your competitive advantage is against other players in this space such as Applied Intuition. From my research, Applied Intuition has a focus in the AV space but has been moving into other verticals. Cheers.
Our actual product is autonomy on real vehicles.
We don't plan to ever build any kind of high fidelity sim, just sticking to basic Gazebo or similar.
Ah so the thing that actually takes 90-95% of the time? "Twillo" is no good to me if I dont integrate it properly.
I struggle to see what you've built here other than gym[0] with a largely useless API (Come on, ML training doesn't suck because of lack of HTTP API's), sure its hyper focused on automating vehicle but thats something that's existed for a while - atleast for drones [1].
Shoving sensors on random devices doesn't work that easy, you know that - I dont need a PHD to tell you that. Farmers likely aren't building prod grade ML data sets (except those SWE who gave up FAANG after making $BANK)
What's the real value prop? Why shouldn't I remake the environment in Gym and just cut you out? Gym doesn't require reams of code for environments, same as you.
[0] https://www.gymlibrary.ml [1] https://microsoft.github.io/AirSim/ (2017)
In Caladan we’re giving you a whole autonomous robot (in sim) that you can order around via a simple API.
In our use case the 5-10% that I’m talking about is really hard, but most teams / projects run out of funding before they get to that point (because making the robot, and making it autonomous, is so time and cost intensive it dominates the project).
In most robotics applications teams are building the equivalent of new speculative phone networks, building specialized infrastructure, acquiring specialized hardware companies to ease computer:phone connections, and running out of money before they can find product market fit for mass texting some sort of consumer.
Sure, they can still integrate Twilio poorly and fail. Or they can build a use case that no one cares about - but at least they won’t need 5 years to get to that point.
Definitely, a lot of our work goes into ensuring our Hardware Abstraction Layer (HAL) works well across vehicle types and sensor types. Check out some of my other replies here to double click on this.
For your other notes, Stefan's already covered it, but the TLDR is that we are building on vehicle autonomy, not simulation environments or systems (Caladan is just to get people an easy way to use the Polymath API, without needing a robot first :) )
Our current test vehicle uses remote operation a lot!
I would love you honest take on ROS as an operative system platform for robot development