Launch HN: Enhanced Radar (YC W25) – A safety net for air traffic control
Recent weeks have put aviation safety on everyone’s mind, but we’ve been thinking about this problem for years. Both of us are pilots — we have 2,500 hours of flight time between us. Eric flew professionally and holds a Gulfstream 280 type rating and both FAA and EASA certificates. Kristian flies recreationally, and before this worked on edge computer vision for satellites.
We know from our flying experience that air traffic management is imperfect (every pilot can tell stories of that one time…), so this felt like an obvious problem to work on.
Most accidents are the result of an overdetermined “accident chain” (https://code7700.com/accident_investigation.htm). The popular analogy here is the swiss cheese model, where holes in every slice line up perfectly to cause an accident. Often, at least one link in that chain is human error.
We’ll avoid dissecting this year’s tragedies and take a close call from last April at DCA as an example:
The tower cleared JetBlue 1554 to depart on Runway 04, but simultaneously a ground controller on a different frequency cleared a Southwest jet to cross that same runway, putting them on a collision course. Controllers noticed the conflict unfolding and jumped in to yell at both aircraft to stop, avoiding a collision with about 8 seconds to spare (https://www.youtube.com/watch?v=yooJmu30DxY).
Importantly, the error that caused this incident occurred approximately 23 seconds before the conflict became obvious. In this scenario, a good solution would be a system that understands when an aircraft has been cleared to depart from a runway, and then makes sure no aircraft are cleared to cross (or are in fact crossing) that runway until the departing aircraft is wheels-up. And so on.
To do this, we’ve developed Yeager, an ensemble of models including state of the art speech-to-text that can understand ATC audio. It’s trained on a large amount of our own labeled ATC audio collected from our VHF receivers located at airports around the US. We improve performance by injecting context such as airport layout details, nearby/relevant navaids, and information on all relevant aircraft captured via ADS-B.
Our product piggy-backs on the raw signal in the air (VHF radio from towers to pilots) by having our own antennas, radios, and software installed at the airport. This system is completely parallel to existing infrastructure, requires zero permission, and zero integration. It’s an extra safety net over existing systems (no replacement required). All the data we need is open-source and unencrypted.
Building models for processing ATC speech is our first step toward building a safety net that detects human error (by both pilots and ATC). The latest system transcribes the VHF control audio at about ~1.1% WER (Word Error Rate), down from a previous record of ~9%. We’re using these transcripts with NLP and ADS-B (the system that tracks aircraft positions in real time) for readback detection (ensuring pilots correctly repeat ATC instructions) and command compliance.
There are different views about the future of ATC. Our product is naturally based on our own convictions and experience in the field. For example, it’s sometimes said that voice comms are going away — we think they aren’t (https://www.ericbutton.co/p/speech). People also point out that airplanes are going to fly themselves — in fact they already do. But passenger airlines, for example, will keep a pilot onboard (or on the ground) with ultimate control, for a long time from now; the economics and politics and mind-boggling safet...
117 comments
[ 3.3 ms ] story [ 209 ms ] threadLittle known fact: some towers don't even have radar at all. They're just up in the cab with binoculars.
> This system is completely parallel to existing infrastructure, requires zero permission, and zero integration. It’s an extra safety net over existing systems (no replacement required). All the data we need is open-source and unencrypted.
The part where you explain that it is integrable in the existing chain of command at Airports is proof enough.
Wishing you all the best for your venture.
Existing models I've tried just do a really terrible job at it.
I did collect a bunch of ATIS recordings and hand-transcribed ground-truth data for it a while ago. I can put it up if that might be handy for y'all.
I spent a lot of time out at PDK when I worked briefly in aircraft sales. Nice airport!
Let me work on this and come back! I think we can ship you an API for ATIS there...
The example in the original post is actually a good case study for why trajectory planning alone breaks down. By the time the aircraft are on a predictable collision course with each other, you've lost 10+ seconds of potential remediation time that you would've had if you detected the error when it was spoken. Those 10 seconds really matter in our airspace.
ADS-B helps with this as it's self-reporting. And systems like ASDE-X are useful to track objects once they hit the ground. But low-altitude deconfliction is a big problem.
One problem for sure is that when you're close to the ground you have to be careful about buildings, cell towers, etc. Terrain is one thing, but when you're a few hundred feet AGL, you could quickly be in the way of tall structures if a TCAS alert goes off.
(By the way, I believe EGPWS would take priority over TCAS anyway.)
I'd be curious about what happens when the ASR fails. This is not the place to guess or AI-hallucinate. As a pilot, I can always ask "Say Again" over the radio if I didn't understand. ASR can't do that. Also, it would be pretty annoying if my readback was correct, but the system misunderstood either the ATC clearance or my readback and said NO.
In the very short term, we're deploying this tech more in a post-operation/training role. Imagine being a student pilot, getting in from your solo cross country, and pulling up the debrief will all your comms laid out and transcribed. In this setting, it's helpful for the student to have immediate feedback such as "your readback here missed this detail...", etc. Controllers also have phraseology and QA reviews every 30 days where this is helpful. This will make human pilots and controllers better.
Next, we'll step up to active advisory (mapping to low assurance levels in the certification requirements). There's always a human in the loop that can respond to rare errors and override the system with their own judgement. We're designing with observability as a first-class consideration.
Looking out 5-10 years, it's conceivable that the error rates on a lot of these systems will be super-human (non-zero, but better than a human). It's also conceivable that you could actually respond "Say Again" to a speech-to-speech model that can correct and repeat any mistakes as they're happening.
Of course, that's a long ways from now. And there will always be a human in the loop to make a final judgement as needed.
This is effectively AGI.
And I've not seen anyone reputable suggest that our current LLM track will get us to that point. In fact there is no path to AGI. It requires another breakthrough in pure research in an environment where money is coming out of universities.
To me, speech to text and back seems like an incremental solution, but the holy grail would be the ability to symbolically encode the meaning of the words and translate to and from that meaning. People' phraseology varies wildly (even though it often shouldn't). For example, if I'm requesting VFR flight following, I can do it many different ways, and give the information ATC needs in any order. A system that can convert my words to "NorCal Approach Skyhawk one two three sierra papa is a Cessna one seventy two slant golf, ten north-east of Stockton, four thousand three hundred climbing six thousand five hundred requesting flight following to Palo Alto at six thousand five hundred," is nice, but wouldn't it be amazing if it could translate that audio into structured data:
...for ingestion into potentially other digital-only analysis systems. You could structure all sorts of routine and non-routine requests like this, and check them for completeness, use it for training later, and so on. Maybe one day, display it in real time on ATC's terminal and in the pilot's EFIS. With structured data, you could associate people's spoken tail numbers with info broadcast over ADS-B and match them up in real time, too. I don't know, maybe this already exists and I just re-invented something that's already 20 years old, no idea. IMO there's lots of innovation possible bringing VHF transmissions into the digital world!Kidding aside, yes, you're exactly right. We're already doing this to a large degree and getting better. Lots of our own data labeling and model training to make this good.
I imagine that aviation regulatory bodies have high standards for this - a tool being fully additive to existing tools does not necessarily mean that it's cleared for use in a cockpit or in an ATC tower, right? Do you have thoughts about how you'll approach this? Also curious from a broader perspective - how do you sell any alerting tool into a niche that's highly conscious of distractions, and of not just false positive alerts but false negatives as well?
There are lots of small steps on this ladder.
The first is post-operational. You trigger an alert async and someone reviews it after the fact. Tools like this help bring awareness to hot spots or patterns of error that can be applied later in real time by the human controller.
A step up from that is real-time alerting, but not to the main station controller. There's always a manager in the tower that's looking over everyone's shoulder and triaging anything that comes up. That person is not as focused on any single area as the main controllers. There's precedence for tools surfacing alerts to the manager, and then they decide whether it's worth stepping in. This will probably be where our product sits for a while.
The bar to get in front of an active station controller is extremely high. But it's also not necessary for a safety net product like this to be helpful in real time.
When, not if. The "artificial intelligence" as it is presently understood is statistical in nature. To rely on it for air traffic control seems quite irresponsible.
Bulk of the instrument flight training is "mindgames" anyways - you see nothing other than instruments, your "seat in the pants" is likely to cheat you..
Possibly going a step further, the state of teaching aids available to CFIs is pretty sad, with online quizzes and pre-recorded videos being the pinnacle of what I have experienced... this would be an awesome opportunity to try and build "automatic CFI" - not counting for AATD time under current rules but better than chair-flying (the process of imagining a flight and one's reactions).
Eh, I guess I can flex a little. Living in the Pacific North West, I do not have to play mind games. I can almost get IMC delivered on demand. :P
Listening on multiple channels might help at busier airports. Ground, ramp, approach, departure, and enroute are all on different channels. Military aircraft have their own system. (That may have contributed to the DCA accident.)
Something like this was proposed in the late 1940s. The FAA was trying to figure out how to do air traffic control, and put out a request for proposals. General Railroad Signal responded.[1] They proposed to adapt railroad block signalling to the sky. Their scheme involved people listening to ATC communications and setting information about plane locations into an interlocking machine. The listeners were not the controllers; they just did data entry. The controllers then had a big board with lights showing which blocks of airspace were occupied, and could then give permission to aircraft to enter another block.
Then came radar for ATC, which was a much better idea.
[1] https://searchworks.stanford.edu/view/1308783
Military aircraft are typically equipped with UHF radios (in addition to civilian VHF). Many of the same systems apply, just a different RF band. And we're in the process of adding UHF capabilities to our product as a lot of these military aircraft land at civilian airports for training exercises.
I can't imagine what would've happened if we adopted block signaling for ATC ...
You don't have to imagine. We already do in many places. The North Atlantic Tracks are essentially exactly that. Aircraft give position reports and estimates, those positions reports are used to decide whether an aircraft can climb though which levels etc.
It's also used extensively in an IFR non-radar environments. Exactly why aircraft have to cancel IFR at uncontrolled airfields in the US or under a procedural ATC service in the UK. You hear it a lot around the Caribbean and Bahamas too.
1. Good overview of the technologies you are using, but what product are you planning on building or have built? I understand what you are doing and it's "extra safety over existing systems" but how does it work for the end user? Is the end user an ATC or a pilot?
2. You will find that introducing new systems into this very conservative field is hard. I've built avionics and ATC electronics. The problem isn't normally technology, it's the paperwork. How do you plan on handling this?
2. Agree
[edit] (oops, sorry, seeing your edit)
2. The regulation allows for escalating assurance levels. We'll start with low assurance (advisory) and climb that ladder. We're definitely not naive about it; this will be hard and annoying. But it's inconceivable that someone won't do this in the next 10 years. Too important.
Do ground vehicles also have GPS trackers with a radio transmitter, or do they just use normal ADS-B?
Why do we still rely on analog narrowband AM voice over VHF to do vital comms like air traffic control? Same way as we did in the 1940s!
We should be transcribing the messages and relaying them digitally with signatures and receipt acknowledgement.
As far as digital decoding delay is concerned, this is a negligible number if implemented correctly.
Like, check in with the controller but most messages are sent electronically and acknowledged manually.
I have your clearance, advise when ready to copy, then you write everything down on kneeboard with a pencil and then manually put it in the navigation system, is a little archaic.
certainly speech to text is a useful transition but in the long run the controller could click on an aircraft and issue the next clearance with a keyboard shortcut. then the pilot would get a visual and auditory alert in the cockpit and click to acknowledge.
I would hope someone at NASA or DARPA or somewhere is working on it. And then of course the system can detect conflicts, an aircraft not following the clearance etc.
This is how big operations handle clearances today, complete with integration into the FMS. The pilot simply reviews the clearance and accepts it.
There already is: Controller Pilot Data Link Communications (CPDLC).
Get an instruction, press to confirm.
At the moment, this is only used for certain types of things (clearances, frequency changes, speed assignments, etc.) along with voice ATC.
https://en.wikipedia.org/wiki/Controller%E2%80%93pilot_data_...
I feel like, with proper UX in the cockpit and on the controller console, making it easy to send/acknowledge the clearance, and intrusively demanding immediate acknowledgment for important messages, with the controller able to talk to the pilot if it isn't immediately acknowledged, structured messages would save time, be more accurate, allow automated checks, i.e. be a superior substitute.
UX needs a ton of work and human factors validation, and would take 20 years to implement. But if you were starting from a blank slate it seems like the way to go!
Also, AM voice on VHF is not full duplex and the blocking problem is very real and could be addressed potentially
I believe scaling laws will hold as we start to feed all of this context data into an integrated model. You could imagine a deep-q style reinforcement learning model that ingests layers of structured and visual data and outputs alerts and eventually commands. The main challenge I foresee here will be observability... it's easy enough to shove a ton of data into a black box and get a good answer 98% of the time. But regulation is likely to require such a system to be highly observable/explainable so the human can keep up with what's going on and step in as needed.
Looking further into the future, it's plausible the concrete structures of today with humans looking out windows will be replaced with sensor packages atop a long flagpole that stream high-res optical/ir camera data, surface radar, weather information, etc into a control room with VR layers that help controllers stay on top of busier and busier airspace.
We've accelerated past our capabilities and need to slow down. ATC has incentive to slot takeoffs and landings as close as possible, but that is in tension with the goal of safety.
> Air traffic control from first principles looks significantly more automated.
We have a system 'designed' by history, not by intention. The ATC environment is implemented in the physical world, everyone has to work around physical limitations
Automation works best in controlled environments with limited scope. The more workaround you have to add, the noisier things get, and that's why we use humans to filter the noise by picking important things to say. Humans can physically experience changes in the environment, and our filters are built from our experiences in the physical world.
Anyway, sorry that isn't a question.
> We've accelerated past our capabilities and need to slow down.
This is a super interesting meditation. As much work as there is to be done now, the demand for air traffic is growing and power laws are concentrating it into tight airspace bubbles. It would behoove us to figure out how to make airspace more dense without compromising safety. There's lots of good economic incentive for this.
My suggestion would be to make things as repeatable and consistent as possible. This would mean forcing some airports to change their practices to be consistent with everyone else, and forcing physical layout changes and construction. Unfortunately an app can't do that =( ... and the benefits are on the other side of a paradigm shift, so it's hard to make it happen naturally.
>> more dense
Large, high passenger capacity airliners have gone out of style but that would have been the best way to get passenger density up.
https://aviationsystems.arc.nasa.gov/publications/2019/SciTe...
Curious if OP has seen this paper / project before?
What were your main takeaways from this paper?
Pitching air traffic management is going to be a years long process.
Getting certified for on-board avionics is similarly challenging.
In the meantime, we'll get better and better at monitoring the airspace system and deploy that technology into unregulated applications like post-operational roles.