Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
In 1981, weather disasters caused $3.5 billion in damages in the United States. In 2023, that number was $94.9 billion (https://www.ncei.noaa.gov/access/billions/time-series). The National Weather Service spends billions annually on its network of weather balloons, satellites, and aircraft sensors – generating hundreds of terabytes of data every day. This data, called observation data, is fed into massive supercomputers running advanced physics to produce global weather forecasts. Despite this cost, there are still places in the US where we don't know what the temperature will be two days from now: https://www.washingtonpost.com/climate-environment/interacti.... And for the rest of the world that lacks weather infrastructure? There’s always the Weather Rock: https://en.wikipedia.org/wiki/Weather_rock.
The most important data for these forecasts come from vertical data ‘slices’ of the atmosphere, called soundings. Every day 2,500 single-use latex radiosondes are launched across the globe to collect these soundings. They stay aloft for about two hours before popping and falling back to Earth. Launch sites for these systems are sparse in Latin America and Africa, and they’re completely non-existent over oceans. This leaves about 80% of the globe with inadequate weather data for accurate predictions.
The coverage gap became painfully evident to Max and Alex during their time at Urban Sky. While building balloons for high-altitude aerial imaging, they kept running into a problem: no matter what weather forecast they used, they couldn’t get accurate wind predictions for the upper atmosphere. They tried all of the free and commercial forecast products, but none of them were accurate enough. Digging into it more, we learned that a big part of the problem was the lack of high-quality in-situ data at those altitudes.
To solve this problem, our systems ascend and descend between sea level and 65,000ft several times a day to collect vertical data soundings. Each vehicle (balloon + payload) weighs less than a pound and can be launched from anywhere in the world, per the FAA and ICAO reg. Here’s one we launched from Potrero Hill in SF, https://youtu.be/75fN5WpRWH0 and here’s another near the Golden Gate Bridge, https://youtu.be/7yLmzLPUFVQ. Although we can’t “drive” these balloons laterally, we can use opposing wind layers to target or avoid specific regions. Here’s what a few simulated flight paths look like, to give you an idea: https://youtu.be/F_Di8cjaEUY
Our payload uses a satellite transceiver for communications and a small, thin film solar panel array to generate power. In addition to the weather data, we also get real-time telemetry from the vehicles, which we use to optimize their flight paths. This includes maintaining the optimal spacing between balloons and steering them to a recovery zone at the end of their lifespan so we can recycle them.
These systems spend most of their time in the stratosphere which is an extremely unforgiving environment. We’ll often see temperatures as low as -80°C while flying...
172 comments
[ 2.4 ms ] story [ 193 ms ] threadOne question that came to mind, and this applies to all weather baloons not yours specifically, with the large number of weather balloons launched daily, how is it That more aren’t sucked into airplane engines causing potential disaster for the airplane? Thanks
Can help with the federal contract side and mass manufacturing etc.
Charles@turnsys.com
Right now, most weather balloons fall back to Earth and stay where they land unless someone happens across them (since they can't be controlled and only last a couple of hours).
What does this look like in practice? As you mentioned I know you don't really have any lateral control, but I imagine you can wait for it to overfly somewhere convenient to descend?
Pull up https://www.pivotalweather.com/model.php?m=nam&p=sfct-mean-i... and pick some point (note the 'click for point sounding'). You can see the wind direction at that location as a function of altitude.
Using this as a vector field, you can do "the balloon is here now, 30 minutes from now it will be there, if it is at altitude Z at that time, it will be follow the wind in this direction" which in turn allows you to predict where it will be in 30 minutes and take the forecast for that location at that time and determine what altitude you want to be at.
Saying I want it to be at X,Y at some time is solving this backwards. Which isn't necessarily easy, but it's computable.
The Balloon Learning Environment https://research.google/blog/the-balloon-learning-environmen... (https://news.ycombinator.com/item?id=31155137 - 73 points | 10 comments)
(2016) Station-keeping of a high-altitude balloon with electric propulsion and wireless power transmission: A concept study https://www.sciencedirect.com/science/article/abs/pii/S00945...
(2022) Station-keeping for high-altitude balloon with reinforcement learning - https://www.sciencedirect.com/science/article/abs/pii/S02731...
(2023) Resource-Constrained Station-Keeping for Helium Balloons using Reinforcement Learning - https://arxiv.org/abs/2303.01173
Chasing the citations from those papers to previous works can provide a fairly deep rabbit hole of things to read.
At least, that's how I understand hot air balloons "steer".
Congratulations for a great non-saas market and product!
Right now we're focused on the stratospheric forecasts because that's what we know really well (and we already have some interested customers). Our data/models are great for all kinds of forecasts, including ground forecasts, and we'll quickly expand beyond the stratosphere.
I'm picturing having a few dozen at launch site containers, launching them at the start of a day of flying, having them programmed to land in a rural area that a member can pick them up from and return to the launch sites.
Loon used pumps and an interior air ballast like blimps do. So clearly there are a few ways.
https://en.m.wikipedia.org/wiki/Thermoelectric_effect#Peltie...
Asking because my research at the University of Oxford was around hyper space-efficient data transfer from remote locations for a fraction of the price.
The result was an award-winning technology (https://jsonbinpack.sourcemeta.com) to serialise plain JSON that was proven to be more space-efficient than every tested alternative (including Protocol Buffers, Apache Avro, ASN.1, etc) in every tested case (https://arxiv.org/abs/2211.12799).
If it's interesting, I'd love to connect and discuss (jv@jviotti.com) how at least the open-source offering could help.
> Our payload uses a satellite transceiver for communications
> When transmitting data over the Internet, time is the bottleneck, making computation essentially free in comparison.
i thought this was an odd sales pitch from the jsonbinpack site, given that a central use-case is IoT, which frequently runs on batteries or power-constrained environments where there's no such thing as "essentially free"
That said, the production-ready implementation of JSON BinPack is designed to run on low powered devices and still provide those same benefits.
A lot of the current work is happening at https://github.com/sourcemeta/jsontoolkit, a dependency of JSON BinPack that implements a state-of-the-art JSON Schema compiler (I'm a TSC member of JSON Schema btw) to do fast and efficient schema evaluation within JSON BinPack on low powered devices compared to the current prototype (which requires schema evaluation for resolving logical schema operators). Just an example of the complex runtime-efficiency tracks we are pursuing.
I would imagine that CPUs are much more efficient than a satellite transmitter, probably? I guess you'd have to balance the additional computational energy required vs. the savings in energy from less transmitting.
ironically the main criticism i've heard of these is how power-inefficient they are :P
However, schema-less is very useful too. The idea with supporting both is that clients can start with schema-less, without bothering with schemas, and already get some space-efficiency. But once they are using JSON BinPack, they can start incrementally adding schema information on the messages they care about, etc to start squeezing out more performance.
Compare that with i.e. Protocol Buffers, which pretty much forces you to use schemas from the beginning and it can be a bit of a barrier for some projects, mainly at the beginning.
That means that you can use any tooling/approach from the wide JSON Schema ecosystem to manage schema evolution. A popular one from the decentralised systems world is Cambria (https://www.inkandswitch.com/cambria/).
That said, I do recognise that schema evolution tech in the JSON Schema world is not as great as it should be. I'm a TSC member of JSON Schema and a few of us are definitely thinking hard on this problem too and trying to make it even better that the competition.
Asking because we use msgpack in production at work and it can sometimes be a bit slower to encode/decode than is ideal when dealing with real-time data.
The TLDR is that is that if you use JSON BinPack on schema-less mode, its still more space-efficient than MessagePack but not by a huge margin (depends on the type of data of course). But if you start passing a JSON Schema along with your data, the results become way smaller.
Please reach out to jv@jviotti.com. I would love to discuss your use case more.
If you want to get deeper, I published two (publicly available) deep papers studying the current state of JSON-compatible binary serialization that you might enjoy. They study in a lot of detail technologies like Protocol Buffers, CBOR, MessagePack, and others that were mentioned in the thread:
- https://arxiv.org/abs/2201.02089
- https://arxiv.org/abs/2201.03051
Hope they are useful!
Of course there is still a lot to do, but the idea being that what you get with JSON BinPack is extremely close to what you would have done for manually encoding your data, except that you don't have to worry about encoding things yourself :) Thus you get the best of both worlds: the nicety of JSON and the space-efficiency of manual encoding.
If it's not proprietary, I'd love to know - how do you "steer" vertically between different wind layers to move in the direction you want to go?
Can't wait to see where you guys take this!
So I can't get into exactly how we do our altitude control, but the Google Loon project has a really great explanation of how they made their (very big) balloons go up and down: https://x.company/projects/loon/
Loon made all of their research public after they shut down, and we're obviously heavily inspired by their work. Our systems use a lot of the tech they pioneered, just on a much, much smaller scale (for reference, Loon's balloons were the size of tennis courts) Here's the PDF in case you're interested in checking out the 400+ page writeup: https://storage.googleapis.com/x-prod.appspot.com/files/The%...
https://www.scientificballoonsolutions.com/news/
This sentence is legendary
You are, as long as you mean balloons to business
https://news.ycombinator.com/item?id=41173161
Such that we can see them?
---
As others mentioned, this is a fantastic launch.
I'd love to have one permanently teathered to a place of my choise using a fiber-optic+carbon/kevlar thread to hold it in place with data coming down the fiber, and have the camera and pico compute data and radios powered by solar.
It would be very cool if you could do an open house for bay area geeks to come and just ooh and ahh at the gadgetry. Even a virtual open house would be cool. Something less than a full demo, and more focused on the story behind the gestation and launch of the project (and then a demo.)
Max, the first engineer at Urban Sky, hit me up and asked if I wanted to build their mission control. At the time, Urban Sky was just a four-person team, so they couldn’t pay me as much, but I jumped at the chance, even though it meant taking about half my usual salary.
Funny enough, my SaaS background actually helped me create mission control software that was way ahead of the curve!
I guess my advice is, find a small company you're passionate about, where you can make a big impact, and be open to taking a pay cut. It helps the company take less of a risk on you, and you get to work on something that really matters. Plus, when you’re solving real problems, things tend to work out, and eventually, you’ll end up making what you should in salary.
Does that surprise someone? I think I would not have guessed this growth to be on such a scale. The chart suggests that severe storms are the main culprit.
I read a paper a few years back which dove into how the data sources for weather damage assessment have changed a lot over the years. Much of the increase is due to more complete reporting and changes in categorization. Also, nowadays more things are insured and modern IT has made gathering the insurance reporting far more exhaustive. Plus local, state and federal agencies responsible for relief and/or recovery are gathering and reporting increasing amounts of data with each decade since the 70s (in part because their budgets rely on it). Factors like these mean in prior decades the total damage costs may have been more similar to today's than they appear but a lot of the damage data we gather and report now wasn't counted or gathered then.
Although I have no experience related to weather science, I remember the paper because it made me realize how many broad-based, multi-decadal historical data comparisons we see should have sizable error bars (which never make it into the headline and rarely even into the article). Data sources, gathering and reporting methods and motivations are rarely constant on long time scales - especially since the era of modern computing. Of course, good data scientists try to adjust for known variances but in a big ecosystem with so many evolving sources, systems, entities and agencies, it quickly gets wickedly complex.
This is definitely part of it. Another part is that people live in more at-risk regions now than in the past (Florida is a great example, population has more than 10x'd since 1950).
Ultimately, the way we think about it is no matter what the underlying cause, weather-related damages could be significantly reduced with better data/forecasts
I agree. My point was only so those surprised by the massive increase in cost estimates can put those numbers in perspective since neither the average quantity nor severity of adverse weather events have changed substantially over the decades.
Providing more granular data to enable more accurate and timely weather forecasts is a sound business thesis even if adverse weather isn't happening 2x more frequently or energetically. It's still a large economic impact where money can be saved. More broadly, better forecasts can improve agricultural yields, reduce business disruption and increase throughput of transportation networks.
Obviously what we're doing can't prevent severe weather from happening, but even very small improvements in accuracy and timelines can have a massive beneficial effect when a disaster does happen. My cofounders and I are all from Florida, so hurricanes are the most visceral examples for us. When hurricanes hit, there are always issues along the lines of "we didn't have the right resources in the right places to respond effectively." Those types of issues can be combated with better info.
[0]: https://www.statista.com/statistics/818411/weather-catastrop...
Using [0] $3.5 bn in 1981 would have been worth $11.7 bn in 2023.
Another comment [1] noted (but unfortunately didn't cite) that two years later the damage was assessed at $36 bn, or $110 bn in 2023 dollars.
[0] https://www.usinflationcalculator.com/
[1] https://news.ycombinator.com/item?id=41295116