Launch HN: Silurian (YC S24) – Simulate the Earth
What is it worth to know the weather forecast 1 day earlier? That’s not a hypothetical question, traditional forecasting systems have been improving their skill at a rate of 1 day per decade. In other words, today’s 6-day forecast is as accurate as the 5-day forecast ten years ago. No one expects this rate of improvement to hold steady, it has to slow down eventually, right? Well in the last couple years GPUs and modern deep learning have actually sped it up.
Since 2022 there has been a flurry of weather deep learning systems research at companies like NVIDIA, Google DeepMind, Huawei and Microsoft (some of them built by yours truly). These models have little to no built-in physics and learn to forecast purely from data. Astonishingly, this approach, done correctly, produces better forecasts than traditional simulations of the physics of our atmosphere.
Jayesh and Cris came face-to-face with this technology’s potential while they were respectively leading the [ClimaX](https://arxiv.org/abs/2301.10343) and [Aurora](https://arxiv.org/abs/2405.13063) projects at Microsoft. The foundation models they built improved on the ECMWF’s forecasts, considered the gold standard in weather prediction, while only using a fraction of the available training data. Our mission at Silurian is to scale these models to their full potential and push them to the limits of physical predictability. Ultimately, we aim to model all infrastructure that is impacted by weather including the energy grid, agriculture, logistics, and defense. Hence: simulate the Earth.
Before we do all that, this summer we’ve built our own foundation model, GFT (Generative Forecasting Transformer), a 1.5B parameter frontier model that simulates global weather up to 14 days ahead at approximately 11km resolution (https://www.ycombinator.com/launches/Lcz-silurian-simulate-t...). Despite the scarce amount of extreme weather data in historical records, we have seen that GFT is performing extremely well on predicting 2024 hurricane tracks (https://silurian.ai/posts/001/hurricane_tracks). You can play around with our hurricane forecasts at https://hurricanes2024.silurian.ai. We visualize these using [cambecc/earth] (https://github.com/cambecc/earth), one of our favorite open source weather visualization tools.
We’re excited to be launching here on HN and would love to hear what you think!
155 comments
[ 4702 ms ] story [ 796 ms ] threadIt seems like this is another instance of The Bitter Lesson, no?
I thought this was a good quote:
> We want AI agents that can discover like we can, not which contain what we have discovered.
Deep Blue wasn't a brute-force search. It did rely on heuristics and human knowledge of the domain to prune search paths. We've always known we could brute-force search the entire space but weren't satisfied with waiting until the heat death of the universe for the chance at an answer.
The advances in machine learning do use various heuristics and techniques to solve particular engineering challenges in order to solve more general problems. It hasn't all come down to Moore's Law.. which stopped bearing large fruit some time ago.
However that still comes at a cost. It requires a lot of GPUs, land, energy, and fresh water, and Freon for cooling. We'd prefer to use less of these resources if possible while still getting answers in a reasonable amount of time.
It's certainly true that "just throw a bunch of GPUs at it" is wasteful, but it does achieve results.
And even though solutions to many such problems were in the NP or NP-hard categories it didn’t mean that we couldn’t get useful results.
But it still gave us better results by applying what we know about search strategies and reinforcement to provide guidance and heuristics. Even Alpha didn’t use the most general algorithms and throw hardware at the problem. Still took quite a lot of specialized software and methods to fine-tune the overall system to produce the results we want.
It's not exactly an LLM but it works in a similar fashion.
Notably forecast skill is quantifiable, so we'd need to see a whole lot of forecast predictions using what is essentially the stochastic modelling (historical data) approach. Given the climate is steadily warming with all that implies in terms of water vapor feedback etc., it's reasonable to assume that historical data isn't that great a guide to future behavior, e.g. when you start having 'once every 500 year' floods every decade, that means the past is not a good guide to the future.
Large Language Model + Large Earth Model
What will your differentiators be?
Are you paying for weather data products?
Better weather predictions are worth money, plain and simple.
What else do you hope to simulate, if this becomes successful?
Signed,
A California Resident
The old ML maxim was “don’t expect models to do anything a human expert couldn’t do with access to the same data”, but that’s clearly going to way of Moore’s Law… I don’t think a meteorologist could predict 11km^2 of weather 10 days out very accurately, and I know for sure that a neuroscientists couldn’t recreate someone’s visual field based on fMRI data!
Essentially random outputs from deterministic systems are unfortunately not rare in nature…. And I suspect that because of the relatively higher granularity of geology vs the semicohesive fluid dynamics of weather, geology will be many orders of magnitude more difficult to predict.
That said, it might be possible to make useful forecasts in the 1 minute to 1 hour range (under the assumption that major earthquakes often have a dynamic change in precursor events), and if accuracy was reasonable in that range, it would still be very useful for major events.
Looking at the outputs of chaotic systems like geolocated historical seismographic data might not be any more useful than 4-10 orders of magnitude better than looking at previous lottery ball selections in predicting the next ones…. Which is to say that the predictive power might still not be useful even though there is some pattern in the noise.
Generative AI needs a large and diverse training set to avoid overfitting problems. Something like high resolution underground electrostatic distribution might potentially be much more predictive than past outputs alone, but I don’t know of any such efforts to map geologic stress at a scale that would provide a useful training corpus.
They're a cool little team based in Copenhagen. Would be useful, for example, to look at the correlation between your weather data and regional energy production (solar and wind). Next level would be models to predict national hydro storage, but that is a lot more complex.
My advice is to drop the grid itself to the bottom of the list, and I say this as someone who worked at a national grid operator as the primary grid analyst. You'll never get access to sufficient data, and your model will never be correct. You're better off starting from a national 'adequacy' level and working your way down based on information made available via market operators.
But it's non-trivial to scale these new techniques into the field. A major factor is the scale of interest. FEMA's FIRMaps are typically at a 10m resolution not 11km.
99 Percent Invisible did an episode about this recently:
https://99percentinvisible.org/episode/nbft-05-the-little-le...
The counter proposal was indeed funded by the City of Miami, to point out how ridiculous it would be to have a 20 foot concrete wall around the city.
As a local resident, I loved seeing this sad 3D render in particular, which even has a graffiti on it nearly spelling "Berlin": https://i0.wp.com/dirt.asla.org/wp-content/uploads/2022/09/0...
In seriousness, it was really cool to see the counter proposal's "nature-based solution" which would design 39 acres of distributed barrier islands around the coastline, to block storm surge naturally.
They’re selling height maps of South-Africa, primary for flooding prediction for insurance companies.
Smart & friendly bunch.
Windy(.com) premium also has a great hybrid weather radar+forecast view which was recently released and which I find has been very effective at predicting rain at a specific location on the map vs "nearby". With smaller weather patterns it is entirely possible for it to rain a few blocks away but not at your location. An 11-KM resolution weather forecast (as referenced above) will not be able to capture this nuance.
If you've ever heard of the Lorenz/Butterfly Effect/Strange Attractors, those chaotic systems were discovered because of a discrepancy between two parallel weather simulations. One preserved the original simulation's calculation train while the other started off with simply the previous results (out to like 10 decimals) and suffered from a rounding error and thus both simulations diverged hugely.
Lorenz was trying to simulate weather by subdividing the atmosphere into tons and tons of cubes. Really interesting reading/video watching tbh.
Regardless, you're just trying to personally attack me. That's a great use of both our time.
You are out here implying these guys are a fraud. Being told to pull your head in is not personal.
EDIT: the post I am responding to was altered to sound much less confrontational. It was originally:
> So what exactly are you “launching” and why does it require venture capital?
It's the first line man. The visual is just a visual, their product is the data being visualized.
They are launching an AI model which they claim produces higher quality weather data than traditional models relying on physical simulation. And they used this visualization library to make an engaging website.
Constructively, you have gotten to this position by overreacting to a perceived “clone” and failing to be enlightened by the numerous comments and the original post explaining the purpose.
Respectfully, I suggest you take a breath and try to disassociate from whatever emotional reaction you are having about this.
From the post.
[1] https://github.com/cambecc/earth
What more did you want from them? (Genuine question.)
nullschool is obscure enough to the general audience that when I saw it there was an immediate red flag.
If only specialized scientists can see the difference between the sites, it's a presentation problem.
- skip reading the post (which explains all of this)
- skip the first link in the post (which explains all of this)
- go straight to the second link in the post, to the interface
- skip the "about" link in the interface (which explains all of this)
The post has been edited.
What exactly is predicted and what is the actual path in those videos?
I had a web app online in 2020-22 called Skim Day that predicted skimboarding conditions on California beaches that was mostly powered by weather APIs. The tide predictions were solid, but the weather itself was almost never right, especially wind speed. Additionally there were some missing metrics like slope of beach which changes significantly throughout the year and is very important for skimboarding.
Basically, I needed AI. And this looks incredible. Love your website and even the name and concept of "Generative Forecasting Transformer (GFT)" - very cool. I imagine the likes of Surfline, The Weather Channel, and NOAA would be interested to say the least.
1. How will you handle one-off events like volcanic eruptions for instance? 2. Where do you start with this too? Do you pitch a meteorology team? Is it like a "compare and see for yourself"?
Re where do we start. A lot of organisations across different sectors need better weather predictions or simulations that depend on weather. Measuring the skill of such models is a relatively standard procedure and people can check the numbers.
Shameless plug: recently we've built a demo that allows you to search for objects in San Francisco using natural language. You can look for things like Tesla cars, dry patches, boats, and more. Link: https://demo.bluesight.ai/
We've tried using Clay embeddings but we quickly found out that they perform poorly for similarity search compared to embeddings produced by CLIP fine tuned on OSM captions (SkyScript).
We did try to relate OSM tags to Clay embeddings, but it didn't scale well. We did not give up, but we are re-considering ( https://github.com/Clay-foundation/earth-text ). I think SatClip plus OSM is a better approach. or LLM embeddings mapped to Clay embeddings...
We tried to search for bridges, beaches, tennis courts, etc. It worked, but it didn't work well. The top of the ranking was filled with unrelated objects. We found that similarity scores are stacked together too much (similarity values are between 0.91 and 0.92 with 4 digit difference, ~200k tiles), so the encoder made very little difference between objects.
I believe that Clay can be used with additional fine-tuning for classification and segmentation, but standalone embeddings are pretty poor.
Check this: https://github.com/wangzhecheng/SkyScript. It is a dataset of OSM tags and satellite images. CLIP fine-tuned on that gives good embeddings for text-to-image search as well as image-to-image.
The biggest issue is that the basic data model for population behavior is a sparse metastable graph with many non-linearities. How to even represent these types of data models at scale is a set of open problem in computer science. Using existing "big data" platforms is completely intractable, they are incapable of expressing what is needed. These data models also tend to be quite large, 10s of PB at a bare minimum.
You cannot use population aggregates like census data. Doing so produces poor models that don't ground truth in practice for reasons that are generally understood. It requires having distinct behavioral models of every entity in the simulation i.e. a basic behavioral profile of every person. It is very difficult to get entity data sufficient to produce a usable model. Think privileged telemetry from mobile carrier backbones at country scales (which is a lot of data -- this can get into petabytes per day for large countries).
Current AI tech is famously bad at these types of problems. There is an entire set of open problems here around machine learning and analytic algorithms that you would need to research and develop. There is negligible literature around it. You can't just throw tensorflow or LLMs at the problem.
This is all doable in principle, it is just extremely difficult technically. I will say that if you can demonstrably address all of the practical and theoretical computer science problems at scale, gaining access to the required data becomes much less of a problem.
IMO the short answer is that such models can be made to generate realistic trajectories, but calibrating the model the specific trajectory of reality we inhabit requires knowledge of the current state of the world bordering on omniscience.
[0]: https://www.santafe.edu/research/results/working-papers/asse...
Specifically, I could imagine throwing current weather data at the model and asking it what it thinks the next most likely weather change is going to be. If it's accurate at all, then that could be done on any given day without further training.
The problems happen when you start throwing data at it that it wasn't trained on, so it'll be a cat and mouse game. But it's one I think the cat can win, if it's persistent enough.
The entire class of deep learning or AI-based weather models involves a very specific and simple modeling task. You start with a very large training set which is effectively a historical sequence of "4D pictures" of the atmosphere. Here, "4D" means that you have "pixels" for latitude, longitude, altitude, and time. You have many such pictures of these for relevant atmospheric variables like temperature, pressure, winds, etc. These sequences are produced by highly-sophisticated weather models run in what's called a "reanalysis" task, where they consume a vast array of observations and try to create the 4D sequence of pictures that are most consistent with the physics in the weather model and the various observations.
The foundation of AI weather models is taking that 4D picture sequence, and asking the model how to "predict" the next picture in the sequence, given the past 1 or 2 pictures. If you can predict the picture for 6 hours from now, then you can feed that output back into the model and predict the next 6 hours, and so on. AI weather models are trained such that this process is mostly stable, e.g. the small errors you begin to accumulate don't "blow up" the model.
Traditionally, you'd use a physics-based model to accomplish this task. Using the current 3D weather state as your input, you integrate the physics equations forward in time to make the prediction. In many ways, today's AI weather models can be thought of as a black box or emulator that reproduces what those physics-based models do - but without needing to be told much, if any of the underlying physics. Depending on your "flavor" of AI weather model, the architecture of the model might draw some analogies to the underlying physics. For example, NVIDIA's models use Fourier Neural Operators, so you can think of them as learning families of equations which can be combined to approximate the state of the atmosphere (I'm _vastly_ over-simplifying here). Google DeepMind's GraphCast tries to capture both local and non-local relationships between fields through it's graph attention mechanisms. Microsoft Aurora' (and Silurian's, by provenance, assuming it's the same general type of model) try to capture local relationships through sliding windows passed over the input fields.
So again - no new knowledge or physics. Just a surprisingly effective of applying traditional DL/AI tools to a specific problem (weather forecasting) that ends up working quite well in practice.
This _class_ of models (not Aurora, or Silurian's model specifically) can potentially improve on this a bit by incorporating forecast error at longer lead times in their core training loss. This is already done in practice for some major models like GraphCast and Stormer. But these models are almost certainly not a magical silver bullet for 10x'ing forecast accuracy.
For one, I always thought it would be informative for things like game engines to have a reference point. How fast to streams typically flow in this type of environment? What tree species are even in this geo?
https://data.neonscience.org/data-api/graphql/explorer/build...
Where we're going, we don't need "Data Products".
Once upon a time I converted spectral-transform-shallow-water-model (STSWM or parallelized as PSTSWM) from FORTRAN to Verilog. I believe this is the spectral-transform method we have run for the last 30 years to do forecasting. The forecasting would be ~20% different results for 10-day predictions if we truncated each operation to FP64 instead of Intel's FP80.
1. The truth is we still have to investigate the the numerical stability of these models. Our GFT forecast rollouts are around 2 weeks (~60 steps) long and things are stable in in that range. We're working on longer-ranged forecasts internally.
2. The compute requirements are extremely favorable for ML methods. Our training costs are significantly cheaper than the fixed costs of the supercomputers that government agencies require and each forecast can be generated on 1 GPU over a few minutes instead of 1 supercomputer over a few hours.
3. There's a similar floating-point story in deep learning models with FP32, FP16, BF16 (and even lower these days)! An exciting area to explore
Best of luck, and thanks for taking the leap! Humanity will surely thank you. Hopefully one day you can claim a bit of the NWS’ $1.2B annual budget, or the US Navy’s $infinity budget — if you haven’t, definitely reach out to NRL and see if they’ll buy what you’re selling!
Oh and C) reach out if you ever find the need to contract out a naive, cheap, and annoyingly-optimistic full stack engineer/philosopher ;)
[1] https://x.com/karpathy/status/1835024197506187617 [2] https://www.youtube.com/watch?v=-KMdo9AWJaQ&t=1010s
Re question 2: Simulations don't need to be explainable. Being able to simulate simply means being able to provide a resonable evolution of a system given some potential set of initial conditions and other constraints. Even for physics-based simulations, when run at huge scale like with weather, it's debatable to what degree they are "interpretable".
Thanks for your questions!