Launch HN: InBalance (YC W21) – Short-term energy market forecasting
We met playing ultimate frisbee in Cambridge, UK, and quickly found common interests in statistics and optimization. Thomas had previously worked on wind turbine placement problems, providing experience with power markets, and we discussed them but didn't see an immediate entry point, so Thomas continued his statistics PhD and Rajan worked in ML research and GPU algorithm design at a startup.
A year ago we heard of a need for better wind power forecasts and started to look at the market more closely. We found a gap emerging from the increase in the prevalence of storage, especially lithium-ion, grid-scale batteries. It seemed like an interesting and useful real-world application of machine learning, particularly with the possibility of reducing carbon emissions, so once the business case looked tenable, we decided to go ahead!
Electrical power markets have become increasingly volatile due extreme weather events and increased prevalence of intermittent renewables. In response to this, producers are bringing on more flexible generation assets such as batteries to even out fluctuations in supply, and electrical consumers are aiming to increase their ability to modulate demand to better take advantage of cheap intermittent power. These assets don't fit into the day-ahead markets designed for mostly traditional steam power plants, making it difficult to choose when to use them. Our forecasts help traders better align their use with power availability, who now do so on gut feeling or low-quality coarse-grained forecasts. We hope this will increase the value enough to make transitioning to renewables more financially appealing.
Most standard machine learning approaches struggle in particular with price forecasting due to the limited data, large number of factors, heavy-tails, high noise, and underlying complexity; even given the bids for each producer and consumer, solving for the prices across a power network taking into account transmission, energy balance, and AC power flow constraints relies on an NP-hard mixed-integer programming problem that can take hours to solve. Of course in reality we don't even know the bids ahead of time, and we still haven't won the battle against the heavy tails today!
Our pilot experiences with a major East Coast utility looking to trade power, a major New England utility managing their demand response program, a battery storage operator in Texas, and a wind trader in Texas, have shown us that every participant has differing needs for their particular asset collection, so we dedicate time to each of our customers to make sure that the product is tailored to their needs. Along the way we've developed a generic forecasting system tuned for power markets to speed up customization, but we know we have a long way to go before we support the full range of forecast granularity, location, range, risk metrics etc we've heard interest for. With over 3000 market participants operating in open electricity markets (including Texas, California, New York, New England, and the mid-Atlantic), we’re hoping to hit 7 figure revenue within the next year.
We need huge amounts of storage to facilitate a transition to zero carbon grids long term, so we hope to minimize risk and maximize the reward for building new storage assets.
We’d love to hear your thoughts, questions, and comments!
43 comments
[ 3.9 ms ] story [ 48.0 ms ] threadBest of luck!
PS: I'm from Argentina and I've also made for-life friends playing Ultimate :)
How would you balance maintaining an open source presence with maintaining a technological advantage based on our ML techniques in the space? We've put a lot of effort into finding modelling approaches that work in the high-noise, heavy-tailed, limited data environment, so we feel we can't just give them away.
P.S. Yeah, its the best! And for some reason it seems to attract all the nerds :)
A quality problem to be solving.
Would like to see a real world detailed walkthrough of how your product is solving a problem with one of those pilot customers and +1 to open source per previous comment.
Perhaps we could show one of our forecasts, and implied battery/turbine/etc dispatch plan, but lagged a day to avoid annoying the customer of said forecasts? We could even prove we had them earlier by first publishing a hash ;).
We haven't looked outside the US and Europe, anywhere you'd recommend?
We have an interconnected network covering most of our population, a wholesale spot market, high penetration of rooftop solar, infeed tariffs, subsidized household storage, and a network that is suffering growth issues in terms of accomodating the generation and storage facilities coming on line.
Our various states are pushing for net-zero generation in the next 5-10 years, while our Federal government is still subsdizing fossil fuel generation.
How do you deal with customers who, on the one hand, generate most of their revenue from the honest work of doing A for person B, while on the other hand, generate profits from completely esoteric financial and political engineering?
The kind of small energy companies that answer the phone when a startup calls.
Presumably if you had an idea for how to do the forecasts profitably you'd just become an energy futures trader yourself.
I don't know, I'm not trying to poo-poo your thing. But the ESG / Biden green energy plan / renewables stuff your people are excited about, you make a lot more profit just by showing up and harvesting the incentives than you do like, doing a good job.
Berkshire did this (https://www.bloomberg.com/opinion/articles/2019-06-04/warren...) and it's funny, those people were a scam. That's what I'm trying to say. Profits are pretty dodgy in the energy industry.
I like this teeny tiny example though, Active Power, that became a Net Operating Loss shell (also common for biotech), because energy has an easier time booking losses as R&D expenses (https://seekingalpha.com/article/4126904-p10-industries-purc...).
I don't know. There are so, so many examples of hilarious bad actors, from which the profits for small energy businesses come. You're not making forecasts for Exxon, you're doing it for little tiny energy providers. How, really, are they profitable at all?
Now, if we could get enough interest to make such a short-term futures market and trade on it directly... we'd love to!
So make sure that whatever you are getting into is a true need that more than one potential customer actually has, rather than basing things on assumptions alone (customer validation of the proposition).
but you're right, some folks we've approached have built large in-house teams
I was thinking of doing my own system too so love what you guys have done. I'm currently just working on my own side projects so open to opportunities if you guys might have some? :)
Most consumers don't have access to wholesale, fluctuating rates, so we're not sure how to effectively reach them with something beneficial. If home responsive demand such as car charging or smart appliances become much more widespread, we'd definitely want to find a way to manage those as well; I guess we'd have to go through their utilities but unsure.
The market is managed by AEMO and Victoria has a deregulated environment, separated into generators, distributors, and retailers.
I recently signed up with amberelectric.com.au which passes through the wholesale market price (set every 30m) and a fixed $15/month charge.
It's been interesting to watch the market in semi-real time and I would love to see an API that I could adopt/adapt to allow a household to participate in the market.
South Australia has an experimental "virtual generator" that is managed with rooftop solar on each of its public housing estates, combined with local battery storage.
One area that is not clear is that the grid is being stressed by the number of new generation and storage sources, so AI/ML work in modelling that environment is very important in Australia.
Sounds like Griddy in Texas. A good business idea, but they failed due to counterparty risk in the recent storm: their customers failed to pay their enormous electric bills, they couldn't cut them off due to regulations on extreme weather events, and so they got left with a huge bill.
How does your provider handle price spikes? Can you set a price cutoff at which you no longer buy?
Do you think there are benefits to making that 1 minute blocks, or even 1 second blocks now that all trading is automated?
It would have the benefit that solar producers no longer have to guess how many clouds there will be in an hour (and waste energy or money if they're wrong). It would mean big companies don't need to decide an hour in advance of switching on or off their equipment. It would mean pricing can represent fine-grained demand, such as surges exactly on the hour as scheduled things turn on.
Now, one could try to replace the existing reserve mechanisms with the shorter-scale market you describe, but given the resistance to change in the grid operators, I think this is a difficult fight.