Yes, it’s hard to predict markets. Because anybody who can successfully predict markets, does so, makes money, and changes the market so their predictions lose their edge.
Time series forecasts are a lot easier if you are forecasting, say, disk use in your servers or whatnot. (By “easy” I mean you can do a simple prediction and get useful insights.)
The question "what probability distribution generated this" is very hard to answer if you're only getting short window before the distribution changes. A lot of distributions could generate relatively "short" set dispersed data points and deciding which distribution is "really" creating the points might not even be meaningful. A distribution is a mathematical device, not something with a physical existence.
This article focuses mostly on point forecasting. There are many other things that are interesting to forecast.
Consider for example the use case of forecasting the average speed on a road segment with a maximum speed of 70mph. Forecasting whether that will be 69.8 or 70.3 is not very relevant. What is relevant is forecasting when the speed drops below a traffic jam threshold. But the exact timing of that might be impossible to forecast due to the inherently chaotic behavior of traffic. Forecasting the probability of a traffic jam occurring may be more interesting to practitioners.
Indeed. For example, volatility is of interest in financial data, because it relates to the pricing of risk. So are the correlations between asset classes.
This corresponds to a restatement of Murphy's Law, namely "life is a bitch and then you die". When you most need a diversified portfolio, diversification is hardest to achieve.
The examples in the posts suggest that the past does not contain all the patterns, or information in general, about the future. If so, isn't it natural that point forecast will fail in some cases?
> Given the lack of forecasting signal, the obvious next step then is to seek out external (exogenous) features in the real world that can help prediction models.
this seems right to me. maybe another interesting approach would be a fusion llm+ts model that does multiple-input-single-output with input metadata and causality narrative. so it "thinks" about what data it has and how predictive it may be of the target variable and when something "interesting" occurs it uses the big priors to synthesize a good guess at what it would look like.
so you'd have something like the time series data plus textual narratives of the causality stories as the training data.
There's a couple "AI forecaster" companies now, Preseen and FutureSearch. My understanding is you give them a query and they research every possible relevant factor and give an estimate of the likelihood of your query, and supposedly they're comparable to if you asked a human to do the same thing, but they're new enough that they're not really proven yet.
A very interesting article, suzyahyah. I especially appreciated your definition of stationarity, a concept with which I struggled in my own time series class. If I understand correctly, it sounds like the basic premise is that a fundamentally statistical methodology (LLMs) can't realistically predict a non-stationary data generation, which makes sense.
Separately, I've wondered for some time if there might be some reliable way to predict non-stationary data. While I don't have the answer, it occurs to me that it will possibly be a non-statistical method due to the fundamental incompatibilities. However, it also occurs to me that, given enough information, every data-generating process actually could be predicted. For instance, in the stock example, if you could model every single input into the system of a single company's stock, including every variable affecting every human that might conduct a transaction of it (daunting and unrealistic as that might be, but this is a thought experiment), then I believe the problem of prediction stops being non-stationary and in fact becomes completely deterministic, if complex. In such a scenario, wouldn't you be able to accurately make your prediction? I believe that perhaps chaos theory could present us with some solutions here where pure statistics (or, rather, simple statistics) cannot.
> Given the lack of forecasting signal, the obvious next step then is to seek out external (exogenous) features in the real world that can help prediction models. After all many real world time series are event-driven (FX, bitcoin) and hence they are exposed to shocks and drifts which can also be measured or accounted for in the data generating process.
This is the approach I'm using at my job, which is incident detection with customer metrics. We're tagging our time series data with common features -- such as country, customer type, etc -- with the idea that we can do a graph-like search to find exogenous variables. We can also use this to identify time series that have a similar "data generating process" and are simply different "realizations" of each other.
We don't need great time series forecasts, just something that detects large deviations quickly. We can then add in an existing dataset of _known_ incidents, indexed by the same common features, as a training/validation set.
I was a little surprised to find someone experienced in time series modeling not mentioning bayesian modeling nor the use of ensemble models... but I do think its more fundamental points were worth making.
Again and again nerds coming from maths/CS/etc to finance are surprised to find that financial time series are actually impossible to predict. After spending almost a decade in finance now with a similar background, the arrogance of the "let's just throw in some neural network/whatever and be done with it" attitude now amuses me. After all, if it was easy - or even possible with any kind of effort - to predict (even within some error margins or probability) what the stock prices are tomorrow, anyone doing it would quickly become a billionaire or trillionaire. If the person could keep doing it, at some point they would own enough of the stock market so that their actions would affect the prices and whatever pattern they found would vanish.
Financial markets are not a natural phenomenon that exist unchanged regardless of whoever is observing them. Their dynamics continuously change in response to collective actions of all of the humanity.
Oil price changed quite a bit when the US attacked Iran. If you are trying to predict the price of oil, your model would have to be able to predict Trump ordering an attack on Iran. Does your model include a full simulation of the mind of the president of the United States (and every other person who have any kind of impact on the world events)? If not, then your time series forecasts are not going to be that great.
Well. Lots of math that boils down to 'predicting the future is hard'. Especially when the future is one of social construction, that's what gets lost a bit here. Predicting the future is easier for planetary motions than for Bitcoin.
I remember in 2017, it was said that "regular" machine learning techniques were better for time series than neural networks. It seems like this blog post is indirectly references that belief and gives. It seems like the idea is simpler ML can be tuned to a given time series whereas neural network require lots of training.
Also, one method works better than another is more meaningful than "predicting the future is hard".
I have to explain this to managers, execs and stakeholders all the time. ML models work great for systems where the rules/dynamics do not change over time.
With Forecasting, in a lot of domains where you want a forecast, everything is subject to change - laws, policies, regs, customers appetites, competitors behaviours, etc.
I highly relate to this. But I always had a hard time to explain this knowledge that is in an engaging and non-jargon way. How do you do it usually? do you have a script/slides/conversation transcript (can be just your vague memory of it) you can share to help me on this
> I’ve been thinking recently about what makes time series forecasting problems so difficult compared to other sequence learning tasks
Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t).
In plenty of other machine learning and statistical modeling tasks this is not the case. You can train on every token you'll ever see and every pixel value you'll ever see, you can do regression analysis on every categorical value you include and an least an observation from within a range of every continuous and discrete value you'll observe. But with forecasting you will always have values you predict that are outside the range of anything you trained on.
You would run into similar problems if you tried to create a statistical model of the density of water given a temperature but your training data only included values between 0-100 C and you went out and started predicting values covering all the temperatures found on Earth.
For whatever reason, when time is a variable we somehow think it is immune from the obvious limitation of predicting on values outside of the range of values you trained on.
> Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t).
I don't get this, time is usually not a covariate in ts models, so why is it a challenge?
I don't understand, this is just a sampling problem?
> You can train on every token you'll ever see and every pixel value you'll ever see, you can do regression analysis on every categorical value you include and an least an observation from within a range of every continuous and discrete value you'll observe.
Can you give an example of this? Lets say you are developing DLSS, you don't have the of a game that have not yet been developed.
> But with forecasting you will always have values you predict that are outside the range of anything you trained on.
A time series of my body temperature will only ever range from 20C to 50C. Outside of that range, I have bigger problems than my prediction being wrong.
36 comments
[ 3.4 ms ] story [ 40.5 ms ] threadYes, it’s hard to predict markets. Because anybody who can successfully predict markets, does so, makes money, and changes the market so their predictions lose their edge.
Time series forecasts are a lot easier if you are forecasting, say, disk use in your servers or whatnot. (By “easy” I mean you can do a simple prediction and get useful insights.)
Consider for example the use case of forecasting the average speed on a road segment with a maximum speed of 70mph. Forecasting whether that will be 69.8 or 70.3 is not very relevant. What is relevant is forecasting when the speed drops below a traffic jam threshold. But the exact timing of that might be impossible to forecast due to the inherently chaotic behavior of traffic. Forecasting the probability of a traffic jam occurring may be more interesting to practitioners.
Interestingly market volatility often goes hand-in-hand with increased correlation between asset prices: https://en.wikipedia.org/wiki/Anna_Karenina_principle#Order_...
This corresponds to a restatement of Murphy's Law, namely "life is a bitch and then you die". When you most need a diversified portfolio, diversification is hardest to achieve.
this seems right to me. maybe another interesting approach would be a fusion llm+ts model that does multiple-input-single-output with input metadata and causality narrative. so it "thinks" about what data it has and how predictive it may be of the target variable and when something "interesting" occurs it uses the big priors to synthesize a good guess at what it would look like.
so you'd have something like the time series data plus textual narratives of the causality stories as the training data.
Separately, I've wondered for some time if there might be some reliable way to predict non-stationary data. While I don't have the answer, it occurs to me that it will possibly be a non-statistical method due to the fundamental incompatibilities. However, it also occurs to me that, given enough information, every data-generating process actually could be predicted. For instance, in the stock example, if you could model every single input into the system of a single company's stock, including every variable affecting every human that might conduct a transaction of it (daunting and unrealistic as that might be, but this is a thought experiment), then I believe the problem of prediction stops being non-stationary and in fact becomes completely deterministic, if complex. In such a scenario, wouldn't you be able to accurately make your prediction? I believe that perhaps chaos theory could present us with some solutions here where pure statistics (or, rather, simple statistics) cannot.
Just my 2 cents..
This is the approach I'm using at my job, which is incident detection with customer metrics. We're tagging our time series data with common features -- such as country, customer type, etc -- with the idea that we can do a graph-like search to find exogenous variables. We can also use this to identify time series that have a similar "data generating process" and are simply different "realizations" of each other.
We don't need great time series forecasts, just something that detects large deviations quickly. We can then add in an existing dataset of _known_ incidents, indexed by the same common features, as a training/validation set.
Financial markets are not a natural phenomenon that exist unchanged regardless of whoever is observing them. Their dynamics continuously change in response to collective actions of all of the humanity.
Oil price changed quite a bit when the US attacked Iran. If you are trying to predict the price of oil, your model would have to be able to predict Trump ordering an attack on Iran. Does your model include a full simulation of the mind of the president of the United States (and every other person who have any kind of impact on the world events)? If not, then your time series forecasts are not going to be that great.
Also, one method works better than another is more meaningful than "predicting the future is hard".
Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t).
In plenty of other machine learning and statistical modeling tasks this is not the case. You can train on every token you'll ever see and every pixel value you'll ever see, you can do regression analysis on every categorical value you include and an least an observation from within a range of every continuous and discrete value you'll observe. But with forecasting you will always have values you predict that are outside the range of anything you trained on.
You would run into similar problems if you tried to create a statistical model of the density of water given a temperature but your training data only included values between 0-100 C and you went out and started predicting values covering all the temperatures found on Earth.
For whatever reason, when time is a variable we somehow think it is immune from the obvious limitation of predicting on values outside of the range of values you trained on.
I don't get this, time is usually not a covariate in ts models, so why is it a challenge?
> You can train on every token you'll ever see and every pixel value you'll ever see, you can do regression analysis on every categorical value you include and an least an observation from within a range of every continuous and discrete value you'll observe.
Can you give an example of this? Lets say you are developing DLSS, you don't have the of a game that have not yet been developed.
> But with forecasting you will always have values you predict that are outside the range of anything you trained on.
A time series of my body temperature will only ever range from 20C to 50C. Outside of that range, I have bigger problems than my prediction being wrong.