A personal note here is that they could done a better job on the tokens because the announcement was so grandiose and maybe they underestimated people with legit interest.
I’m using all libs from Nixtla with active advocation and I did not have a token; meanwhile lots of guys posting their usages on Twitter.
No, because when every player is using it, they will need something else to give them an edge. Anyways, quants do a lot more than time-series forecasting.
Most quants work on the sell-side where direct forecasting is almost irrelevant (to an academic, a trader perhaps not...), in that they are usually attempting to "interpolate" market prices to be able to price derivatives.
> Most quants work on the sell-side where direct forecasting is almost irrelevant
Those aren’t real quants. Even the sell-side quants know they aren’t real quants. For those unfamiliar, sell-side quants typically work at banks like Goldman, HSBC, JPMorgan, etc.
The real quants are buy-side quants/traders: prop shops, hedge funds, endowment/pensions funds, etc.
The M7 forecasting challenge makes this goal explicit. Not the only use of forecasting, and IMO it would be good to have other timeseries data to present to models.
“Some popular models like Prophet [Taylor and Letham, 2018] and ARIMA were excluded from the analysis due to their prohibitive computational requirements and extensive training times.”
Anyone who work a lot in time series forecasting can explain this in some more details?
I’ve def used ARIMA, but only for simple things. Not sure why this would be more expensive to train and run than a Transformer model, and even if true, ARIMA is so ubiquitous that comparing resources & time would be enlightening. Otherwise it just sounds like a sales pitch and throw more obscure acronyms for a bit of “I’m the expert, abc xyz industry letters” marketing.
I immediately tried to find a comparison with ARIMA as well and was disappointed. It's difficult to take this paper seriously when they dismiss a forecasting technique from the 70's because of "extensive training times".
Even then, 500 years of daily data is less than 200k observations, most of which are meaningless for predicting the future. Less than 16B seconds of data. Regression might not handle directly, but linear algebra tricks are still available.
High training times could be cost prohibitive. Currently, its over $100mil to train GPT4 from scratch (which possibly includes other costs related to RLHF and data acquisition). Not sure how this model compares, but its likely not cheap.
If true, then beating it and looking good will be easy.
Having trained ARIMA models in my day, I will say that long training times and training cost -- compared to any deep learning model -- is not something that ever crossed my mind.
While I could find some excuses to exclude ARIMA, notably that in practice you need to input some important priors about your time series (periodicity, refinements for turning points, etc) for it to work decently, "prohibitive compute and extensive training time" are just not applicable.
That part is a bit wanky, but the rest of the paper, notably the zero shot capability, is very interesting if confirmed. I look forward for it to be more accessible than a "contact us" api to compare to ARIMA and others myself
Eh it's not as if you could just project down the 300k time series to something lower dimensional for forecasting. The TimeGPT would have to do something similar to avoid the same problem.
Though I can't quite figure out how the predicting works exactly, they have a lot of test series but do they input all of them simultaneously?
I have need doing time series forecasting professionally. ARIMA is computationally one of the cheapest (both training and inference) forecasting models out there. It suffers from many deficiencies and shortcomings but computational efficiency is not one of them.
We love ARIMAs. That is why we put so much effort into creating fast and scalable Arimas and AutoArima in Python [1].
Regarding your valid concern. There are several reasons for the high computational costs. First, ARIMA and other "statistical" methods are local, so they must train one different model for each time series. (ML and DL models are global, so you have 'one' model for all the series.) Second, the ARIMA model usually performs poorly for a diverse set of time series, like the one considered in our experiments. The AutoARIMA is a better option, but its training time is considerably longer, given the number and length of the series. Also, AutoARIMA tends to be very slow for long series.
In short: for the 500k series we used for benchmarcking, ARIMA would have taken literally weeks and would have been very expensive.
That is why we included many well-performing local "statistical" models, such as the Theta and CES. We used the implementations on our open-source ecosystem for all the baselines, including StatsForecast, MLForecast, and Neuralforecast. We will release a reproducible set of experiments on smaller subsets soon!
> “Some popular models like Prophet [Taylor and Letham, 2018] and ARIMA were excluded from the analysis due to their prohibitive computational requirements and extensive training times.”
Yes, I've done some work in time series forecasting. The above sentence is the one that tipped me off to this paper being BS, so I stopped reading after that. :) I can't take any paper about timeseries forecasting seriously by an author who isn't familiar with the field.
this is my thought as well. reading the paper now.. who would have thought that the quote, "those who don't learn from history is doomed to repeat it" might be useful in predictions.
Time series works amazing until a fundamental assumption breaks or the time frame extends too far. It's pretty much just drawing the existing pattern out further with mathematical precision. It only works till the game changes.
Yeah, that’s part of the game though. You can’t get perfection from modelling a complex system with the (relatively) few variables that you can actually measure. Assumptions are always evolving, and are always going to be broken at some point or another.
That’s the opening line, right? Uncertainty is a fact of life. With time series forecasts, the best you can ever hope to do is give probability bounds, and even then you can only really do so by either:
- limiting by the rules of the game (e.g. the laws of physics, or the rules of a stock exchange)
- using past data
The former is only useful if you’re the most risk averse person on the planet, and the latter is only useful if you are willing to assume the past is relevant.
Good response. People seem to think that what I call “single pass” inference is the only thing that matters - a monolithic single process system
When in fact the world and intelligent agents inside it are ensembles of ensembles of systems with various and changing confidence that flow and adjust as the world does
Out come the fingers made of foam
They finally open-sourced Rehoboam
Our societies have been thusly blessed
What it predicts could have anyone guessed
Targeted advertising wherever you roam
Aren't LLM's already zero shot time series predictors? Predicting next token and forecasting seem like the exact same problem. I will admit some small tweaks in tokenization could help but it seems like we're just pretraining on a different dataset.
One idea I was interested in was after reading the paper on introducing pause tokens[1] was a multimodal architecture that generalizes everything to parallel time series streams of tokens in different modalities. Pause tokens make even more sense in that setup.
I agree - You could frame LLMs this way. Tokens over "time" where time just happens to be represented by discrete, sequential memory.
Each token could encode a specific amplitude for the signal. You could literally just have tokens [0,1,...,MAX_AMPLITUDE] and map your input signal to this range.
In the most extreme case, you could have 2 tokens - zero and one. This is the scheme used in DSD audio. The only tradeoff is that you need way more samples per unit time to represent the same amount of information, but there are probably some elegant perf hacks for having only 2 states to represent per sample.
There are probably a lot of variations on the theme where you can "resample" the input sequences to different token rate vs bits per token arrangements.
141 comments
[ 1.6 ms ] story [ 204 ms ] threadA personal note here is that they could done a better job on the tokens because the announcement was so grandiose and maybe they underestimated people with legit interest.
I’m using all libs from Nixtla with active advocation and I did not have a token; meanwhile lots of guys posting their usages on Twitter.
Inverted Transformers Are Effective for Time Series Forecasting
https://news.ycombinator.com/item?id=37848321
I can appreciate people doing things from the heart, but in that case, it better be more poetic than a BBC documentary.
Is that not what it is at this point ? Probabilities ?
I would have liked to see more detailed benchmarking, such as the electricity, exchange and weather benchmarks.
Those aren’t real quants. Even the sell-side quants know they aren’t real quants. For those unfamiliar, sell-side quants typically work at banks like Goldman, HSBC, JPMorgan, etc.
The real quants are buy-side quants/traders: prop shops, hedge funds, endowment/pensions funds, etc.
Anyone who work a lot in time series forecasting can explain this in some more details?
I’ve def used ARIMA, but only for simple things. Not sure why this would be more expensive to train and run than a Transformer model, and even if true, ARIMA is so ubiquitous that comparing resources & time would be enlightening. Otherwise it just sounds like a sales pitch and throw more obscure acronyms for a bit of “I’m the expert, abc xyz industry letters” marketing.
Having trained ARIMA models in my day, I will say that long training times and training cost -- compared to any deep learning model -- is not something that ever crossed my mind.
While I could find some excuses to exclude ARIMA, notably that in practice you need to input some important priors about your time series (periodicity, refinements for turning points, etc) for it to work decently, "prohibitive compute and extensive training time" are just not applicable.
That part is a bit wanky, but the rest of the paper, notably the zero shot capability, is very interesting if confirmed. I look forward for it to be more accessible than a "contact us" api to compare to ARIMA and others myself
arima, sure
Even if true, they could take a random subset of size 100 out of the 300k and compare on those.
>Even if true, they could take a random subset of size 100 out of the 300k and compare on those.
Sure...but there's a chance that ARIMA won't even finish training on that subset either.
If you write a paper and exclude comparisons to state of the art, this what happens.
They could have done something, and didn’t.
“It’s hard so we didn’t” isn’t an excuse, it’s just a lack of rigor.
Though I can't quite figure out how the predicting works exactly, they have a lot of test series but do they input all of them simultaneously?
EDIT: typos
Regarding your valid concern. There are several reasons for the high computational costs. First, ARIMA and other "statistical" methods are local, so they must train one different model for each time series. (ML and DL models are global, so you have 'one' model for all the series.) Second, the ARIMA model usually performs poorly for a diverse set of time series, like the one considered in our experiments. The AutoARIMA is a better option, but its training time is considerably longer, given the number and length of the series. Also, AutoARIMA tends to be very slow for long series. In short: for the 500k series we used for benchmarcking, ARIMA would have taken literally weeks and would have been very expensive. That is why we included many well-performing local "statistical" models, such as the Theta and CES. We used the implementations on our open-source ecosystem for all the baselines, including StatsForecast, MLForecast, and Neuralforecast. We will release a reproducible set of experiments on smaller subsets soon!
[1] https://nixtla.github.io/statsforecast/docs/models/arima.htm...
Yes, I've done some work in time series forecasting. The above sentence is the one that tipped me off to this paper being BS, so I stopped reading after that. :) I can't take any paper about timeseries forecasting seriously by an author who isn't familiar with the field.
I’m willing to bet 90% of the people who upvoted this did so based on the title alone. Sorry, no time traveling GPT for you.
Purely on rational basis, this, as in predicting time series data, doesn’t seem plausible. But maybe it is.
Completely false.
In reality, there are more public signals than can be exploited.
The market is most definitely NOT efficient.
That’s the opening line, right? Uncertainty is a fact of life. With time series forecasts, the best you can ever hope to do is give probability bounds, and even then you can only really do so by either:
- limiting by the rules of the game (e.g. the laws of physics, or the rules of a stock exchange)
- using past data
The former is only useful if you’re the most risk averse person on the planet, and the latter is only useful if you are willing to assume the past is relevant.
When in fact the world and intelligent agents inside it are ensembles of ensembles of systems with various and changing confidence that flow and adjust as the world does
The zero-shot nature is certainly impressive but it doesn't look like you'd be able to _do_ much with it?
Regardless, need to see more benchmarks to better understand its true performance. If it holds up it would be a big win for time forecasting.
[1] https://www.youtube.com/watch?v=n7luRRyxLoQ
LICENSE Apache-2
https://github.com/Nixtla/statsforecast/blob/main/LICENSE
Mentions ARIMA, ETS, CES, and Theta modeling
One idea I was interested in was after reading the paper on introducing pause tokens[1] was a multimodal architecture that generalizes everything to parallel time series streams of tokens in different modalities. Pause tokens make even more sense in that setup.
1. https://arxiv.org/abs/2310.02226
-Arm chair ML practitioner
Each token could encode a specific amplitude for the signal. You could literally just have tokens [0,1,...,MAX_AMPLITUDE] and map your input signal to this range.
In the most extreme case, you could have 2 tokens - zero and one. This is the scheme used in DSD audio. The only tradeoff is that you need way more samples per unit time to represent the same amount of information, but there are probably some elegant perf hacks for having only 2 states to represent per sample.
There are probably a lot of variations on the theme where you can "resample" the input sequences to different token rate vs bits per token arrangements.