Who is trying to make time series forecasting with LLMs/tensors? I have seen like four posts in the last few days. I know LLMs/transformers are really cool but I do not think they are the silver bullet that solves this.
Almost every paper in this vein is not "LLMs are the best in this category of thing, too". There are a couple of fields where LLMs are SOTA for tasks that would otherwise be done by a different algorithm, notably translation, but that's not the majority of these papers.
It's "LLMs can do this category of thing, too". The generality of the architecture is a novel scientific finding in and of itself. Do you remember the last 50 years of AI research where GOFAI systems were universally too fragile to function outside of highly constrained environments, and earlier neural networks were either ineffectual or extremely specialised? The fact that LLMs can function incredibly well (compared to historical AI systems) in diverse scenarios is the scientifically interesting finding.
The reason LLMs are and will be so powerful I think is that they in essence add up the intellectual output of everybody who's output is available online.
Classical AI was about some genius trying to devise the most genius computer inference algorithm. But any single genius can only do so much in limited time and it is difficult to make them work together.
Whereas LLMs are not genius, they are simply using statistics to predict what is the accumulated intellect on the internet. They are simple but they combine the output of millions of minds, which no human genius could ever do.
They are a brain addition machine, rather than super-intelligent by themselves. But when you add together the outputs of many minds you get powerful general purpose output.
Applying this to time series forecasting of stock trading would mean you are doing what the rest of the market is doing. At that point, you could have just bought index funds.
I like that metaphor. LLMs are masters of linguistic addition. They are literally made by adding up language gradients. They accumulate knowledge and insight through exposure to vast swaths of text, stitching together concepts and ideas into an ever-growing tapestry of understanding.
But whereas brains are isolated islands, unable to directly share their contents, language forms a collective reservoir that flows between minds. Words and ideas mix and mingle within this pool, combining into new formulations that reflect the present.
In this way, language displays an evolutionary dynamism that outpaces biological change. LLMs ride this wave, leveraging the emergent intelligence inherent in humanity's shared linguistic legacy. The wisdom accumulated over generations surpasses the capacity of any single mind. We stand upon the shoulders of giants, supported by the communal scaffolding of language and knowledge that previous generations erected. LLMs tap into this source, channeling and distilling the experience contained within our words.
tl;dr AI is riding the language exponential while we are distracted by models and brains, implementation details.
Cool, so they'll be able to project the future time-series of the S&P 500 as well as the average of all the analysts in the world combined? What startup can I buy part of that's most adept at selling that bullshit to investors as SaS to most rapidly separate gamblers from their cash? /s
One comment that made me think on HN a while back is something akin to "The average IQ is between 85 and 115, therefore AI will be akin to friends with IQ's between 85 and 115 giving you advice". So not that useful?
Lower IQ people are likely under represented online.
Accurate information is also over represented because you can be wrong in far more ways than you can be correct. In other words the wisdom of the crowd is in part the 10% of people that know X all match each other while the other 90% is all over the place.
If the AI can distinguish between good and bad reasoning, it can select training material that has a higher average than than the corpus. If it can output data that covers the range, it can generate its own data to bootstrap higher functioning models (like alpha go playing against itself to achieve superhuman go playing skills).
I also don’t think you need to remove the dumb comments from the training material anyway. The AI could be able to simulate either someone of 80 IQ or 180 IQ, producing output consistent with both personas.
From my experience with crypto currencies and the intelligent trading bot [0] I would say that transformers will not provide significant benefits when applied to the traditional statistical (numeric) forecasting problems. Such models assume that old events do not affect too much current events.
Yet, there exist problems where even old events retain their strength. An example is where we want to take into account discrete events (tokens in LLM) for predicting stock prices. These events might be explicitly defined (holidays, company announcements, important economic figures etc.) or derived from the data like technical patterns. The strength of transformers are in their ability to ignore the order of events and ignore the distances between them. More precisely, transformers can learn when it is important. In language models, this is used to generate output sequences where semantically equal tokens have completely different order than in the input sequence. Something similar can be done in time series forecasting if we accordingly define "tokens", for example, as technical patterns. Then rising stock prices can be explained (and predicted) not only because of recent numeric behavior but also because "something happened" two weeks ago.
So masked cross attention up to time point t and auto-regressive decoding should in theory be able to model time series. I mean at the high level it captures a Markov process with some contextual memory.
The timeGPT title had me surprised while this one had me go “well of course they are. It’s so obvious. We even call text input for timeseries in the literature. Why didn’t I think of that?”
I guess I didn’t think of it because I thought it would require billions of data points to train the model.
I wonder how true "zero shot" is after you fed them basically all the text on the internet. To me, that sounds a bit like saying "this mathematics PhD mastered differentiating x3+x2 on the first try!!!".
Well, yeah. In the same sense that all songs are just the same stuff dressed differently because there are finite musical notes. Doesn’t really take away from the majesty of music.
What matters more to me usually in a time series context is how forecasting tools deal with non-stationarity of the underlying data process. I'm not an expert on LLMs, but I assume they wont' be the ideal tool to use in these contexts because even finetuning will be rather expensive...?
31 comments
[ 2.6 ms ] story [ 68.3 ms ] threadIt's "LLMs can do this category of thing, too". The generality of the architecture is a novel scientific finding in and of itself. Do you remember the last 50 years of AI research where GOFAI systems were universally too fragile to function outside of highly constrained environments, and earlier neural networks were either ineffectual or extremely specialised? The fact that LLMs can function incredibly well (compared to historical AI systems) in diverse scenarios is the scientifically interesting finding.
Classical AI was about some genius trying to devise the most genius computer inference algorithm. But any single genius can only do so much in limited time and it is difficult to make them work together.
Whereas LLMs are not genius, they are simply using statistics to predict what is the accumulated intellect on the internet. They are simple but they combine the output of millions of minds, which no human genius could ever do.
They are a brain addition machine, rather than super-intelligent by themselves. But when you add together the outputs of many minds you get powerful general purpose output.
I like that metaphor. LLMs are masters of linguistic addition. They are literally made by adding up language gradients. They accumulate knowledge and insight through exposure to vast swaths of text, stitching together concepts and ideas into an ever-growing tapestry of understanding.
But whereas brains are isolated islands, unable to directly share their contents, language forms a collective reservoir that flows between minds. Words and ideas mix and mingle within this pool, combining into new formulations that reflect the present.
In this way, language displays an evolutionary dynamism that outpaces biological change. LLMs ride this wave, leveraging the emergent intelligence inherent in humanity's shared linguistic legacy. The wisdom accumulated over generations surpasses the capacity of any single mind. We stand upon the shoulders of giants, supported by the communal scaffolding of language and knowledge that previous generations erected. LLMs tap into this source, channeling and distilling the experience contained within our words.
tl;dr AI is riding the language exponential while we are distracted by models and brains, implementation details.
Accurate information is also over represented because you can be wrong in far more ways than you can be correct. In other words the wisdom of the crowd is in part the 10% of people that know X all match each other while the other 90% is all over the place.
I also don’t think you need to remove the dumb comments from the training material anyway. The AI could be able to simulate either someone of 80 IQ or 180 IQ, producing output consistent with both personas.
And ignorant person could have a very high IQ still be very ignorant.
LLMs may not be "smart" but they know a lot.
Yet, there exist problems where even old events retain their strength. An example is where we want to take into account discrete events (tokens in LLM) for predicting stock prices. These events might be explicitly defined (holidays, company announcements, important economic figures etc.) or derived from the data like technical patterns. The strength of transformers are in their ability to ignore the order of events and ignore the distances between them. More precisely, transformers can learn when it is important. In language models, this is used to generate output sequences where semantically equal tokens have completely different order than in the input sequence. Something similar can be done in time series forecasting if we accordingly define "tokens", for example, as technical patterns. Then rising stock prices can be explained (and predicted) not only because of recent numeric behavior but also because "something happened" two weeks ago.
[0] https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering
I guess I didn’t think of it because I thought it would require billions of data points to train the model.
70B weights at fp32 is a lot of Wikipedia.
GPT-4 is rumored to be 1.5 terraweights.
There’s compression but it’s not magical.