"Share-price time series going back decades still contain far less information than, say, the image data used to train Facebook’s facial-recognition algorithms."
What?
Clearly these ML models aren't trying hard enough. Correlating stock movements with the weather, geo populations, ... there are potentially infinite patterns buried in the data that a hedge fund could uncover.
I can pay for high resolution images of every walMart in the US-
Use math- estimate foot traffic- estimate average sales per person- I can estimate profitability of Walmart. Let's say it's a good quarter. I buy a bunch of Walmart stock - they then disclose quarterly earnings- investors see the good numbers- the share price goes up. My position is long-I sell. Profit.
edit: obviously many steps of this process can use machine learning in general and deep learning in particular
no you can't. take the absolute highest resolution images you can buy, and take 100 of the best mathematicians and using only maps you won't even be able to guess within 20%...analysts can already estimate within 1% of expected earnings
> take the absolute highest resolution images you can buy, and take 100 of the best mathematicians and using only maps you won't even be able to guess within 20%...
Wouldn’t it be easier to create an incentive for customers to directly give you data about where they shop? Like make an app that pulls up coupons specifically for Walmart..
i think some firms buy analyses of imagery to predict oil market movements by looking at whether oil storage containers are empty or full. could probably imagine doing similar things in other commodities markets (corn etc)
There are about 30,000 institutionally-investable stocks in the world. They have ~252 closing prices per year. This is the thing you're trying to predict, so it forms the outputs of your training set (be sure to hold some back for out-of-sample testing and there are probably distinct regimes you need to switch between). Yes, there may be lots of features (weather, geo populations, etc., as you point out), but you're often hard up against the curse of dimensionality here. Sure, intraday data is more voluminous, but there are fewer economically reasonable influences on that.
That only adds a few dozen items. There's only so many currencies and commods, bonds. Even if you include really illiquid stuff maybe you get 150 things.
> There's only so many currencies and commods, bonds
Currencies are traded in pairs. Bonds are numerous, if illiquid off the run. Not to mention the bursting menagerie of derivatives our species tends to.
Some people are already trying and succeeding. And since a lot of these signals already incorporated into the market prices of various financial instruments, you can apply deep learning techniques to these available prices (and time series) to get similar results, to make buy and sell decisions/algorithms.
Just 32x32 image space is, like, larger the the atoms in the universe. Even per-tick financial data is quite small in comparison.
Or, from another angle, say you've got 10 ticks of a stock per second for 12 hours = 432,000 data points. This is the same as a single (relatively small) 380x380x3 image. If you assume there are roughly 30,000 instruments like this in the world of this volume, that's probably, I dunno, the number of images facebook sees in a second. Those are all guesses but you can see the scale of differences involved here.
1. This is all observational data - even if you have all the features in the domain it's pretty clear that you won't have seen all of the generative theory.
2. You don't have all the features because the features change, renewable power suddenly makes average wind conditions relevant, and so on.
3. This domain is clearly chaotic; you can induce the various attractors and theories, but you cannot measure the conditions well enough to predict the shift between attractors.
There are many people who are using ML to predict closed parts of the financial world where they have an deep and well founded understanding and excellent data. But there are lots and lots of people who are feeding data into models and getting good results and then going on good runs who are going to get a shock.
I worked at a hedge fund for 9 years. Regression models are used for EVERYTHING. Regression is machine learning. This article has a narrow focus and doesnt understand what machine learning is.
Usually « machine learning » is used to talk about things beyond basic statistical methods that predate computers. On the other hand, basic statistics are often good enough...
Other machine learning techniques widely used on wall street for over a decade
- Dimensionality Reduction
- Clustering
- Classification techniques using trees, SVM, bayesian models
I feel like machine learning is one of those terms that has lost all defined meaning, under a bombardment of hype, corporate buzzwordism, and other horseshit.
The article distinguishes between using a regression to generate trade ideas for a human (say, data science), or actually having the machine trade on the idea without intervention (that would be AI).
The article claims the former happens but not the latter. This is entirely compatible with your statements.
I don't think technically sophisticated funds are going to be sharing any details at all for pieces like these. There is bound to be fancy ML in play at hedge funds, though - do not underestimate the lengths to which people will go to get an edge in the markets.
I like the shoutout to Numerai. I think they are so left field that traditional hedge funds don't even have the ability to understand how much better Numerai can be.
Huh? I did my commute with a guy who worked for an automated trading shop when I worked in London in 1994 and he talked constantly about what sounded just like machine learning.
Many quant funds are market neutral. For every dollar long they have a dollar short. Such funds aren’t necessarily benefiting from an upward rise in the market.
A friend of mine, upon reaching a status of a "high-net worth individual", was invited to invest in a hedge fund associated with a bank of regional importance (major for that small country). They demonstrated the best of the tech of the time, and then shifted the tone to, "but we also use more traditional approaches".
Unusually large portion of their "consulting research staff" was female and spread Europe-wide. Not because they were progressive or pursuing diversity agenda; these were call girls. The call girls were paid for timely tips. Even if they only report that a particular top executive stayed overnight far away from home, was in a foul mood, and a couple of top-ranking colleagues were present, this can be used to detect an significant internal event not reported publicly. If the girl was crafty enough to extract more information, even better.
42 comments
[ 145 ms ] story [ 383 ms ] threadWhat?
Clearly these ML models aren't trying hard enough. Correlating stock movements with the weather, geo populations, ... there are potentially infinite patterns buried in the data that a hedge fund could uncover.
edit: obviously many steps of this process can use machine learning in general and deep learning in particular
Why is that?
https://www.cnbc.com/id/38722872
Currencies are traded in pairs. Bonds are numerous, if illiquid off the run. Not to mention the bursting menagerie of derivatives our species tends to.
Or, from another angle, say you've got 10 ticks of a stock per second for 12 hours = 432,000 data points. This is the same as a single (relatively small) 380x380x3 image. If you assume there are roughly 30,000 instruments like this in the world of this volume, that's probably, I dunno, the number of images facebook sees in a second. Those are all guesses but you can see the scale of differences involved here.
1. This is all observational data - even if you have all the features in the domain it's pretty clear that you won't have seen all of the generative theory. 2. You don't have all the features because the features change, renewable power suddenly makes average wind conditions relevant, and so on. 3. This domain is clearly chaotic; you can induce the various attractors and theories, but you cannot measure the conditions well enough to predict the shift between attractors.
There are many people who are using ML to predict closed parts of the financial world where they have an deep and well founded understanding and excellent data. But there are lots and lots of people who are feeding data into models and getting good results and then going on good runs who are going to get a shock.
The article claims the former happens but not the latter. This is entirely compatible with your statements.
[1]: https://www.youtube.com/watch?v=FoYC_8cutb0
A friend of mine, upon reaching a status of a "high-net worth individual", was invited to invest in a hedge fund associated with a bank of regional importance (major for that small country). They demonstrated the best of the tech of the time, and then shifted the tone to, "but we also use more traditional approaches".
Unusually large portion of their "consulting research staff" was female and spread Europe-wide. Not because they were progressive or pursuing diversity agenda; these were call girls. The call girls were paid for timely tips. Even if they only report that a particular top executive stayed overnight far away from home, was in a foul mood, and a couple of top-ranking colleagues were present, this can be used to detect an significant internal event not reported publicly. If the girl was crafty enough to extract more information, even better.