A better title would be 'Quantopian wants to turn stock trading algorithmic for everyday investors'. These techniques are used by quants already (as the article states).
"Backtest your algorithm, for free, against our 11-year history"
Hahahaha, yeah, I don't think so. If you guys are really interested in this sort of thing, the MOST important thing is to have a comprehensive database free of biases and errors. For example, survivorship bias is a huge deal, and I wouldn't trust any platform that isn't serious enough to have a historical dataset that is longer than 11 years long to be free of these kinds of things.
> The data includes all companies that were traded, including companies that have subsequently gone out of business. This is very important in order to avoid survivor bias. Without this complete data set, your algorithms would be be blind to the possibility of bankruptcy and the resulting losses.
Good to see at least that. Personally, 11 years is not enough time to see enough things to Black Monday or the tech crash. Hell, it's not enough time to see a REAL bear monster bear market. Also, what else is included in the database? So far, it just seems that prices are the only thing included. I looked through their documentation, and I can't find things like valuations and accounting data.
It's not perfect, but I don't think it's useless. Historical fundamental data are difficult to come by, the only public (not just internally used at a fund) source I know of is http://www.globalfinancialdata.com/Databases/USStocksDatabas... and it costs a lot. The vast majority of algorithmic equity trading uses just pricing though.
Why is this? Why is historical data so hard to find, when in fact the stock market is one of the most computer-heavy operations? These are numerical facts, so I don't think copyright applies either. It just surprises me that the data is so inaccessible.
The "numerical facts" as you say are not copyrighted, but every data vendor I've seen does explicitly prohibit you from making their data available to anybody outside of your license. The more inaccessible the data is, the better their corner on the market, the more they can charge. In similar fashion, real-time tick data is far more expensive to come by than EOD daily data. Real-time data requires slightly more computational overhead (transmission, storage, etc.) but really they just charge much more for it because they can.
In any case, I believe the scarcity the parent was referring to was for the fundamental data, i.e. earnings, value/share, dividend yields, cash on hand, etc. This data is harder to come by compared to the standard open-high-low-close pricing data, which is what most algorithms use (although not necessarily the best algorithms ;-)
I understand that the limited amount of data can make for some easy false positives on trading algorithms, but how relevant do you really think market movements from over 11 years ago really apply to today?
I hear a lot of people talk about needing more than 10yr of data for backtesting, and this is something that is often overlooked. The driving forces behind market data are unobserved and extremely complex, so you simply can't control for all the variables. If you can control for at least most of them, then you are in a situation where more data is NOT always better.
The one elephant that jumps to mind is liquidity. The mechanisms by which liquidity in the markets affect the end results are absolutely key to how your algorithm's interactions with the data will pan out. With that in mind, the methods by which we come by liquidity have dramatically transformed over the years, and market makers from >10 years ago operated in an entirely different way. That's my $0.02, anyhow.
Anecdotal evidence: I have first-hard experience implementing algorithms that chugged along just fine with 10 years of data for backtesting.
Exactly. After Reg NMS and other massive regulatory changes, it's an open question as to whether the post-2007 market system is the same as it was before.
You could ask the same question about many different regulatory changes, but Reg NMS is likely to be the biggest.
Daily data from 10+ years is still relevant,intraday I have found less so,probably due to algo trading/market making reducing the time horizon of momentum based strats and the change to the one cent minimum tick size ( if we are talking about US markets)
11 years just stops short of the chaos of Feb 2000, where the NASDAQ lost 50% of value over the course of the year. Every dip, and people bought more thinking it was the bottom; and then lost it all.
So, is this an effort by more sophisticated players to plant the seeds for a new market to reap? You know, since algorithmic trading is starting to dwindle as a cash cow.
I think you may be mistaken. High-frequency trading is becoming less profitable as an industry, but algorithmic trading in general is most definitely not.
Really? As I understand it the world is quickly becoming saturated and major players are starting to take their chips and leave the table. The risk is quickly catching up to the reward and consolidation into a type of commodity equilibrium is inevitable.
What areas of algorithmic trading are escaping the Great Chill?
"Algorithmic trading" is just a broad umbrella that encompasses any trading system where a computer algorithm is making the trades based upon pre-defined rules. Buying the S&P every Tuesday can be considered "algorithmic trading."
And of course, the answer to your question about which algorithms are working, they are the ones on the opposite side of the trades of the ones that are losing. (Or, alternatively, the ones you can't read about in academic papers or books.)
I am not talking about 'warehouse picking order' style algorithmic trades that are designed to make transactions cheaper or more automated merely for convenience.
Equilibrium is coming.
But that is Ok. Quants can go into material science and model new and useful materials by studying their higher dimensional black hole equivalents [1].
So yields, returns, and inflation are finally stabilizing as the market matures and approaches its "true value"? Nope, because equilibrium is a concept which works well in the natural sciences, but quite erroneous as applied in economics. This is George Soros's argument in his books about the Principle of Reflexivity, which differentiates social systems from natural ones.
What is happening, is merely that trading volume/volatility has tapered off since the '08 crisis. That's expected; empirically, volatility continues to be clustered and autoregressive (a self-exciting process). Its a cycle that will pick up again (one can expect), eventually.
But I am not talking about the level of social systems. Of course the market itself is not approaching equilibrium. But the liquidity enhancement market is. There will always be money to be made there but not the bonanza that's existed up to about two years ago.
But that's mostly due to the much lower volume and volatility (rather than more competition). When its low, so is the demand for liquidity provision. But that demand will come back with the next volatility cycle.
Actually, volatility is only clustered when measured in calendar time. Its much more constant when measured in event/transaction time (plot price against cumulative volume, rather than date/time).
I'm going to say a couple of things about algorithmic trading and the knee jerk reactionaries.
First--there is an industry that makes pretty good money from practicing algorithmic trading. I work with some of them. Some of them do well. However, on the average, returns in that business are subpar. Then again, on the average, most people are idiots. On the average, most startups fail. "On the average" is usually not taken as a reason to not do something by this community (HN), at least allegedly.
Second--there are plenty of reasonable issues to raise whenever this comes up. For instance--your average investor is going to have a tough time of this--for starters due to transaction costs, taxes, etc. Then again, your average investor is going to have a tough time of the stock market in general. Again, the average, by definition, is not impressive and can never do any better than--well--average. But we don't want to be in the average. That's the whole point of much of what we do here. Except when it comes to exploiting financial markets, in which case the average is all that matters?
Point being--is it possible to have a reasonable discussion about statistical analysis of markets and what, if anything, might be done to exploit any regularities found? Is it really so difficult to believe that regularities in relative price movements exist? And yeah, I get it--there's something generally slimy about the whole industry, and there have been plenty of hucksters over the years (and in the present). Then again... isn't this true for the entire tech industry, more or less?
So, you don't have to take part in it, you don't have to learn about it, but can we PLEASE have a better response than "oh, backtesting, that doesn't work, dur"? I mean, yes, there's a fair amount of "lead into gold" type stuff there--but I really don't think it's an unbelievable stretch of the mind to say that regularities exist in markets. No doubt those regularities change, but wouldn't it be interesting to see how quickly and why? And maybe some people can be better at it than others (that doesn't happen anywhere else right?)
So yeah, don't get too excited, but then again, don't be so damned pessimistic. Markets are a system like any other, probably some of the most interesting systems around because they are so hard (some say impossible) to grasp.
I dunno about average algorithmic traders being idiots. I think people have to have a certain level of smartness to even grasp what algorithmic trading is, let along attempt it. If the average return is subpar, then that's out of the people that are smart enough to be in the business, grasp the concept, and try it enough to qualify for whoever's doing the measuring.
My issue with the appeal of the this is that it targets people who are smart in one domain. So they are above average in say programming (everyone is above average, of course, right? ... ;-) ). So it makes them think, ah yes, I am smart, I do algorithms, I should be good at this.
You see, the real market is not the stock market (which I assume is already exploited at small and large scale for all conceivable regularities by now), the real market is in creating novel trading platform.
Kind of like the money is not attending a "how to get rich seminar" but in inventing, promoting and running "get rich seminars".
> So, you don't have to take part in it, you don't have to learn about it, but can we PLEASE have a better response than
In other areas perhaps that is the right approach, but I personally think when it comes to the stock market the position that there are profitably exploitable regularities by your average (or even slightly above average) individuals should be questioned.
Another anecdotal example, this is like the gold rush. Except at the tail end. When all the large chunks of gold has been mined but people are still coming. The best thing is to start selling food and tools to those people.
Honestly the only type of algorithmic trading I would trust is algorithmic trading based at least partially on fundamental data, of which Quantopian has none.
Good luck to all you momentum/technical types though, I'll be sitting in the corner reading 10k's.
I'm not aware of any quantopian algorithms that make use of fundamentals, but that doesn't mean they aren't there. There are lots of fundamentals that are available as processed variables (P/E ratios, etc.) which can be used as an indicators of financial health. Now, certainly that doesn't replace roping off an afternoon and diving into the last 5 years of 10K's to get a real understanding of what's happening with that company, but it doesn't mean that algorithmic trading can't include fundamentals. Indeed, considerable effort has been expended trying to intelligently scrape+process 10K's, which are all available from the SEC, in an (semi-)automated way.
Also, as a point aside from the one above, in the ideal world markets would be driven entirely by the real value underlying the securities being traded. But that's not the case. Markets are highly influenced by news, to give one example. So to say that only the fundamentals can be trusted is a little bit narrow minded (in the pure sense of the phrase, not meant as a shot at you). I think that particularly as the time horizons of your positions get longer, it becomes more and more critical to include fundamental data in your analysis.
I never said algorithmic trading can't include fundamentals, quite the opposite, I just said that until Quantopian includes fundamental data, it's not all that interesting to me. P/E barely scratches the surface, I'm talking about all the data revealed in standard financial statements. If I were ever to trust my money to a security selection algorithm, it would have to analyze more than just price movements.
Honestly the only type of algorithmic trading I would trust is algorithmic trading based at least partially on fundamental data, of which Quantopian has none.
35 comments
[ 0.21 ms ] story [ 89.6 ms ] threadHahahaha, yeah, I don't think so. If you guys are really interested in this sort of thing, the MOST important thing is to have a comprehensive database free of biases and errors. For example, survivorship bias is a huge deal, and I wouldn't trust any platform that isn't serious enough to have a historical dataset that is longer than 11 years long to be free of these kinds of things.
> The data includes all companies that were traded, including companies that have subsequently gone out of business. This is very important in order to avoid survivor bias. Without this complete data set, your algorithms would be be blind to the possibility of bankruptcy and the resulting losses.
In any case, I believe the scarcity the parent was referring to was for the fundamental data, i.e. earnings, value/share, dividend yields, cash on hand, etc. This data is harder to come by compared to the standard open-high-low-close pricing data, which is what most algorithms use (although not necessarily the best algorithms ;-)
The one elephant that jumps to mind is liquidity. The mechanisms by which liquidity in the markets affect the end results are absolutely key to how your algorithm's interactions with the data will pan out. With that in mind, the methods by which we come by liquidity have dramatically transformed over the years, and market makers from >10 years ago operated in an entirely different way. That's my $0.02, anyhow.
Anecdotal evidence: I have first-hard experience implementing algorithms that chugged along just fine with 10 years of data for backtesting.
You could ask the same question about many different regulatory changes, but Reg NMS is likely to be the biggest.
What areas of algorithmic trading are escaping the Great Chill?
And of course, the answer to your question about which algorithms are working, they are the ones on the opposite side of the trades of the ones that are losing. (Or, alternatively, the ones you can't read about in academic papers or books.)
Equilibrium is coming.
But that is Ok. Quants can go into material science and model new and useful materials by studying their higher dimensional black hole equivalents [1].
[1] https://www.simonsfoundation.org/features/science-news/signs...
So yields, returns, and inflation are finally stabilizing as the market matures and approaches its "true value"? Nope, because equilibrium is a concept which works well in the natural sciences, but quite erroneous as applied in economics. This is George Soros's argument in his books about the Principle of Reflexivity, which differentiates social systems from natural ones.
What is happening, is merely that trading volume/volatility has tapered off since the '08 crisis. That's expected; empirically, volatility continues to be clustered and autoregressive (a self-exciting process). Its a cycle that will pick up again (one can expect), eventually.
Actually, volatility is only clustered when measured in calendar time. Its much more constant when measured in event/transaction time (plot price against cumulative volume, rather than date/time).
First--there is an industry that makes pretty good money from practicing algorithmic trading. I work with some of them. Some of them do well. However, on the average, returns in that business are subpar. Then again, on the average, most people are idiots. On the average, most startups fail. "On the average" is usually not taken as a reason to not do something by this community (HN), at least allegedly.
Second--there are plenty of reasonable issues to raise whenever this comes up. For instance--your average investor is going to have a tough time of this--for starters due to transaction costs, taxes, etc. Then again, your average investor is going to have a tough time of the stock market in general. Again, the average, by definition, is not impressive and can never do any better than--well--average. But we don't want to be in the average. That's the whole point of much of what we do here. Except when it comes to exploiting financial markets, in which case the average is all that matters?
Point being--is it possible to have a reasonable discussion about statistical analysis of markets and what, if anything, might be done to exploit any regularities found? Is it really so difficult to believe that regularities in relative price movements exist? And yeah, I get it--there's something generally slimy about the whole industry, and there have been plenty of hucksters over the years (and in the present). Then again... isn't this true for the entire tech industry, more or less?
So, you don't have to take part in it, you don't have to learn about it, but can we PLEASE have a better response than "oh, backtesting, that doesn't work, dur"? I mean, yes, there's a fair amount of "lead into gold" type stuff there--but I really don't think it's an unbelievable stretch of the mind to say that regularities exist in markets. No doubt those regularities change, but wouldn't it be interesting to see how quickly and why? And maybe some people can be better at it than others (that doesn't happen anywhere else right?)
So yeah, don't get too excited, but then again, don't be so damned pessimistic. Markets are a system like any other, probably some of the most interesting systems around because they are so hard (some say impossible) to grasp.
You see, the real market is not the stock market (which I assume is already exploited at small and large scale for all conceivable regularities by now), the real market is in creating novel trading platform.
Kind of like the money is not attending a "how to get rich seminar" but in inventing, promoting and running "get rich seminars".
> So, you don't have to take part in it, you don't have to learn about it, but can we PLEASE have a better response than
In other areas perhaps that is the right approach, but I personally think when it comes to the stock market the position that there are profitably exploitable regularities by your average (or even slightly above average) individuals should be questioned.
Another anecdotal example, this is like the gold rush. Except at the tail end. When all the large chunks of gold has been mined but people are still coming. The best thing is to start selling food and tools to those people.
Good luck to all you momentum/technical types though, I'll be sitting in the corner reading 10k's.
Also, as a point aside from the one above, in the ideal world markets would be driven entirely by the real value underlying the securities being traded. But that's not the case. Markets are highly influenced by news, to give one example. So to say that only the fundamentals can be trusted is a little bit narrow minded (in the pure sense of the phrase, not meant as a shot at you). I think that particularly as the time horizons of your positions get longer, it becomes more and more critical to include fundamental data in your analysis.
Why fundamental data?