59 comments

[ 2.8 ms ] story [ 119 ms ] thread
(comment deleted)
(comment deleted)
One of Quantopian's studies shows that backtesting poorly predicts live trading results: https://blog.quantopian.com/using-machine-learning-to-predic.... From the forums, it looks like 90% of the effort is devoted to getting a good backtest, a yardstick which might not have any bearing on reality. How do real quant traders deal with this discrepancy?
-Out of sample testing. Across markets, across time. -Logical economic explanations.
Real firms use tick data to backtest, not the 1 minute bars that Quantopian uses, and spend a lot of time simulating network characteristics such as data and order latency. They also use machine learning, which isn't possible on Quantopian, to build models, which requires downloading lots of data (and not just equities). No serious quant will ever use such a tool. You might as well go to the casino.
Edit: tick data is usually collected over time, because data sources have different characteristics, and you should be testing over the same realtime data that you'll be using to trade live. You need to know the bid/ask prices and volumes, in order to know where to place limit orders. Otherwise you are just paying commissions and spreads to brokers and market makers.
(comment deleted)
> How do real quant traders deal with this discrepancy?

A lot of it comes down to access to information and speed. All the major traders and quants have access to what effectively boil down to mailing lists (that their companies pay top dollar for) that have the best analysts in the world producing in-depth analysis on various topics[0]. Also, many/most have access to a Bloomberg terminal[1] to get the Bloomberg new feeds and the AIM clone, which costs $25,000/year/person.

This is why amateur day-trading is silly. You're not competing in a level playing field- they know more than you do.

[0]http://www.investopedia.com/articles/analyst/03/031803.asp [1]https://en.wikipedia.org/wiki/Bloomberg_Terminal

Sounds like I should get into building tools for traders!
It's a good idea. The user base is a group of people for whom hundreds of dollars per month is peanuts if it gives them a 1% edge over their competitors.
There was a great article in the WSJ about quant "factors"

>A former student of Mr. Fama, Cliff Asness, founder of quantitative hedge-fund manager AQR Capital Management, said he tries to avoid being caught out by false findings by trading on anomalies he can explain, economically or through investor behavior. To assess whether the market anomalies will continue, he looks for ones which carried on after being identified, can be seen in other markets or asset classes, and where minor changes to how they are defined don’t much affect the result. These include most famously value, momentum and corporate quality, among others.

https://www.wsj.com/articles/an-algorithm-an-etf-and-an-acad...

A Bayesian mindset. When investigating a signal you have to ask: Is it too good to be true? Does it make economic sense?

Then simulate rolling optimisation and trading to see what returns could have been achieved with knowledge only available at the time.

We are very mindful of overfitting our models, and very skeptical of backtests on their own. Successful strategies often have some fundamental theory about why they should work. If not, the backtest needs to be very robust, and demonstrate that it is not overfit by cross-validation, and other methods.
(comment deleted)
You're right, backtest overfitting is a problem. At Quantopian, we spend a lot of time on education helping people avoid overfitting like this (https://www.quantopian.com/lectures#The-Dangers-of-Overfitti...). It is easy to make a backtest look good by tweaking things here or there and that's just textbook overfitting. If you follow good statistical principles for model development based on a sound economic basis, the backtest is more of a sanity check to make sure that whatever signal you have found survives the introduction of market impact. If you have consistency between the statistical aspects of in-sample performance (your backtest) and true out-of-sample performance (paper trading on entirely new data), there as an increased probability that you have found a viable signal.

Disclaimer, I work at Quantopian.

We've been seeing these quantopian fund stores for years now.

Where are the numbers? I mean at some point you have to produce right?

https://news.ycombinator.com/item?id=12171843#12173142

https://news.ycombinator.com/item?id=12950276#12952666

https://news.ycombinator.com/item?id=12335272#12336086

THe biggest issue I see for a quantopian fund is the "skin in the game" rule. All of the successful fund managers I know have a very unhealthy portion of their net worth in their own funds.

It's not clear to me that the quantopian algo designers will be able to, or be forced to, put their own money into their strategies. I find "skin in the game" to be one of the most promising alpha signals that still persists to date.

> THe biggest issue I see for a quantopian fund is the "skin in the game" rule. All of the successful fund managers I know have a very unhealthy portion of their net worth in their own funds.

I feel like there's a very large survivor bias here. How many managers with unhealthy amounts of money in their funds lost it all?

Having some money in your own funds certainly aligns manager interests with investors. It can, however, also lead to biased decision making by the manager. Not having skin in the game can actually lead to more rational decision making, and long-term success.

I don't have any numbers to back this up (though I think studies have been done), but on the other hand, neither did you :P.

(comment deleted)
I have a very simple algo (like 30 LoC) that is currently performed 9x S&P returns on >100k in initial capital. Granted, this is over a relatively short span of time and it doesn't observe the requirements to be part of Quantopian's contest. However trading live, it is doing significantly better than any back-testing I've done.

There's nothing preventing designers from putting their own money in. Contest winners get profits in addition to whatever they invest themselves, on the condition they share the algorithm (I think closed source?) with Quantopian. That being said, Quantopian probably has enough information to reverse-engineer at least the main ideas of anything you share, given they can observe the orders made and probably also your network requests.

How long has your algorithm been running live, and how many trades has it made?
Months and dozens. No leverage, no shorts. It just goes long and reallocates every couple of weeks. So it hasn't really sustained a serious bear market, but its also never had a drawdown more than 0.5% less than that of the S&P500.
Having algorithms that work in one regime are not that hard. The problem is detecting when regimes change and how to adapt algorithms to that.
This is an interesting comment as we have no way of validating how much money designers are putting into their own algorithms.

I started building algorithms on Quantopian 2.5 years ago, using it as a way to teach myself how to code because I come from a finance background and had no coding experience. I now have a portfolio of algorithms that consistently produce alpha (so far). My favorite algorithm does valuations and then buys and holds for long periods of time. I don't actually invest any money into this algorithm but rather use it as a prescreener for my own personal investments. I have a significant amount of 'skin in the game' because of the amount of time I have invested in developing my algorithms in addition to half of my (very little) net worth being allocated based on the suggestions of my algorithm.

I wouldn't be surprised if other designers were in the same boat. This is purely anecdotal but I hope it helps.

Do you know if there is something similar which can run against stocks traded on the Japanese markets? I would love to write a pre-screener for myself.
Uh, isn't "the market" just the ultimate crowd-sourced fund?
The market is like a giant voting machine but not really a fund, since nobody is managing it.
Index funds exist since the seventies.
Index funds actually still have a manager, whose only job is to rebalance periodically and accurately.
haha I had the same thought when I saw the title, 'wait, that's just regular commerce right?'
The nice thing about companies like numer.ai, is they give you a clean nicely formatted dataset to work with. With data that isn't public and that cost lots of money to obtain. There are a lot of people who are good at machine learning, but wouldn't be able to participate in the market without a service like this.
It might be an interesting idea to sample the market using this (e.g. to obtain behavioral economics signals), even though I don't believe this is their original intention.
Here's my "Tom Clancy movie plot" evil fund. Someone starts a fund that's actually based on figuring out the portfolios of Senators, moderately high net worth congresspersons and other Washington insiders. However, instead of duplicating the entire portfolios of the whole populace, you first filter for IQ, an age range where people are starting to be very well connected but still making their fortunes, and develop another filter based on a quantitative proxy for riskiness of each investment.

This is what the fund will really be based on, though there will be a fig leaf fictional method advertised. The purpose of this is to piggyback on the (technically-not) insider trading such people will manage to do while still staying within the letter of the law. (Somehow, the collective portfolios of US representatives manage to greatly outperform the stock market indexes.)

> Somehow, the collective portfolios of US representatives manage to greatly outperform the stock market indexes.

I would totally believe this. Do you have a source?

The investor mentioned in the article is the basis for Axe Axelrod in Billions. Which is a wildly outrageous portrayal of him but probably has a lot of elements of truth (eg extreme paranoia) since he was clearly guilty of insider trading (his subordinates indicted, his firm charged but not him).
I think the characters name is Bobby Axelrod.
Why exactly would that be evil? Everything you're talking about is public information.

And how on earth would you measure IQ? By some proxy measure? That seems like it would introduce unnecessary noise. It's not clear to me that any proxy for IQ would be a good predictor of outsized returns.

It seems like the best bet is just to filter on powerful Republican Senators: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2712297

Why exactly would that be evil?

You misread "evil fund." The fund itself may or may not be evil. However, it relies on not-illegal-or-not-caught activity to provide it information to purchase investments.

This sort of exists. It's not evil or secret though (funds can't make the large $ amount sized trades they need to do and keep it secret). Also while some funds do employ a type of 'regulatory risk arbitrage' it's usually part of a larger strategy, e.g. event-driven funds, which are slightly more sophisticated and a few miles ahead of what Tom Clancy could come up with. For example lobbyists and legal data would be relevant to the analysis as they may be forward indicators. 'Truth can be stranger than fiction'...
This sort of exists.

Yes. No surprise there.

It's not evil or secret though

Through legally obtained aggregate data, you can be pretty sure that some evil is going on in the observed population. Then you can use the same sort of data to also profit off of that evil activity. It's an "evil fund" in the same way that you can have a gold industry ETF. The ETF itself doesn't buy, sell, produce, or own gold. However, such a gold industry ETF can still piggyback off of its profits. More precisely, it would be a "surreptitious legislative evil" fund.

event-driven funds, which are slightly more sophisticated and a few miles ahead of what Tom Clancy could come up with.

I'm not an investment guru. This is just what I could come up with off the top of my head.

Gotcha. Well the funds are not illegal or secret, let's put it that way :)

I do think the world of Wall St. and Washington would make a crazy interesting movie. Maybe the "evil" funds buy stakes in various companies then lobby greedy politicians to make new laws aimed at giving advantages and profits to only those specific companies. At the same time these Dr. Evil types employ a revolving door strategy where they send trusted fund employees to work in government and they hire retiring politicians for huge salaries to make up the years they were underpaid "doing public service". Just an idea.

I'll repeat what I've said before about these platforms - I don't see why any institutional money would ever flow to them. Most startup funds require at least 5 years of audited history before any capital allocator will touch them. They might get some small quick money from fund-of-funds, but I think they'll be hard-pressed to keep it. Given the relative short history on these platforms and the likely churn in algos, I can't see anyone putting serious money into them.

One of the other problems I see with these platforms is that there is nothing that stops me, as far as I can tell, from pulling my algo from that platform once it is successful. Now there are lots of reasons why I could see people not doing that (running your own fund is hard, raising capital is difficult), but I don't see why anyone that is successful wouldn't immediately exit the platform in order to maximize their return.

All automated trading technology providers suffer this paradox: failing customers will fail, excellent ones will profit and move on to develop their own platforms. It's unavoidable. You must continually attract new customers and hope some are successful for your own reputation.
For numer.ai you can't leave. The data you get is private and encrypted. So even if you come up with a great algo, you depend on them to do anything with it.
Unless some other fund buys the same expensive data?
(comment deleted)
The problem with Quantopian, many current robo-advisors (including some with large valuations) and other market-related fintechs is that by and large they don't seem to have anyone who has had real success actually trading automated quantitative strategies at a serious hedge fund or tier-1 proprietary trading group on their founding teams. I've seen successful VCs, well-known academics, market gurus, people with a background in some aspect of running a mutual fund and all manner of other people who seem like they should be good, but nobody with an actual track record. I've seen people who worked on technology at hedge funds but the technology group at a hedge fund builds what amounts to plumbing like clearing, reporting etc, not the actual trading technology, certainly nothing that can actually impact PNL.

Jonathan Larkin at Quantopian comes closest to what is needed and since he joined Quantopian has certainly had good success, but even his experience was more along the lines of recruiting and risk-managing portfolio managers, not actually running a large book. He certainly helps Quantopian but Quantopian is coming from a place where when it was founded, I personally had to explain what selection bias meant to John Fawcett and despite Jonathan Larkin being there, they still seem to be making some pretty basic mistakes in how the platform is setup.

Spending a few years working on a tier-1 automated trading desk is absolutely essential because, what is deployed at those firms (and what you are competing with) is years if not decades ahead of academia and the rest of the industry and you learn more in a week of working on a successful trading desk (which only happens if you demonstrate a lot of not just academic aptitude), with people sharing knowledge available nowhere else than in a decade in academia or anywhere else, even other groups at the same firm, potentially sitting 10 feet away from the trading group. I'm not suggesting people steal IP or anything like that but you do have to have sone sense of what the state-of-the-art actually is if you are going to claim to have developed something state of the art.

I suspect it's going to end up like the search space, where the space will be taken over by the second generation of firms that nobody has heard of who decide to do things differently from how investment management is currently run offline taking advantage of their knowledge of how mutual funds including index funds are picked off by sophisticated traders.

Interestingly, Igor Tulchinsky at Worldquant and his team who are a tier-1 trading shop have basically been running a very successful version of what Quantopian hopes to become without a lot of hoopla or publicity for years, decades if you include the time they were doing this as an independent team at Millenium.

> how investment management is currently run offline taking advantage of their knowledge of how mutual funds including index funds are picked off by sophisticated traders.

This is news to me, I was not aware that most index funds are being front-run to such a large degree. I understand it was possible with the Russell 2000 index at some point. But e.g. the Vanguard Total Market fund (CRSP index) has almost identical performance to the Fidelity Total Market index fund (Dow Jones Total Market). And both funds replicate the performance of e.g. the Ibbotson book.

How is it possible that this is occurring if the index funds are being picked off? Or do you mean style / sector index funds, not broad market?

I don't mean to be snarky but I fail to understand why you believe that the fact that the Vanguard Total Market Fund has almost identical performance to the Fidelity Total Market fund and that both funds replicate the performance of e.g. the Ibbotson book is an argument either for or against my statement that sophisticated traders are able to make lots of money due to the actions of index funds. I don't mean to be snarky and I would really like to give you a meaningful answer but I'm not sure how to proceed and I'd like to understand why you believe the two have anything to do with each other so I can give you a higher quality reply.
they follow different indices and have similar returns. Please explain how this is possible while still underperforming what they 'should' be returning. Are both indices being front-run? But if they reconstitute at different times how is this possible.

And yes, you do sound snarky.

> they follow different indices and have similar returns. > Please explain how this is possible while still > underperforming what they 'should' be returning.

They are both equity indices and equities are highly correlated to each other. The first principal component of global equity returns explains over 50% of the variance.

> Are both indices being front-run?

Yes! If a stock is getting added (or dropped) from an index, this is typically announced (depending on the index) between 3 days and a month before the date on which the change to the index is made. The index itself is calculated assuming you bought (or sold) precisely at the close on the day of the reconstitution. Index fund managers are incentivized to match the index. They actually do worse personally if they modestly outperform the index and potentially get fired if they underperform. So every index fund manager wants to buy (or sell) the stock entering (or leaving) the index at precisely the same time on the same day. This means you could potentially have as much as a few months trading volume wanting to transact on the same side at precisely the same time. Smart traders take advantage of this by a) Buying (selling short for deletes) the stock over time before the index is reconstituted. b) Selling what they bought and short selling more (or buying) stock to the index funds at the close and c) Covering their shorts (or selling the excess stock bought) over the next few weeks. The fund managers don't care because even though a typical proprietary trading desk makes tons of money doing this, as far as their clients are concerned, they are matching the index. You could move a stock 50% in the rebalance but you wouldn't notice if you were comparing a fund's returns relative to the index.

This is fairly obvious if you simply take a look at price charts of stocks entering and leaving indices around index reconstitutions. Academic studies (use google scholar for a few dozen references) estimate that the typical index addition or deletion to the S&P 500 index moves between 2-4% between announcement and reconstitution. This is an underestimate of what actually happens because adds/deletes are predictable and stocks start moving long before the announcements are made. S&P 500 stocks are the most liquid stocks on the planet and the effect is much larger for other indices. The Russell indices are not much more or less gameable than other indices per add or delete but the effects are concentrated since all Russell Index rebalances happen on a single day (fourth Friday in June) which creates massive jumps in PNL for proprietary trading desks right around that time.

It's also fairly obvious if you look at the data carefully that in recent years with the increasing popularity of index funds, the indices are understating the potential returns available in the stock market. The market has done "better" than the oft-quoted returns on the indices would have you believe.

> But if they reconstitute at different times how is this > possible.

Why would the timing of the reconstitution have anything to do with why this is possible or not?

I explicitly avoided mentioning the S&P500. Why did you bring it up? It makes your point nicely but I'm not talking about the S&P500 or the R2k as examples where front running of any significance is happening. The index additions/deletions in the Total Market indices happen at the margins: micro-cap stocks (and IPOs). Hard to argue much happens of any effect with micro-cap stocks.

I don't necessarily disagree with you on the S&P500 or R2k, but it's much harder to make the same argument for total market indices.

Please justify how you are saying the market returns understate potential market returns. If it is not an index you are using, what is it?

> Why would the timing of the reconstitution have anything to do with why this is possible or not?

Because they have similar returns and add/remove stocks at different times. If Vanguard's total market fund adds a stock after Fidelity's, then any jump ("front-running") in share price due to Vanguard's purchases would be captured in a higher return for Fidelity since it already owned the shares.

I think I found a new interview question: write an algorithm that outperformed the market for the past ten years...you have 35 minutes...
That's easy and doesn't take 35 minutes. Here's the algorithm:

If year < 2009: short stocks

else: buy stocks

Add some leverage to this and you made a lot of money. Doesn't mean it'll work in the future, though.