26 comments

[ 2.0 ms ] story [ 29.6 ms ] thread
Cautionary observation: The same company that judges the contest also makes more money when this specific kind of contestant wins.

> "With the Quantopian platform, anyone with a bit of coding skill and a mind for finance can start writing algorithms. The best ones win."

> The winning algorithm each month is deemed to be the one that, based on Quantopian’s analysis, offers the best combination of [...]

It is entirely possible that everything was aboveboard and honest, but there's still a latent conflict-of-interest since the company may have leeway choosing what constitutes the "best combination" of metrics.

We work very hard to make our interests aligned with our community members' interests. We don't literally make money from the algorithms in the contest. What we're doing is encouraging hundreds and thousands of new people to write algorithms. The best ones will be invited to join our hedge fund, we'll negotiate compensation with them, and we'll take investment from pension funds and endowments and the like. In that sense - yes, when our customers win, we win too.

As for the judging, I think you'll find it be very transparent. There are 6 return and risk metrics calculated for both the backtest and the paper trading. The 6 metrics are weighted equally to generate an overall score. You can see the metrics for every contestant and the combined score on the leaderboard. You can verify it all by downloading the CSV. https://www.quantopian.com/leaderboard

At the risk of being overly cynical:

Week 1: Take 512 people. Mail half of them a letter that says "the market will go down this week". Mail the other half a letter claiming it will go up.

Week 2: Take the half who received a correct prediction, split into 2 groups, repeat.

Week 3: Repeat.

Week 4: Repeat.

Week 5: "I've predicted market movements perfectly! To get on board with my foolproof strategy, send your investment check to..."

This seems to be the opposite.

Week 1: 512 participants. Half of them mail in "the market will go down this week". They lose.

Week 2: Of those who made a correct prediction, half make a correct prediction again. The rest lose.

Week 3-8. Etc

Week 9. We found a winner, one who could predict the market flawlessly!

To be fair, we invested our own money in the winner.

The composition of the algo portfolio for the fund != the winners only. Our challenge is to choose the optimal set of algorithms from a huge pool. Optimal means maximizing returns while also minimizing correlation between the strategies.

The holy grail of investing is 10 or so uncorrelated return streams. We're building a community and platform to consistently turn out algorithms with uncorrelated return streams.

Too much noise - this means nothing. This is exactly the reason why serious investors give fund managers money not just based on performance, but on process.
Or invest in an index with no active risk.

I have trouble taking seriously any "serious investor" that invests in a fund and expects to beat the market. It's been researched to death and it's a losing proposition that doesn't acknowledge the rules of the system.

You can beat the return of a broad market index just not by paying someone else to do it.

I take it neither of you have ever read "What Works on Wall Street"? crack it open, and tell me it's impossible to beat the market. Just because lazy people advocate buying index funds doesn't mean it's automatically the best choice. If you're just satisfied with mediocrity, then fine, do it. However one trait of successful humans is that they seek to improve upon the status quo.
Process means nothing if it doesn't have good returns. You can have the best analysts and if something goes wrong, you still have good "process" but you'll have peanuts to show for it. At the end of the day, what matters? Your returns - that's why people invest.
I was really hoping to enter that contest with an entry that just used rand to flip a coin and invest. Never had the time unfortunately, but would still love to see how a completely random strategy compares to some of the algorithms.
We have a contest every month. The next deadline is April 1, and the prize awarded on April 30th.
It really depends how you're using the random coin tossing. Are you going through a list of securities then tossing a coin and if it's heads you'll buy it, if it's tails, you'll sell? How will you allocate your capital? If it's heads you'll put commit 10% or your balance and if it's tails, 20%? Will you use coin tosses for your exit parameters as well?

While coin tossing in and of itself is random, the manner in which you use it is not. So at the end of the day, your 'completely random strategy' may not actually be all that random... Just something to think about.

The algorithm you need to win a contest is the highest risk algorithm you can get away with.
Here's how I think about our contest: If you only had backtesting, you could win by fitting. If you only had 1 month of paper trading, you could win by luck (taking excessive risks).

But our contest is 2 years of backtesting + live trading for a month. I don't think it is likely you can both overfit and be lucky with one strategy.

Assuming you blackbox test the algorithms, you could include some signatures of past data. If those signatures are detected, you know the future and can trade optimally. If they are not detected, go for blind luck.
Finding anything meaningful in results like this is a bit tough since everything is looking backwards. It would have been nice if they said (for example) "okay, here's the deadline, you can't make changes to your algorithm after this date, and then we'll test it over the next six months of data."

Also, I wonder whether they're taking costs into account since any strategies that involve making lots of trades will quickly see their profitability drop due to fees.

It's both forward-testing and backward testing. The algos have been locked since submission - some were submitted as early as 1/15, all were submitted by 2/2. That makes it both an in-sample and out-of-sample test.

Yes, the Quantopian platform includes default commissions. It also includes default slippage. No model is perfect, of course, but this is a tool that's had a lot of development.

(I work at Quantopian)

isn't that effectively how it worked? "a combination of live paper trading results during [the month after submission], as well as two years of backtesting against historical market data"
It's such a shame that we have the lives of such smart people being sucked up by such a vacuous, socially value-less and damaging industry.
Investing is planning for the future. It is a critical tool for society. Everyone from individuals to important institutions like universities and pensions need to invest. It is a need of the many, not a diversion for the few.

Because of HFT, it has become popular to blindly condemn investing.

I went to college because an endowment was able to provide me with a scholarship. Investing has a huge effect on society and on people.

It's such a shame that we have people posting comments online when they haven't the slightest clue what they're talking about. Do you really believe finance is a valueless and damaging industry?

If so, do you realize that all the electronics you're using to post your nonsensical comment were made possible BECAUSE of finance?

Sure not everything in finance is good - but that's the same in any other industry. In tech you have people who write programs to destroy other people's data, in biology, you have people trying to create bio-weapons. EVERYTHING has the potential to be damaging!

But finance, finance makes it possible to have all the amazing advances we've had! How could small companies be able to grow and spend on R&D as they did without the aid of financing?

Just because it's hip to bash finance, doesn't mean it's right. Don't be a brainless sheep mindlessly following others. Try thinking for yourself, you might be surprised with what you come up with.

Is it possible for others to see the source code of the submitted algorithms or is that reserved only for the Quantopian employees?

If you have a good algorithm, why would you share it with anyone? A good one would be worth more than $100k.

A good algorithm is useless unless you have enough capital to work with. You might have to spend years just building up enough capital to make a decent income off your trading; or, lack of capital might well induce you to trade at high levels of risk, and lose the little capital that you have.
We don't look at the source code of algorithms submitted to the contests, and the code is not made public unless the entrant chooses to share it in our forum.

(I work for Quantopian.)

(comment deleted)
Honestly, I don't understand why there's so much cynicism regarding Quantopian and their contest. I think it's great that there's a tool people can use to get started with automated system trading.

For all of you who are whining about stuff, do realize that no one is being forced to enter the contest. Did you get that? NOT REQUIRED.

On the other hand, for those of us who want to learn or try our hand, this is a great idea/product.