HN Explorer

Ask HN: Trading Algorithms

72 points by jchonphoenix ↗ HN
So I'm a university student that has been working with recommendation systems recently (netflix challenge etc) and wanted to try my hand at implementing some trading algorithms for the stock market to see if I could at least break even. I'm fully expecting to lose everything I invest, so that's not a problem, but I figure it'd be a good learning experience. Does anyone have some links to good papers about machine learning applied to stock trading (or any trading algorithms in general)? How about API's that could allow my programs to perform trades?

Thanks for all your help HN!

34 comments

[ 3.2 ms ] story [ 72.1 ms ] thread
owyn
I stumbled across this paper a while ago on HN and found it interesting:

http://www.cs.bham.ac.uk/~wbl/biblio/gp-html/Dempster_2000_Q...

twp
Dempster was my PhD supervisor and I built on his and Chris Jones's work.

The idea was to use genetic algorithms to find combinations of technical indicators that would turn a profit in high frequency foreign exchange trading. Another PhD student (Yazann Romahi) tried a reinforcement learning approach.

The GA found combinations that worked OK on the training set as long as transaction costs were very low. Outside the training set they sometimes did OK, but, this being genetic algorithms, you could never be sure that it was just luck and over-fitting to the training set. As soon as transaction costs approached realistic levels everything lost money. Another student based his PhD thesis on identifying the 50 successful runs out of 1000 which were significant at the 5% level.

I wouldn't recommend this as a useful approach - the best way to make money in FX is to be a market maker or broker, i.e. make money on the bid/ask spread or to get commission on every trade.

Ultimately I found the mix of finance and artificial intelligence to be a case of the blind leading the blind. The financial people had lots of money and deeply wanted to believe that there were magic patterns in the market that computers could discover. On the other side, the AI people wanted research funding and deeply believed that their magic AI black boxes (neural nets, genetic algorithms, support vector machines, etc.) could discover these patterns. The naivety of and misplaced belief of both sides was quite depressing to observe, although many did (and do) exploit it for their own personal profit - primarily through exorbitant consulting fees.

Kjeldahl
For API connectivity and instruments, few beat Interactive Brokers, at least for the price offered. As for strategies, avoid walking into the complex indicator traps (combining indicators x,y,z to make a new super indicator). Stick with the simple stuff for trends etc (moving averages), and try to combine with other stuff, possibly other data (from other markets, or completely different sources). And forget about having a model that runs unattended on its own while you're at the beach. Markets change and you should stay on top of it, adapting the models for whatever the markets demand (watching out for sell-offs the last couple of trading hours is one recent theme). And don't forget about money management. Good luck!
greyman
Question: Why combining indicators doesn't work?
wisty
The curse of dimensionality, I'd guess.
ggruschow
It can, but it's usually an impossibly huge search for a way to turn garbage into gold.
pictional
Overfitting, curse of dimensionality, etc. As in many machine learning tasks, extracting good starting-point features is very very important, and that often requires domain-specific knowledge.
ashika
Seconding interactive brokers for this type of experiment.

For a reasonably accessible, yet complete introduction to basic trade design, 'Pairs Trading' by Vidyamurthy is quite solid. The strategy may well be picked clean, but the book does a commendable job in exposing the general process of finding edge and minimizing slippage.

billpg
"I'm fully expecting to lose everything I invest"

You'll never get the taxpayers to cover your losses with an attitude like that.

retube
depending on your leverage you could lose a lot more than you invest....
henrikschroder
I actually thought googling "bankruptcy engine" would yield some good results, but apparently not. Oh well.
[deleted]
(comment deleted)
Mamady
Have a look at http://www.collective2.com

They have many trading systems, and have automated trading with various brokers. By looking at them, you can probably figure out which brokers have APIs for stocks, forex, options etc.

You may even be able to reverse engineer some of their trading algorithms ;)

CoachRufus87
i'm curious as to how you plan on implementing this system...algorithmic trading has always been an interest of mine, but I never quite understood how it really worked
iworkforthem
I am into proprietary trading too. For retail investors like myself, seems only NinjaTrader & MetaTrader allow algorithmic trading capabilities via API. But both uses its own language.
waveman
http://seykota.com - not what you think you are after, but very useful regardless.

Also http://www.trendfollowing.com/

Trading is very hard. Markets are efficient at a first approximation and where they are not efficient, simple arbitrages do not exist. You are up against smart, well capitalized people. Once you find an edge your irrational brain cuts in.

Books: "Investing by the numbers" by Jarrod Wilcox. "Trend Following" by Covel. "The only three questions that count" by Fisher. And of course the "[Stock]Market Wizards" books. Typically the top traders seem to have read a couple of hundred books on the topic and have done thousands of hours of research.

Your assumption that you will lose at first it probably right. But that's better than winning at first, which is likely to falsely convince you that you're a genius.

Beware the following which can look like skill for a while. 1. Randomness (temporary luck) 2. Excessive Leverage (-> gambler's ruin) 3. Selling insurance without realizing it eg selling out of the money options.

andr
One of the key lessons in finance is that you can't make money doing what everyone else is doing. Every known technique is being applied a lot and can rarely bring you abnormal returns.

I suggest reading through the links other commenters posted and trying to think of ideas that you never saw mentioned. Brainstorm wild ideas - for example, can you adapt a lossy compression algorithm to predict the stock market?

ggruschow
One of the key lessons in finance is that you can't make money doing what everyone else is doing.

This is only true in the long run, and not true for individuals. As a simple example, consider the practice of selling far OTM options.

ritonlajoie
You may like Ninja Trader, it's free for personal use. This is a great free automatic trading application, you code in C# using their API.
hogu
don't loose all your money, interactive brokers has paper accounts that let you fake-trade without consequence.
ggruschow
Don't try to teach the machine things you don't know yourself. Study the rules and actors in the systems you're studying, and come up with theories to test against the data. You'll need to do this work anyway, so it's at least a good way to start before you complicate things further with AI/machine-learning.

Understand this first though: The vast majority of money made in trading is made without anything more than a trivial prediction of the future. Investing is different, but very very few have enough money to make active investing worthwhile.

Start out with "Algorithmic Trading & DMA" by Johnson: http://www.amazon.com/dp/0956399207 If you choose not to, you're better off not reading anything about the actual trading than trying to pick another.. The vast majority of published material is so bad that it'll set you back.

lrm242
This is a fun game and is very challenging. If you apply yourself you can be successful.

I would highly suggest picking up the book, Trading and Exchanges: Market Microstructure for Practitioners (http://www.amazon.com/dp/0195144708) by Larry Harris. To trade effectively you need to understand how the markets work and this book provides and outstanding tour through the markets, who participates in them, and why they do or don't make money.

There are innumerable ways to make money in the markets. Long term, short term, technical or fundamental, with retail platforms like Ninja or going very sophisticated and connecting directly to an exchange like NASDAQ using native protocols like ITCH and OUCH. Don't let naysayers distract you from your goal--for every naysayer there is always a counter point.

If you want some motivation, read through this IamA at reddit: http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...

arthurdent
Interactive Brokers has paper trading and good docs http://www.interactivebrokers.com/en/p.php?f=tws&p=p http://www.interactivebrokers.com/en/p.php?f=programInterfac...

http://www.maxdama.com/ talks about the IB api a bit.

I skimmed your comments and kept seeing the word python. There's a 3rd party python-IB package here (can't vouch for it): http://code.google.com/p/ibpy/

http://epchan.blogspot.com/ talks a little bit about basic pairs trading strategies. My experience has been that adding complexity off the bat is usually not the way to go. I don't personally know anyone who is using AI/machine learning as their primary strategy generation mechanism, but maybe I just run with the wrong circles.

I'm interested in collaborating with smart people re trading ideas, so if you're interested as well, fire me an email.

jchonphoenix
I've heard of IB before so its a name I'll probably look into.

I'm mostly interested in machine learning because I feel like a lot of the algorithms I'm applying to recommendation systems could be applied to stocks. If you think of the stock market as a giant system where someone wants to recommend stocks that will go up, it appears these ML techniques may be applicable. At this point however I don't have a way to model the stock market as an acceptable input to the most promising algo I have, which is one that I developed myself and has and extremely low RMSE for certain types of data.

arthurdent
Seems there have been a lot of posts on HN in the last week or so about algorithmic trading.

There doesn't seem to be a shortage of interest in this field either, but the threads quickly get lost in the HN jumble. Does anyone want to volunteer to take this conversation somewhere else?

RockyMcNuts
I would probably start with the Turtle Rules - an ultra simple system that was used successfully in the olden days, as a good overview of the elements of a successful system.

http://bigpicture.typepad.com/comments/files/turtlerules.pdf

Possibly of interest - http://www.amazon.com/Trading-Systems-That-Work-Evaluating/d...

MetaStock - system backtesting software - http://www.equis.com/

regularfry
Seconding Turtle Rules - I've just read it, and it's a good, quick, informative read. It's more a meta-strategy book in that it tells you how to approach building and running a system, rather than focusing on the system itself.
regularfry
Nobody has suggested it yet, so for context I'd suggest reading The (Mis)Behaviour of Markets, by Mandelbrot. It's not about trading algorithms per se, but it does put the maths behind a lot of them into perspective. It'll also tell you why certain strategies (like selling cheap options) are much, much more dangerous than others.
[dead] DISFA_Global
Machine learning and Artificial Intelligence are making their way in trading and investment.

Strongly recommend you "The Elements of Statistical Learning: Data Mining, Inference and Prediction" by Hastie, Tibshirani and Friedman.

One of the technologies our company develops is called Alimp. This is intelligent software for automatic creation and testing of robust trading systems powered by advanced A.I.

The idea is to run Darwinian process of trading systems evolution. Alimp uses combinatorial intelligence to generate 500 000 trading systems candidates per day. Then research group picks up the best survivals to package (to add money management, position sizing, risk control, etc.) and to test.

Maybe our white paper will be of some use to you. You can read it here: http://disfa.com/downloads/Technical%20paper_ICAI08.pdf

Good luck in your endeavors! You can let me know if you have any questions: olga.grinenko@disfa.com