Ask HN: Anyone making money through algorithmic trading?
Is there anyone here making money on smaller trading strategies (i.e. in the stock market or cryptocurrencies) that would not be interesting enough for larger algorithmic trading firms?
I'm aware the standard advice is that you will lose your shirt attempting to compete with algorithmic and HFT firms. But are there opportunities out there for smaller strategies to generate alpha? (I'm assuming yes, but would be great to find people who actually do this -- no need to disclose _how_ you actually do it, obviously)
253 comments
[ 6.0 ms ] story [ 3954 ms ] threadhttps://github.com/renasboy/crypto-trader-bot
You can use http://www.quantopian.com to try out different algorithms. It will tell you how well your strategy works.
Source: 5+ years in HFT development
Edit: actually, see my response to the neighboring comment too.
Languages like python are immediately out, they make no attempt to be fast (which is fine for their niche). Even languages like Java are out, the JVM is too smart: it turns out that the algorithm needs to analyse a few thousand possible trades where the answer no trade before it gets one where the answer is yes, as a result Java will optimize for the common no path. In C++ the programmer will lie to the optimizer claiming that the yes path is the most likely even though it isn't! As a result code written in C++ will always have the yes path as most optimal: when a trade should be made the java program pays a CPU branch misprediction penalty and the C++ program will not thus C++ wins the trade. Of course for the above to work you need cleaver programmers who spend time at the profiler and know how to make the CPU work for them.
C++ has one other useful advantages over some of the competition: it won't waste CPU checking for things like dereferencing a null pointer. Speed is important, you should verify via other means (unit tests, formal methods static analysis, or have the checks that you run on dummy data and compile without them for production)
Thus C and C++ are the only useful choices: the compilers have good optimizers (this rules out a lot of other compiled languages), they don't use CPU to check for "can't happen" errors, and you can lie to the optimizer in useful ways. There are other advantages, but the rest are things that a good programmer could easily work around (ie write a new stat structure from your CS textbook)
This means that whoever is not the first to take that opportunity doesn't get it, and if you're reliably a millisecond slower than a competitor then you might as well not even try.
1. Write a converter to convert a proprietary interpreted business rule language to C++
2. Make all messages fit the maximum ethernet frame size to avoid fragmentation overhead
3. Make the transported message format the same as the memory object format to avoid the packing/unpacking overhead
4. Predict and pre-allocate object pools in heap memory to avoid heap lock overhead
5. Statically link all libraries
6. Optimize key functions at assembly level
7. Turn all record ID's into zero based indexes and covert most lists and maps into arrays for rapid lookup
8. No indexes or foreign constraints in the rapid-write areas of DB
9. Hand rolled on-disk cache file formats that only operate in append mode to prevent seek overhead
10. And finally (this was after my time) trying to implement the TCP stack in FPGA
Worth a read.
Your time is an order of magnitude more expensive than that. Other expenses besides your time include making sure you have the fee structure to be competitive & getting a clearing partner that is ok with letting you fire off enough orders to make it worth while.
Over the last 3 months, January to March, I've had a return of just under 50% (in spite of the massive down turn in the crypto market). Which sounds great, but I only turned it loose on about $250 so we're not talking about buckets of money here. With higher amounts of capital you run into other issues that would need to be addressed in the algorithm/strategy.
I've often been told the same thing (you can't beat HFT, large firms, etc) but in the end it's not about beating them. It's about finding a strategy that works, that can be automated, and having the patience to let it run and do it's thing. I only trade about 1 to 2 times per day (not HFT) and only rely on fundamental data (no inside info, no "get the data before everybody else and act on it", etc). Keep it simple.
Happy to answer any questions.
To get started, I worked backward. The strategy is simple enough that you can execute it manually (e.g. read/interpret the indicators on a chart/graph and trade when the time is right). Once I determined a given strategy might be viable I formalized the strategy by writing a script to backtest it on historical data. Once it checked out I wrote a script that accepted real-time data from exchanges (I use GDAX, but most others with a solid API will work equally as well) and traded based on the incoming data.
I wasn't comfortable/familiar with any of the "hosted" trading platforms out there for cryptocurrencies so I built my own. It's very simple but it gets the job done and has proven very stable.
The most important part, for me, was to get the data streaming right. I ended up writing a Node.js server that listens to the GDAX web socket and feeds the data to a Redis instance. From there I have a separate process for each strategy I'm running that listens via a redis pubsub channel for new data. This was important for scalability and reliability. It also served to make the platform modular. For example, it can handle any number of data sources (exchanges) simply by adding a "connector" to the data source that feeds the data to redis. It can also handle as many strategies as you can think to throw at it simply by adding another script/process that subscribes to the relevant pubsub channel for the data it needs to execute the strategy.
It's probably overkill but it's made testing new strategies very easy and fast. I can take an idea for a new strategy, code it up, and have data streaming to it in real time in less than 10 minutes. Then it's just a matter of fine tuning the strategy.
A little long winded but I hope that answered your question...
That's sounds like a lot of work. What made you uncomfortable? Have you looked into using self-hosted trading platforms such as ccxt? It's an open-source JavaScript / Python / PHP cryptocurrency trading library with support for more than 100 bitcoin/altcoin exchanges
https://github.com/ccxt/ccxt
To be fair, I didn't look very hard for a good platform. I was aware of a couple that I had heard about either through friends or tangentially through HN comments on other stories but when I looked into them they either looked super shady and untrustworthy (yeah, I know, judging the book by it's cover) or forced you to implement your strategy in their language of choice (one was Python, one was a proprietary script-y language) and I wasn't interested in that.
Surprisingly it wasn't as much work as you'd think. It started as a hobby on the side so I wasn't worried about building a full-fledged trading platform that supported every exchange and every imaginable strategy type so I was free to keep it simple and focus on the one exchange I had an account at (GDAX) and the two or three strategies I had in mind at the time.
Find those patterns and trade on them.
Do you know if people are doing this?
The problem is those patterns quickly disappear as automated trading picks them up. The window goes from 5-10 minutes to seconds or less. Much harder to act in such a small window.
I'm not sure what the technical term is for a time-lag correlation though, since that's what you're really after; it's not an interesting correlation for your model if you don't have time to trade ETH on the BTC signal.
[1]: https://www.quantopian.com/lectures/beta-hedging
Pure voodoo
PSA: Don't do this
I found an algorithm that was wildly positive, and traded it on 3 separate markets every night. I learned a lot and I love everything I learned, but it was a very expensive lesson. The #1 thing I learned was that algo trading is mostly psychological, at least for me. I was making big bets (a few thousand dollars per trade) every night and it was emotionally exhausting, and I couldn't handle the pressure. The worst part is that I didn't trust the algorithm, and would cut the trades short instead of waiting for the full profit (or loss). That messed with my results, and in the end it turned into a disaster.
But programming for myself and using real money was such an educational experience. Any little bug meant that I could lose a lot of money so I bug-tested the most I've ever done in my life. And I did things like write my own multi-threaded backtester, working on hundreds of gigabytes of data, so I learned a lot there too.
It was a lot of fun, very very expensive fun.
Not if you're leveraging! For an extreme example, this guy lost $210k of borrowed money: https://www.bogleheads.org/forum/viewtopic.php?f=10&t=5934&s...
Starting with around $10-$20,000 in what were college loans, Sacca realized that brokers weren’t accounting for margin usage on a real-time basis. The result was that even though firms were only allowing 50% margin, as long as a customer closed a trade before the trade settled, T+3, and showed proper equity in the account, positions larger than 50% margin usage could be opened.
Picking winners of stocks that by Sacca’s account had risen 40x, and betting with positions well above his margin requirements, his account grew to $12 million in 18 months. But, as we all know, the record levels of the Nasdaq and the dot com bubble of that time eventually burst....When it did burst, and even though the damage was from holding just two stocks, Sacca found himself in the hole with a $4 million negative balance.
[1] https://www.financemagnates.com/forex/brokers/chris-saccathe...
Paper trading is nothing like real money trading. That's also one of the first things you learn, it's like a different dimension.
My guess is what you really want to know is "What is my expected gain if I try to employ an algorithmic trading strategy?" I have my suspicion of the answer to that question, but I don't believe your current question will shed any light on the answer.
See /u/Fletch71011 on reddit-- he's always happy to discuss things. He's made millions trading options, mostly algorithmically as I've understood it. The methods he uses are sufficiently complex that you need to be very well acquainted with the intricacies of derivatives to follow along, but basically he trades volatility instead of price movement. Regardless of whether the price of the asset goes up or down, he makes money. In his opinion, it's foolish to try to trade price direction, and you're basically flipping coins and likely to lose money.
I tried understanding what he was doing and abandoned the attempt. I had to conclude I was not quite so clever as he.
It takes just as much skill to guess if volatility will go up or go down as it does to guess if prices will.
The best models we know of (say GARCH) will still pace the information in the option prices.
Big moves either up or down would be profitable. The only unprofitable move here would be no substantial moves in either direction. (In which case you lose your entire bet, but no more.)
The problem is that much more often than not the “only unprofitable move” is the one that happens. Options in general are too expensive and that’s why selling “insurance” is profitable most of the time. Maybe he can identify consistently mispriced vol, though.
This was the method I used, as described in another comment.
Clearly I lack a basic understanding of the concepts involved.
Sort of like how different companies with the same cash flows can trade at different multiples, otherwise identical options in two companies (or different expirations/strikes in the same company) can trade at different prices because of the opinions of market participants. Volatility traders act when the different prices/error terms are too far apart, counting on the prices/error terms to converge a.l.a. pairs trading.
Since they are trading the error term directly, they attempt to construct positions that remain relatively flat in value as the stock moves around, but are designed to only change in value when the error term changes. That is how they can make money "both ways", because they can profit if the stock goes up, down, or stays the same, as long as the error term moves in the correct direction.
The reason you only see sophisticated people doing this kind of trading is because you need a large and complex position with many hundreds of options to be in a truly market-neutral environment. You can't take advantage of mispricing without such a large position because buying/selling single options involves a tremendous amount of risk, so you need to do that as a part of a larger portfolio to spread that risk. Retail traders tend to spread the risk by doing 2 transactions (the mispriced option and a well-priced but mirrored hedge option), but that is a) much more expensive from a commissions standpoint and b) really limits the range of market-neutrality forcing you to adjust more frequently to stay market-neutral, again, with commission costs.
[ed: that's to say, you need a way to get more/accurate pricing information than reflected in the market - but for options, not assets]
If you buy put options for X at 10, and call options for X at 10, then if the price moves down you exercise the call option, and if the price goes up you exercise the put. Unless the price is totally fixed, you make some profit. The real question is whether this profit outweighs the price of both your options.
You need the price to move sufficiently for this plan to be worth it. Especially because in any case, either your put option or your call option is worthless. Thus, you need twice as large a price move as when buying only puts or calls. The upside is that you don't need to care about the direction of the movement.
I agree with you in that it is a possible explanation, but I disagree in that it is the only one possible.
They may have something, but since you can't determine if it's so and there is a lot of just gambling, then most likely it's that.
It is all moving to algos, though.
With vol it's one of those specialized areas where it's probably quite hard to learn without having sat on an options desk. Even the way it's presented in the books does not give a good picture of what you're supposed to do.
Individual trading strategies often become less effective over time, though. Whether this kind of success can be sustained at the level of a trading firm over many years is an entirely different question. Whether they can beat the market after fees is a third, also entirely different question.
The market is negative sum, so any abstract strategy has an expected excess return which is negative. It should be everyones assumption without competing evidence
Algorithmic strategies include such gems as "buy on mondays and sell on thursdays", and there is no inherent magic to them making them better than my "buying stocks with names I like".
Long answer: not in the beginning, then a long period of breaking even, and eventual profitability.
My algos trade commodity futures(nasdaq, 30-year bonds, etc). My platform is Multicharts.NET, which supports writing your strategy components in C#.NET. I'm not a .NET fan, but the platform is solid and this is about dollars, not language preference.
Also...regarding HFT - those aren't likely what you think they are. In commodity markets, HFT trading typically follows a simplistic algorithm of "above or below XYZ bar EMA", which anyone can do. The HFT portion of it comes in through the process bidding the inside bid(on the way up) or offering the inside offer(on the way down) faster than the other HFT algo. So where a price may eventually see 100 bids on the way up, and 20 of those will be filled, the HFT's goal is to place bid #2 or #3 out of that 100 - competing with hundreds of humans and other HFT's for that spot in the queue. Trying to compete with HFT is very difficult unless you have enough capital to colo next to the exchange(CME in my case), as well as handle commissions(through paying them, or paying for a seat to negate them).
I've attached a screenshot of the chart output from my algorithm today. Assume every time you see "SE" or "LE", that a long(LE) or short(SE) position was established. Positions close when the first of 4 events happens: stop loss, profit target(25pts for today), trailing stop(10pt), or an opposing signal is generated. It's simple, it's not that sophisticated, but it is consistently profitable. You can develop your own similar algorithms, or use many out-of-the-box algorithms from places like iSystems, or strategies that come built-in with your platform(Multicharts.NET has many). The key is backtesting, properly scheduling around economic events, and having enough capital to survive the inevitable drawdowns.
Link to screenshot of today's chart output from algo: https://www.dropbox.com/s/bms9kuqn529iyqg/Screen%20Shot%2020...
If you go down this road, I wish you the best of luck. It's fun, it's difficult, and incredibly rewarding if you get it right.
Have to create income that doesn't multiply by hours spent, period.
I think that's doing something.
Edit: Not saying it's easy to reach that level of profits, just that it's possible for professional/sophisticated traders.
https://www.straitstimes.com/business/companies-markets/bitc...
I wrote this repo a few months ago for people to get started with cryptocurrency algo trading: https://github.com/jsappme/node-binance-trader
Feel free to contact me at herve76@gmail.com if you need me to write your own custom trading algorithm.
Why would you do this when you are already making $500/day ($182,500 per year)?
This is entirely meaningless without knowing how much you started with. There's a reason why ROI is often stated as a percentage.
If you have a billion dollars invested, gaining $500-1000 isn't much. If it's $10,000, then that amount is pretty significant.
It's like claiming you drive a fuel efficient car because you can drive 500 miles on one tank without disclosing the size of your tank.
https://projectpiglet.com/
Basically, it tracks experts and makes decisions on investments given what the experts say. Think about how many times you've seen someone say: "I work at Google, our cloud is doing X" or something like that.
The fact insiders talk, let me track them and make money.
Unfortunately, that doesn't mean I'll make money tomorrow. The market is always correcting.
1 = https://www.sec.gov/fast-answers/answersinsiderhtm.html
A single piece of non-public information that would move the market. Trading on that information is insider trading. Think "we have undisclosed losses equal to 5 times annual earnings" that your brother told you at a bbq. That is insider trading.
Inferring the existence of some information based on many other pieces of information isn't just legal, it is encouraged. "These guys must have some big undisclosed losses, remember how Mary used to talk about engineering projects and how she talks at BBQs now? Bob was saying his HR dept. is insanely busy. Ahmed resigned and took a job that looks like a step down for him. Ok I'm a sell on this." That is equity research and to be encouraged.
https://www.nature.com/articles/srep01684
The article is from 2013, but if anything it may be more predictive today given the ubiquity of Google search. At the very least, since it explains the method they used to find this signal, even if the specific keywords they used the trends for are no longer predictive, you may be able to find others that are. The strategy yielded a theoretical (backtested) return of 326% over a 7 year period.
So, while all HFT trades are algo trades, reverse isn't true.
AFAIK some(maybe a lot) of algorithm or quant firms hire people who can read the latest investment research, form a hypothesis and test out the hypothesis to see whether there is a winning system.
A side tip - If someone says their algorithm relies on some sort of TA, run for the hills.
Care to explain? I'm genuinely curious as I've had some success in this area. Curious if I should be aware of something that I'm not...
However, obviously many people do rely on TA, which means TA is influencing the price (you can beat their TA algo with your own TA algo). The problem is, you never really know what everyone else is doing. Relying on TA amounts to playing rock-paper-scissors, blindly, with 1000 opponents, and hoping you choose the winning move against most of them.
Keep in mind there are only 2 outcomes, win or lose, and they occur randomly, so it’s really easy to fall victim to selection bias and think you have a winning TA-based algorithm.
This is not true at all. Price movements show auto-correlation, for example.
One of the biggest flaws is that TA indicators tend to repaint. So an awesome winning MA crossover in hindsight might never really execute during real trading.
Additionally using TA for trading also involves self-fulfilling prophecies. And if there are any markets which follow this prophetic tendencies - it is cryptocurrency.
https://www.investopedia.com/terms/t/trendanalysis.asp
On the negative side, the spreads, fees, and latency funds and banks get are smaller than what you can get on online trading platforms. So focus on longer-term strategies (with a holding period of a few hours or more), because you'll lose out to the big guys with any medium to high frequency trading strategies.
1. Systematic trading doesn't necessarily require an algorithm. For instance this rule might work (over a 5 year horizon, don't try it monthly): "buy very large cap stocks if their P/E goes below 4 and sell if it goes over 10." But you don't need an algo.
2. The market has long bull runs. So you might have an algo that has some long bias. It will seem to perform above chance. You should compare it to just holding the market.
2b. If the market is going through a bull run and your algo has some leverage built in, it will outperform just holding the market.
3. If the market had a massive crash in the data set and your algo has a short bias, then you should check it against just shorting the market.
4. The issue of models, markets and biases mirror the same debate in science theories, data and statistics.
If the entire stock market P/E goes below 4, you could use it. However when a single stock goes that low it implies that somebody knows something. Don't invest in a company with a P/E below 4 if they are going out of business. (the obvious example that probably never existed: a y2k company in 1999 - they are probably making a ton of money this year, but next year they will go out of business).
There are also "cyclical companies". They have a long history of having boom and bust years, it is well known that you buy when the P/E is high - the bust years when prices are low - and sell when the P/E gets low - those are the boom years when prices are high - but the boom will not last.
There are many other company specific things that can get in the way of any formula.
I used Python and ccxt.
This could happen because of fraud by the exchange, fraud against the exchange, hacking of the exchange, or regulatory risks where other financial intermediaries stop working with an exchange or regulators threaten to punish an exchange if it processes certain transactions.
Some people have suggested that because arbitrage opportunities are pursued aggressively, most price differences between cryptocurrencies and cryptocurrency exchanges that persist are probably mainly due to people taking account of counterparty risk. In that case you could still profit some of the time by betting that a risky exchange will remain solvent, but you might be taking a larger risk than you realize.
Another interpretation is that some apparent cryptocurrency arbitrage opportunities are really opportunities to earn a premium for helping people evade capital controls and other regulatory restrictions on moving money around. For example, there are lots of people in China who would be happy to pay you a premium if you'd accept a payment in China and make a corresponding payment in Canada. In that case you might feel like you're being a clever arbitrageur but you're largely receiving a payment for helping someone circumvent regulations¹.
(The implicit moral opprobrium that might be read there isn't intended, but I think it's interesting to consider how cryptocurrencies can sometimes make people feel very clever when they aren't, in fact, the cleverest ones in the situation!)
(¹ which isn't necessarily unreasonable to describe as arbitrage)
The bot uses a NN for predicting the price. I've been working on it for 3 months and so far the bot is profitable.
If anyone out there is interested in this space I'm looking for a partner. Also open to business offers.
The whole pipeline (data collection, data processing, trading bot, backtesting, model training, etc.) is built on golang, aws, and training with keras.
Shoot me an email [redacted]
The key for me is to focus on long-term trading strategies that are at least a year long. The HFT guys and people who spend their time on quantopia and the like have a day trader mentality.
Let me know if you have any questions.
Thanks for asking this question, I will look for you on twitter.
Check out Berkshire Hathaway's performance. There are plenty of years they have UNDER performed. But they are doing OK.
http://www.berkshirehathaway.com/letters/2017ltr.pdf
How many trades did you do over the course of the year? Was your volatility lower than the market overall?
I care so little about volatility that I'm not even measuring it yet. And I admit that might be dumb.
[1]: https://dqydj.com/2017-sp-500-return/