HFT isn't necessarily a scourge - if anything it discourages people from short term investing, which is a Good Thing.
Long term investing isn't affected by HFT, except in providing more liquidity at the point you want to exit any position. If you're planning on holding for 5+ years you're competing against relatively few people and can do good research. Any shorter timeframe and you're competing against millions, against computers and against unknowables that will distort the price in the short term.
It's weird how this "providing liquidity" distraction never seems to go out of fashion. HFT provides liquidity if you happen to be another HFT algorithm. Otherwise, not so much.
Why would short term investing be bad, and how long do you think HFT algorithms hold their "investments"?
I hope you're not just trying to blow smoke up my ass here.
It isn't just about liquidity, though it does. But it also reduces the bid/ask spread, i.e. price discovery. The HFT trader may make a tiny profit when it processes a trade from me, but I've benefited because I can buy/sell at a price close to the quoted exchange price, which didn't use to be the case.
In terms of liquidity those trades are available to all, they love buying from non-HFT trades because they can make a small profit from their less accurate pricing. You benefit from the lower spreads and that liquidity. Before HFT you'd have paid a greater cost due to the spread on the price.
Short term investing is "bad" because you're not investing in the future success of the company you're trading on the basis of what you think the market is going to do. To the extent to which that is ever knowable it is unlikely you have the skill, experience and data to be able to beat the large number of professionals doing it.
It sounds like you work for the financial industry too.
> The HFT trader
You say it like it's a person making trades.
> I've benefited because I can buy/sell at a price close to the quoted exchange price, which didn't use to be the case.
What's a "quoted exchange price" if it's not the price you actually pay?
> Short term investing is "bad" because you're not investing in the future success of the company you're trading on the basis of what you think the market is going to do.
Isn't this what HFT is all about?
Trading on insider information is bad, of course. But if you're on a level playing field, I don't see anything wrong with profiting from something you correctly predicted short-term.
I do - but a 'proper' long term investor that cares about the price in 20 years time.
The quoted exchange price is an average of the buy/sell offers available, there is no guarantee if you try and trade that you will pay that amount. Especially if you are shifting large blocks of stock.
There is nothing wrong with "short term predicting" but it isn't investing. It is often a very easy way to lose money, "bad" doesn't just mean morally so.
It comes down to time horizons, how much you want to make, and how many people are also trying to predict that event. You can correctly predict that event and still not make money as it is already priced in.
> What's a "quoted exchange price" if it's not the price you actually pay?
The actual price you have to pay is based on how much you want to buy and how much other people are willing to sell at what price.
People who want to buy give a "bid" price and people who want to sell give an "ask" price. Whenever there is an ask price that is lower than a bid price, a sale takes place. This results in the lowest ask price always being higher than the highest bid price. The quoted exchange price will be somewhere in between the two prices.
If you want to sell right away, the most you can get is the highest bid price on the books. If you want to buy right away, the cheapest you can get it is the lowest bid price on the books. This means that you have to pay more than the quoted exchange price to buy and receive less than the quoted exchange price to sell.
Investing in company stock is much more about investing with a global pool of liquid capital seeking risky asset exposures these days. When global capital changes it's mind (in whatever direction) those watching the market closely will be the first to know and the first to profit from the market signal. You can go broke investing in companies that look like they have a bright future.
Long-term investors trade every day. You wake up and the market has gone down a few percent, you're going to rebalance at least. You decide to buy 30 year treasuries you're not going to hold them for thirty years - you're going to sell 29.5 year bonds and buy 30 year bonds in a few months time.
And managers turn over long-term investor portfolios quickly. The average holding period for an SP500 company shareholder is 100 days.
And every time a trade happens you run the risk of getting clipped by the faster guys who see you coming.
But long term investors don't care about the "clip" that HFTs take, they care about the 80% rise in price over the next 5 years. Rebalancing portfolios again isn't something where getting a few basis points off the HFT best execution price is an issue.
I work for a proper long term investor, and we trade over a period of days for the big trades. HFTs aren't even an issue apart from their occasional spectacular blowups that cause "could this happen to us?" memos from upstairs. Long term investors don't need to evolve from HFTs, they're more than happy making money not involving computers or trying to compete with them.
> And every time a trade happens you run the risk of getting clipped by the faster guys who see you coming.
I think it is natural, but fallacious, to apply line-of-sight properties to trading. If you plan to trade, then there are two ways that another participant can "see you coming":
1. If you don't have direct market access, your order gets routed through a broker. The broker sees your order before it hits the market, and if he jumps in line ahead of you then that is front-running, and a Bad Thing. Your broker can get in a lot of trouble for this sort of thing.
2. If you are trying to move a large position by sending multiple orders to the market (one after another), then all market participants have the chance to react to the first order. It's really a game to try to move a lot of inventory at once, without tipping your hand to anyone else in the room. Thems the breaks.
Nobody else gets to see anyone's order before the matching engine has already processed it, so there's no way to jump "ahead" of it.
OTOH, maybe your long-term investor is trying to time the market: wait for a signal intraday, and pick that moment to send an order. In that case, if it is a good intraday signal then it is likely that someone else will compete. It is unlikely for a long-term trader to have spent as much on infrastructure as a HF trader, so the juicy signals will result in missed executions that _look_ like front-running.
Yes, very loose terminology. I was thinking of a combination of two events:
1. VWAP based trading of large positions, which creates assymetric momentum effects in volume and price (and which is then somewhat forecastable)
2. Not then recognizing that you are forecastable (as a result of playing the VWAP game). The extra information that someone who looks at intraday price relationships has over someone who doesn't. If you wade into the middle of a market that is short-run forecastable (eg it's trending downwards to a new level and the market maker/HFT guys are battling their battles), and you don't check as to whether it's short-run forecastable, then you're probably the patzy at the poker table.
No, I'm already in the finance industry. There are two main reasons I'm doing this:
- I think the finance industry is very closed when it comes to intellectual property development, and an open source approach can be seriously competitive. An open approach may well be the future when it comes to being 'connected'
- HFT is an interesting multi-disciplinary problem and the shear breadth of expertise required - modern chip design, low-latency software/hardware interaction, lock-free concurrency, fault-tolerant system design, adaptive learning algorithms, k-means clustering - means I'm learning heaps every day.
I just don't agree that HFT is a scourge. It's an ecological shift (neither good or evil) and longer-horizon investors need to evolve.
"HFT is an interesting multi-disciplinary problem"
This makes this a very worthwhile project: whether or not anything comes of it (I considered writing technical analysis trading software for the exact same reason).
Some documentation about the patterns and techniques you used would likely benefit the community a tiny bit more. Not everyone understands Haskell (or, forbid, has the time to peruse large codebases).
There are many other interesting multi-disciplinary problems that are not as contentious as HFT and probably more publically acceptable and worthwhile. But if HFT is the domain you care about, go for it.
> an open source approach can be seriously competitive
Do you really think any Joe Schmoe off the street could just grab your open source HFT and start making money with it? If not, how does your project benefit anyone?
Are you working in the industry, but not in HFT? Maybe this is project is just practise for getting a job in HFT? I imagine that's where programmers get the fattest paychecks in the world. Only a fraction of what the sociopaths running the show get, of course, but good money nevertheless.
Good points. To be more precise, I think that an open source approach to system research and design can be hyper-competitive versus closed-door secret-squirrel development run by a committee looking for short-term wins. This project benefits me firstly because I get feedback on my initial scratchings and maybe even collaboration on areas outside my comfort zone. And it could well enable me and others to avoid having to work for the sociopaths.
I am biased in thinking that monitoring the market and trading at low latency isn't fundamentally a bad thing, and shouldn't be taxed because others are too lazy to do the same thing. I certainly think that hft has been used by evil people to front-run unknowing third-parties, often in collaboration with middle men and women who turn a blind eye to the morality of their business models.
Others are two lazy? It's my understanding that the average Joe has no access to the hardware and low latency network connections to be able to do this on his own.
There is a rather large economic rent attached to HFT at the moment. If my project or others like it can eat into that rent a touch, it might not stay that way.
Exactly, if you take the position that gains from HFT are basically "stolen" profits from otherwise more productive market participants, then any that can be "stolen back" is a win.
> It's my understanding that the average Joe has no access to the hardware and low latency network connections to be able to do this on his own.
Yes, but for US equities this is purely economical: _anyone_ can buy colocated rackspace, direct market data feeds, and direct connectivity. The average Joe does not want to do this any more than he wants to build his own tennis-shoe factory.
Trading (as in HFT) isn't the same as investing. While the mechanics can sometimes be similar, a trader is different from an investor.
What HFT does is segment the market. It's probably best described as a form of arbitrage. Arbitrage is necessary, but in a good market hopefully it is pushed to some form of equilibrium.
HFT has an impact on the trading market. However, that doesn't automatically extend to investment.
> As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from.
This is the line peddled by the popular press but it isn't true. HFT works to reduce spreads, which benefits both buyers and sellers.
Latency arbing is a scourge, and makes limit orders practically useless. Quote stuffing, would be illegal if trading were still done on little slips of paper. Imagine dumping 10,000 slips of paper on the trading desk, and then shouting "just kidding!" In the words of Lawrence from Office Space, "You'd get your ass kicked" Or even worse, 10,000 empty bids. That's why people hate HFT.
I'm not trying to be argumentative, but I am curious why you find latency arb so problematic, and what you think it's impact is on limit orders?
I for one think latency arb is one of the bigger net wins for hft. As a market participant, each venue I have to maintain a presence at is a cost to me. I'm willing to pay the latency arb shops their cut to provide me price consistency because for my models it is much cheaper to do so than to continually reevaluate and redeploy to every possible venue. It frees me to shop for venues that provide the best features and fees.
As for quote stuffing, you are absolutely right it is awful. That's why almost every venue out there has taken or is taking steps to curtail it. They did so because their customers agree with you.
'Quote stuffing' is there, because of absolutely ridiculous SEC rules. If you'd put in fractional prices and remove NBBO rules, there wouldn't be any 'quote stuffing'. And latency arb. What's wrong with latency arb? How else can you move information from one exchange to another fairly?
Think of HFT's as of part of the financial network infrastructure. Where exchanges play role of nodes and HFT companies role of links/queues/buffers.
> I'm not trying to be argumentative, but I am curious why you find latency arb so problematic
That is a loaded term. "Latency arb" as you described it is HFT keeping all protected exchanges synchronized, and it is a good thing. It means that everyone else can ignore the 13 exchanges, and send their orders to the market with the most competitive pricing for connectivity. "Latency" arb" as described in most literature critical of HFT is the specific practice of submitting and canceling non-bona-fide quotes to an exchange with the intent of slowing down the matching engine. If you can slow down the matching engine that most other participants are using, you can effectively delay the public response your actions on the other 12 exchanges. A trading strategy that operates on the basis of a DoS attack on one exchange is definitely problematic.
What you are describing is quote stuffing. Nearly every participant in HFT/algo trading agrees that it is a problematic practice. Nearly every venue has either enacted or is enacting policies and procedures to either prevent it or severely penalize it, because that is what all of their customers want.
Yes. Sometimes it is referred to as latency arbitrage. Hence why people _nominally_ talking about that subject may, in fact, be talking about different things.
> Nearly every venue has either enacted or is enacting policies and procedures to either prevent it or severely penalize it, because that is what all of their customers want.
Regarding penalties to prevent or penalize it, I disagree. NASDAQ's policy uses a ratio weighted by distance from the top-of-book. There is no penalty for excessive order submit-cancel loops at the top-of-book. Their matching engine also operates in a fashion which specifically encourages cancel/resubmit loops at the top-of-book, in that they accept and subsequently display limit orders at a different price than submitted. If they were serious about preventing quote-stuffing, they could fix this simply by rejecting those orders. Presumably they either don't care (because their system doesn't get bogged down), or there is pressure from some big customers to keep the status quo here.
Quote stuffing is differentiated from plain old vanilla latency arbitrage by fact that the purpose of quote stuffing is to cause artificial latency to then take advantage of. IMO one can still make defensible arguments in favor of latency arbitrage when they have invested in faster technology than other market participants. There is certainly no defensible argument for causing the latency and then taking advantage of the problem you caused to extract a profit.
A) Top of book quote stuffing is a pretty dangerous game to get into. Can't confirm that no one is doing it, but I'd be surprised if it were a big problem.
B) NASDAQ is one of the better technology platforms and does not experience as much lag as the others, so it could be as you say, they just don't care.
C) the charges away from inside market are only one of several ways NASDAQ discourages excessive quoting. They also have tiered rebate pricing based on quote/fill ratios and PSX has a order rest time requirement.
So I guess if you were a shop that didn't mind playing with fire quoting top of book, and you never actually wanted to market make on NASDAQ, you could still quote stuff them.
I stand by my statement that venues continue to enact penalties to discourage quote stuffing and is such is not nearly the problem people make it out to be.
As a market participant, if the NASDAQ is not providing you with an execution platform to your liking (whether due to laggy matching or anything else) you are free to choose another venue and thanks to latency arb shops you are probably not going to pay much of a price premium to do it.
The biggest problem in all discussions of HFT/algo trading especially when related to internet forums and expose reporting is people using incorrect terms. We don't let people get away with it in other technical settings because it leads to unnecessary strife. I think the same thing applies to electronic trading.
No worries, I'm specifically talking about the folks that are constantly testing for buys with tiny (and fake) orders and incrementing the ask until they find the limit, then buying all they can, holding for only milliseconds and reselling at the higher price. Queue jumping tricks like that which really add no value.
I'd point out that a lot of the things that are going on in HFT, are rehashes of old trading scams, and either are or would be illegal if there was any adult supervision. I was tempted to say something about the SEC being left behind the technology, and doesn't understand it. But for that to be true, and have this crap go on for so long, they either need to be complete fools, or they benefit from the status quo.
>and it's a game only the biggest and best-connected players benefit from.
That's definitely true in the case of latency arbitrage. But here again. It's the exchanges themselves that have decided that some players get an advantage over others. As an example, latency arbitrage, and a whole host of other problems could be sorted out by putting traders on exchange-hosted virtual machines. http://www.dailyfinance.com/2010/06/05/rigged-market-latency...
If not virtual machines, there are other ways to fix these problems, but first you need to get the players to acknowledge that there is a problem.
Nanex is wonderful to read, but IMO they frequently misinterpret the data. Off the top of my head:
- They only use consolidated feeds for US equities, never direct market-data feeds. The consolidated feed necessarily contains less information that direct feeds (to satisfy more stringent bandwidth requirements), which masks some "interesting" effects of how the exchanges publish their data.
- They disregard that the CME feed publishes a fixed depth-of-book, and whenever they look at total liquidity in the book it can appear to flicker when deep levels fall "out" of the back of the book, even if liquidity is actually improving with the presence of a new inside level.
- They make a big deal about wholesaler matching only occurring when the consolidated book is not locked. Their rationale is that subpenny prices are always wholesalers, and (erronously) therefore a lack of subpenny-priced trades must mean a lack of wholesale matching.
These mistakes sound believable, but they do not hold up to any of scrutiny. Use their site to find interesting events, but be very careful about taking their conclusions at face value.
That's why I thought the DNQ iqfeed was a suitable choice - no consolidation at all. Would love to know what the "interesting" effects are (beyond the fixed book CME effect) - will be looking a bit harder at the stream to find them!
I'm curious about the "best trading platform" claim - what is going to make it better than any other?
Especially as the real High Frequency Traders are busy spending fortunes on placing their hardware as close to the exchange as physically possible.
Speed is only one issue with autonomous algorithm design. Yes, the speed thing grabs the headlines but the boiler-plate objective is to front-run the slower players. There's a wealth of opportunity in processing the event stream in a more robust way than others and faster too. Think semi-HFT, semi-autonomous.
Most trading platforms are primarily loss leaders for the 'professional' version and otherwise attached to a non-open source business plan.
You've hit a pet peeve of mine so sorry about the tirade. People use the term "front run" incorrectly and it is important to point this out. Front running is a very specific activity and is illegal. If there is evidence of front running people need to be prosecuted.
That said, I've worked on several systems and never seen any of them that would allow a faster market participant to see your order flow without your permission. This is what front running is.
People who don't understand the term seem to think seeing market data before other participants is front running it's not. There have always been and will always be some traders that see (and can react to) market data faster than everyone else. In the era of electronic trading this information latency cost is going down, not going up.
Yeah, IBrokers is great. R is even better! Everytime I need a specialist piece of code it seems to already exist as an R package and to have been released in the last few weeks.
The annoying thing about the IB API is that there's no framing; that is, you can't simply consume the message types you are interested in without parsing the entirety of every variable length messages.
ib-zmq resolves this annoyance by parsing incoming messages and placing them individually into ZeroMQ message frames.
I also wrote an alternative to the IBrokers R package with a much nicer interface using this ZeroMQ adapter. It parses most IB API messages, but hasn't been used in production yet.
I am curious about this project because I love this subject.
However, it is impossible to build an open source HFT system especially because it will be open source. The other sharks will know how you trade and they will only need to play against you by sending opposite orders every time. That is what happens when the source code of a big US bank were stolen. So it should be more appropriate to say it will be a fast trading system.
The other reason is you can't build a perfect HFT trading system because it's a moving target. It is the war in the micro seconde market. The sharks will send you wrong orders just to disrupt your system or they will manipulate the market.
And finally there is a hardware problem. To do a HFT trading system you need to rent your own severs inside the market, use FPGA cards, etc.
In conclusion, I only say there is a wrong goal to say "I will build the best open source HFT trading system" rather than "I will build the best open source fast trading system". This project is interesting (and there is Haskell cool!) so I will stay connected to see how it evolves.
I intend to publish and collaborate on private algos. Even if others have the code they still don't know where I choose to operate and deploy capital. And if they get rich on the code good luck to them - maybe they'll sponsor a dedicated server for 1.0
Yes you send your code on the servers of a company and you use their API to hit the market with their servers, or you rent your own racks. It will cost you a lot but in the HFT world it is the only solution. Any nano second counts. You are the fastest or you are not.
This could be a lot more fun than classic codewars.
For 99% of the "system" there's no particular reason you'd have to connect to the NYSE and trade real stocks for real money. Looks like they're feeding from iqfeed, but I think it would be huge fun to create an imaginary competition exchange, convince about 100 algo writers to compete, and shove them up against each other purely for the fun of it, see who's a better algo writer.
With modern virtualization, it should be pretty easy to distribute a pack of images, including practice data, to replicate an entire financial system on a small scale in your basement, then once you think you have a decent algo, upload your image to the competition league. If your hft codewars league wants limits for storage or cpu cycles, virtualization is a pretty easy way to implement it.
Superficially the most obvious thing to do is totally make everything up, from top to bottom, but it would probably work just as well to use real live market data and overlay on top of it. Just as air traffic control simulation has very slowly moved over the decades from purely random to sorta realistic to actual airport data.
I like your thinking V, and that's not too far away from one of our ideas. Take some algos that are representative of typical market makers and trader behaviour and fit the parameters so that the simulated trading starts to look like the live data. Maybe it's a better idea to use real algo makers.
If you need smart algorithms operating autonomously versus motivated enemies your go to must be open source. The security guys proved that a long time ago.
You're right in that the HF bit doesn't really fit but 'fast trading' doesn't have the same ring to it.
I don't think your security analogy applies to trading algorithms. Open source is good for security because it aids in the discovery of vulnerabilities. However, good trading algorithms are usually reliant on some form of information asymmetry. If you make a discovery and develop a trading algorithm to exploit it, I would think it would be advantageous to keep it a secret for as long as possible. This is not my field of expertise so please correct me if I'm misunderstanding your analogy.
Edit - I love the project by the way. As others commented already, there's a really strong argument to be made in favor of open sourcing the infrastructure side of this. And that appears to be where most of your effort is focused (e.g. open infrastructure, private trading algorithms).
Thanks Dan, the analogy is a stretch maybe, but most algorithms fail due to being narrow models of reality, and criticised by too few eyes. If I discover something, it's extremely unlikely that others will believe it anyway (unless I prove it by becoming filthy rich as a result in which case I'm just fine).
You're pretty right about good trading algorithms but that doesn't necessarily apply to good traders or good trading systems. A good system - hft or otherwise - needs to be flexible and evolve to stay in touch with an evolving ecology. And to do this, it needs to keep track of many different trading ideas and algorithms, each of which may have its day in the sun.
The algo is only one part of a real trading organization. Granted it's the one that gets all of the credit, and is perceived as the sexy side but operations and reporting is easily as important. If you had the best algo, but you kept making mistakes or mis-reporting because your platform was crap, you'd be out of business as fast as if your algo didn't make money. I'm talking here of hedge funds where assets under management are all import and generally external. It might be a different story in a market maker or prop shop where getting research into production fast is all important.
This could be an interesting course for people wanting to know more about 'computational investing' and the algorithms behind stock (market) analysis and trading: https://www.coursera.org/course/compinvesting1. Not specially targeted at HFT, but interesting basic background information nevertheless.
I thought so too for a long while, then I tried to do a touch of concurrent code in c++ and had to gouge my eyes out. I'm excited about the speed up haskell brings to development. You can plan things in haskell you can't imagine in other languages.
Interestingly I had this exact conversation last week with a guy from an HFT firm that was asking about how do you write fast code in haskell. Haskell won't be as fast as C, but we are not a high frequency shop so it doesn't matter. Much time in engineering at an HFT firm is spent doing things like turning an L2 cache miss into an L1 cache miss. Haskell makes it hard to do things like that.
Our platform is fast and could be used for some slower strategies. If we replaced pieces of it with C we could be competitive but probably never less then 10 microseconds wire to wire.
I'm building an open source matching engine. I would love to pair up the two systems for a stress test. I'll keep tracking your project and ping you again when I have a system up a running if you're interested.
Yeah, Go is a great language. However, I am trying to develop some low level queues that should be faster, and less generic, than the ones that Go provides. But I am having some difficulty putting memory fences into my code.
But if I can get a reasonable level of performance out of the standard Go queues then I will just use those in the meantime.
I see a comment about ITCH in there... Are you modeling your interface after NASDAQ's ITCH/OUCH interfaces? If so, I think it would be interesting to look at the protocol specifications for all of the other major US equity exchanges. I think they are all freely-downloadable.
I was using ITCH data to try to test the matching algorithm. However, I found that the number of hidden trades made it fairly poor for validation, which was what I was most interested in.
I _was_ using the ITCH format for messages and switched away from it. I may revisit that decision in the future.
I agree that it would be really interesting to look at the messaging protocols of real exchanges, and I have made a quick stab at it. But I am focused on getting something very simple working right now.
If you wanted to play around with ITCH files the executable generated in the itch directory will run any ITCH file through the matching engine sort of like a debugger, allowing you to step forwards and backwards in time and see how the internal state of the matching engine changes.
No, and I am making no claims. I have a market event feed coming in, a good idea of what the event processing looks like and a rough idea of how to send an order to a broker. I think the project needs to get to production fast though.
Have you considered python for the research/algo side of things?
I work in a hedge fund in London at the moment and there's a massive shift away from R towards python in the algorithmic shops that I know about here. There's also the great work that quantopian are doing on their backtesting framework zipline [1]
Yes, I am aware of that trend and considering a switch. I'm personally more used to R is all, but if I found collaborators I'd jump to where the code base goes.
The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.
I see the research as production. There's an autonomous system processing the event stream and deciding on trades that's hands free. I see that as a haskell codebase with C speedups for critical stuff, with no R anywhere to be seen.
But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that.
I've had good success so far with using R as a fast column database, solving big data hassles, but there may be better solutions in that space.
I've had a quick read through the .org file now and it appears that you're using R for things like moving averages over the stream of prices. Have you considered using one of Haskell's FRP (or pipes/conduit) libraries for this side of things?
Replying here as I can't add on another reply below
The way we do our monitoring/UI is with zeromq/protocol buffers as an external surface to the trading system (basically a PUB socket) and a very thin bridging and translation layer to WebSockets and JSON in python. That way you get to use d3 or anything else on the front end. We use backbone, knockback and d3 with coffeescript.
Thatsvery similar to what I/we use to build our realtime network monitoring tools, swapping out python for PHP (regrettably). It's validating to see other shops in different industries zeroing in on the similar stacks.
zygomega, I'm one of the zipline maintainers and we'd love to collaborate with you. There are a few people doing HFT research with zipline, and there's a lot of work to do. At quantopian (my day job), we focus on longer hold periods, so there is room in the zipline ecosystem for you to do HFT.
The main benefit we've found with python as the algo language is that it allows for stat programming with pandas, but also OO or functional programming for the algo logic. This smoothes the transition from research to production, just as you're describing with R -> haskell, but you can stay in one language.
I think one of the biggest potential wins with parallelization is if you can assume all positions are closed overnight, most often true for HFT. That way, you can simulate all the trading days in a test range in parallel. This is quite similar to the parallel processing we do to handle the large number of concurrent backtests running at quantopian. We did all of that with python, but I'd be fascinated to see it done with haskell.
I would see haskell very much as the plumbing side of things. The tools available for handling and reasoning about streams are streets ahead of anything I've seen elsewhere. With zeromq and protocol buffers (that's what we use in our stack) you could very nicely separate the plumbing of the data from its consumption. I'd love to see something like this as well!
How would you handle the position sizing part of the algo if you're testing all days in parallel? Wouldn't the trade size depend on the all of the previous day's PNL?
Hi fawce, I'm a big fan of quantopian but didn't realize that zipline was a separated project. Will have a good scout around the project and see what you guys are up to :)
It is a Lisp, semantically, with a MATLAB-like syntax (including all the linear algebra sugar), and a LLVM back end.
The stated goal of the language is to give to scienctific programmers the convenience of high level languages for prototyping, yet the speed of low level ones. It has multiple dispatch, based on a powerful type system, homoiconicity and macros (optionally hygienic).
The package collection is growing steadily [1]. There's also a mailing list dedicated to doing statistics with Julia[2].
yes. Julia looks like a quality language but suffers the usual new kid on teh block hassles - very few packages so you have to spend too long rolling your own.
Are there specific packages that you would currently need, but aren't there (yet), or is it a more general concern about the relative package scarcity and unforseen changes to / future needs of your code?
It's not specific, it's just my recent experience. Hadley released his bigvis package a few weeks ago, and it contains the beginnings of a weighted average routine I sure can use.
mmap is a package that let me create a fast db from scratch in a day. ggplot, lattice and all the charting routines are a joy to work with.
I'm interested in random forests today, and I bet there's a package that links to the standard libraries whatever they are. This is also probably true for python, but Julia you'd have to build the API.
i've looked at how it compiles to llvm, and it basically punts any optimization to the llvm side, aside from the most basic tracing jit method monomorphization/specialization.
(as in, I spent part of an afternoon end of last week skimming the entire code base).
This is troubling, because when you don't design your language to make it "manageable" to do static analysis / optimization on, it be becomes DIFFICULT to add that capability later. Julia is about as dynamic as JS or Scheme or Lua, plus it has all those runtime type tag cases in code... but whereas JS has relatively unbounded manpower (relatively!) to make a fast JIT, whereas lua is a small enough language that we have the amazing impressive LuaJIT, whereas Scheme is designed with very thoughtful semantics / specifications in mind, Julia lacks.
more succintly, Julia lacks a clear enough thoughtful choice in static/dynamic semantics for the pre LLVM side to have an easy optimization story given a small sized core team, such optimization engineering will either take a long time to develop, or will require some commercial entity to sink serious capital into writing a good JIT / optimizer. LLVM is a good backend, but it can't leverage the semantics of the input language, just the semantics of the LLVM bitcode you provide. You really really can't punt on optimization on the before LLVM side.
also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) )
Let me repeat: Julia doesn't have clear static semantics / phase distinction, and doesn't really seem to do anything beyond method specialization before passing code on to LLVM. This means it can't get compelling performance compared with any other LLVM backed language.
LLVM is amazing, and i use ghc -fllvm exclusively, and while theres some really really awesome optimizations that LLVM does for haskell code (for example), a HUGE amount of high level code performance is really depending on the language (any language!) doing optimizations before passing things onto the LLVM.
I didn't look at the implementation, and would be out of my depth if I were to. I'll relay your concerns to the authors (I share some of them, actually), but, meanwhile, here are a my thoughts:
The fact is that the performance of Julia is already quite good. Julia is semantically much closer to C than Haskell is, and it needs fewer optimisations to get reasonable speed.
The necessity of static analysis for performance is a myth that's been debunked for a while now. From the language semantics, I think that the runtime model of Julia must be about as complex as Lua's, including the LuaJIT FFI (swap metatables with type-based multiple dispatch).
As you said, though, whereas LuaJIT has its own tailored bytecode, Julia compiles to LLVM IR, and potentially useful information is lost in the translation process. LLVM-Lua is reportedly 2-3x slower than LuaJIT. I suppose that type specialized traces and immutable types help somewhat.
An area that clearly needs improvement is vectorised code. Currently, an new vector/matrix is allocated for each intermediate step. Bad. This is being worked on.
> also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) )
You're conflating type system with static type checking. Julia is dynamically typed, but its programming model (the UI of sorts) is built around the type system and multiple dispatch, it is one of its main features of the language, actually.
Thanks for checking Julia out. I fear there is a fair amount of misinformation in your comment, however, so it seems that a few things were lost in your reading of Julia's code (some of which is admittedly quite tricky). I hope you don't mind if I address some of it.
The first point is that Julia already has excellent performance on a par with most compiled languages, including, e.g. Haskell, whether they are using LLVM or not. Straight-forward Julia code is typically within a factor of two of C. That's shown in the microbenchmarks on Julia's web site [http://julialang.org/], but, of course, that's not entirely convincing because, you know, they're microbenchmarks and we wrote them. However, similar performance is consistently found in real-world applications by other people. You don't have to take my word for it – here's what Tim Holy [http://holylab.wustl.edu] has to say:
https://groups.google.com/d/msg/julia-users/eQTYBxTnVEs/LDAv.... Iain Dunning and Miles Lubin also found it to be well within a factor of 2 of highly optimized C++ code when implementing realistic linear programming codes in pure Julia: https://github.com/JuliaLang/julia-tutorial/blob/master/Nume.... The benchmarks appear on page 7 of their presentation.
This statement about Julia's high-level optimizations is entirely wrong:
> i've looked at how it compiles to llvm, and it basically punts any optimization to the llvm side, aside from the most basic tracing jit method monomorphization/specialization.
Julia does no tracing at all, so it's definitely not a tracing JIT. A relatively small (but growing) very crucial amount of high-level optimization is performed on the Julia AST before generating LLVM code. In particular a dynamic dataflow-based type inference pass is done on the Julia AST. Since Julia is homoiconic, this type inference pass can be implemented in Julia itself, which may be why you missed it: https://github.com/JuliaLang/julia/blob/master/base/inferenc.... Don't be fooled by the briefness of the code – Jeff's dynamic type inference algorithm is one of the most sophisticated to be found anywhere; see http://arxiv.org/pdf/1209.5145v1.pdf for a more extensive explanation. It's also highly effective: 60% of the time it determines the exact type of an expression and most of the expressions which cannot be concretely typed are not performance critical [see section 5.2 of the same paper]. You are correct that we leave machine code generation to LLVM – after all, that's what it's for – but without all that type information, there's no way we could coax LLVM into generating good machine code. Other very important optimization passes done on the Julia AST include aggressive method inlining and elimination of tuple allocation.
> more succintly, Julia lacks a clear enough thoughtful choice in static/dynamic semantics for the pre LLVM side to have an easy optimization story given a small sized core team, such optimization engineering will either take a long time to develop, or will require some commercial entity to sink serious capital into writing a good JIT / optimizer.
There is a very clear and thoughtful choice in static vs. dynamic semantics in Julia: all semantics are dynamic; there are no static semantics at all. If you think about your code executing fully dynamically, that is exactly how it will behave. Of course, to get good performance, the system figures out when your code is actually quite static, but you never have to think about the distinction. And again, Julia already has excellent performance, and we have accomplish...
I'm also in New York if you want to chat about it over a beer some time. I doubt I can convert a hardcore Haskell programmer on my "scruffy" unchecked, dynamic point of view, but we might have a good conversation about it. We've occasionally quipped that Julia drags Matlab programmers halfway to Haskell, so maybe there's some common ground.
Extremely helpful - this really clarifies the differentiation in the Julia approach and I'm excited your team is taking this direction. There's a lot of people cheering Julia on, even if we're wimping ut remaining on the sidelines.
It would help if this explanation was a bit more prominent.
The notion that Julia is not thoughtfully designed, and therefore we won't have the resources to make it run fast, is close to being the opposite of the main idea of the project. We don't want to go to extreme lengths to get performance, so we leverage design.
For example, one of the biggest challenges in dynamic type inference is getting the types of fields of mutable heap-allocated objects. We avoid this debacle by letting you declare field types, which also has benefits for code clarity and specifying memory layout. Everybody wins. But once you are writing types, you need fairly flexible types, to avoid being stuck with only Int and Any (everybody's favorites). We then follow the implications of this as far as we can.
As for "all those runtime type tag cases", the key is that they form a lattice, which feeds nicely into dataflow analysis. The lattice-theoretic properties of a language's universe of objects typically do not get enough attention, especially in dynamic languages, where it is actually most needed. Typically the lattices are either trivial (for example, scheme's fixed set of types), or highly uncooperative. You want to hit a sweet spot where you can compute greatest lower bounds that are actually somewhat interesting.
I haven't worked in anything as high performance as this (though I've been doing lots of number crunching in Python recently) but there are a couple of great libraries that you can use before you jump into Cython. Check out Numba [1] and numexpr [2]
Python is definitely preferred on the research rather than the production side generally. Some organisations do use it on the infrastructure side too, but these guys are algorithmic traders, and not HFTs so latency isn't that big a deal.
WRT to the problem of deploying prototyping/research code, there are the following unsolved problems that I'm aware of:
* Going from matrix operations over the whole timeseries (taking care to avoid problems with your algo looking ahead) for speed in a research setting to deploying to an environment that streams updates to the timeseries one at a time. I think that this is an area that haskell has the potential to excel at, given it's strong guarantees on structure.
* Concurrency. The options in python all suck to some degree - especially if you have to interact with C libraries or extensions. I don't think it will ever make sense to build your real time market date in python. Again here haskell has an advantage.
However the python ecosystem seems to be almost perfect for researchers:
* Excellent and flexible data slurping/munging/transforming.
* numpy, scipy, pandas, theano, scikits... 'nuff said
> Going from matrix operations over the whole timeseries ... to deploying to an environment that streams updates to the timeseries one at a time. I think that this is an area that haskell has the potential to excel at, given it's strong guarantees on structure.
python is a better toolset than R maybe, but the R problem domain seems broader in the last few months anyway.
My interest is in monitoring thousands of algorithms in real time directly within the messaging environment, and before the data hits a database. That type of concurrency is where haskell can muscle up and do the job.
Some of the work that continuum have done on blaze [1] look to be tackling the problem of streaming in python too. In fact some of the ideas in this library come directly from haskell and one of the main developers is Stephen Diehl, who posted the very popular "what I wish I knew about haskell" slides recently.
Stephen here, yes boothead is right. Using a combination of Haskell and Python you can make a really powerful trading system. Python has a lot of user-facing algorithm tools and Haskell has the robustness and parallelism for the backend that Python doesn't.
If you're interested in advice on how to bridge the two worlds let me know, there's a lot of of upcoming technology ( LLVM, Blaze, pipes, zeromq, cloud-haskell ) that could be very useful.
By putting it on FPGAs maybe. Stage one is to get a robust process for measuring and researching latency - I'm in Australia and US data is bouncing into my mac via a lousy connection and a tcp port in 250 millsecs on average. There's a long way to go.
I suspect that if you can keep within a bulls roar of the low-latency crowd the big gains will be on the algorithm processing side. Algorithms still add up a few million numbers every time they recalculate a moving average - there's better ways to do it.
Isn't the barrier to HFT the fact that you need enough capital so that your profits cover the cost of co-located servers and FPGAs in the exchange datacenter, without which you have a latency handicap? (in addition to, of course, coming up with good algorithms)
Sorry to burst your bubble but high-frequency trading is only viable and necessary for market makers.
The end result of any type of algorithm you think of is always a curve-fitting function of existing data.
No need to apologise - curve fitting is a big issue. But I would think that everything that happens on kaggle would be curve fitting by your definition right? And how is the stuff that happens in brains something other than curve fitting? It's how you connect the data dots and what you choose to focus on that makes the difference.
1. Timestamps in your seed data might benefit from nanoseconds if you really talking about "high" frequency.
2. I agree with you comment that it is easier to think about concurrency in Haskell than in something like C++, however you can't really compete with C/C++ in Haskell. Not even with cgo (Go packages that call C code), not with OCaml or any other higher level beasts that promise the speed. Fortran would be the only one faster for the "algo" part of your initiative. But again, if this is just an exercise, Haskell and others (I prefer Clojure for example :) will do just fine.
3. Would make sense to split the "platform" in two (very different) parts: "Quantitative Analysis" (a collection of tools and rules) and "Technical Glue to Read and Stream". Each can/should be divided further of course, but the two above are essential yet very different for a true "HTF Platform".
I wish I could get down to nano units. iqfeed (a good value feed) just got millisecs in so will settle for that.
I'm preparing some speed tests between C++ and haskell on an identical block of processing so stay tuned! You might be surprised - haskell is way ahead of clojure on compiler smarts.
The split you suggest is exactly what I think is wrong with the way things get done right now. I'd like to integrate the quant inside the read and stream - now that's potentially a large speed up that might compensate a tight budget.
Keep in mind that if you're not an experienced C/C++ coder, you're going to be (largely) benchmarking your relative ability in either language rather than the intrinsic speed of each language.
I would be much more interested in an open source solution to fight against HFT, or at least make it more difficult.
I see the financial world as a necessary evil, and I think some of its parts have benefits for the rest of the world. HTF is not one of those parts; I see it as a pure burden.
You could have made an open source Patent Trolling system and I would feel the same.
Wow, that's a pretty low ranking being shoved into the patent troll category of evilness. When someone goes and patents an obvious algorithm at some point in the future and my little project shows up as prior art would I be redeemed :)
Aside from all the other points raised against this, there's also the fact that the timing on HFT is so close that it requires close colocation of the physical hardware to the trading systems. Only big boys get to have access to those racks.
I have no experience in the financial industry, but HFT has always fascinated me as a potential source for very high rates of "events". Could you share your general insight about the sheer volume of data that commonly gets pushed through an HFT system? I'd also be terribly interested in a multi-megabyte/gigabyte "recording" of HFT trade data.
This project is not HFT. It does not use a full book data feed. Really even the feeds that come from the exchanges are not precisely timestamped enough and HFT firms stamp their own . The data is not huge although trading system need to be able to deal with large spikes in the rate of events. Usually now people are using 10G/40G/Infinibad as a connect to the matching engine.
So, yes, on my toy wannabe hft feed, I'm currently clocking about 10 million ticks per day on about 8 futures tickers I'm tracking. The raw feed is between 1 and 1.5 gig per day. I can imagine the petabyteness of the bigger guys.
This is a nice little research project, and I hope you learn a lot from it. Having written algorithmic trading systems I think you are missing on a couple of central points:
1. There is no such thing as the "best" algorithmic trading platform, because algorithmic trading is such a broad term. Architectures that make sense for one class of trades does not make sense for other class of trades.
2. Contrary to popular opinion on this forum, algorithmic trading is not necessarily only low-latency and high-frequency. Frequency, latency, and algorithmic levels are really dials on the specification system and along with cost dictate how you will build the system. Do some work figuring out the trades you are targeting before setting those dials.
3. The actual execution system part of an algo trading system is usually the easiest part. If you've written one before, writing new ones is usually trivial. Finding the trade, building appropriate risk systems/practices, building back testing frameworks and exchange reference data systems are all much more challenging and take the majority of the time in these systems.
4. Finally, if you are truly targeting low-latency and high frequency events concurrency is your enemy, not your friend.
To everyone saying that the cost in these systems is the cost of co-locating servers or fpga cards, you're wrong. You can get hosting deals/leases on those kinds of things for the same cost as high-end web hosting. The cost of running these systems are 2 fold: 1 paying the employees (because your competitors can pay them a lot) and 2 having deep enough pockets to survive the bad days/weeks/months. These are the same costs that have always existed in the trading space and have nothing to do with electronic trading. In fact, electronic trading has lowered the information asymmetry and made it easier for new participants to be involved in the markets.
> ...truly targeting low-latency and high frequency events concurrency is your enemy, not your friend.
This reminds me of the LMAX financial trading platform where they started with a concurrent model but ended up using a single-thread "...that will process 6 million orders per second..."
Yes, it is interesting, when you are becoming highly concurrent the penalty for context switching start taking its toll, but that's only when the rest of your trading system is already optimized.
Yes, but the concurrency has been shifted to the two disruptors at either end of the business process logic. And I suspect the business logic is fairly simple and lends itself to the straight through solution.
If you want to react to a news event (say), you have to somehow send a signal to the feed requesting recent history (last 20 minutes say), have something waiting to look at it, crunch the numbers when they arrive to see if somethings up, then interupt the existing trade decision/risk management schedule with a potentially better trade. Complexity increases quickly and concurrency helps do all of that at high speed.
As your complexity curve increases, your latency expectations must go down. That is what I meant when I said there is no single trading system that can be the "best".
If what you are interested in is complex decision making then it may make sense to use a different sort of messaging technology than LMAX, but you won't be getting into anything remotely "low latency". Nothing wrong with that, just needs to be a known expectation.
Yep, I've poured over the technology and concepts there and its awesome. So awesome I think you can increase complexity without too much of a latency penalty. Just behind the fast guys but way ahead of the straight algo guys is where I think the opportunities lie (I'm kind of answering your previous question about where the dials sit in my mind). I might be wrong but, and would like to test out all dial positionings at once before locking in. And that's where a flexible haskell version could shine.
I have no opinion about the choice of haskell. That said, you are already making decisions about the dials if you go in with an architecture that relies on massive concurrency, functional languages, etc. You cannot for instance get the latency dial very low with that central architecture decision.
Again there is nothing wrong with that. It's just better to state it up front as an expectation and/or a goal than to assume you are going to be able to pivot easily once you have an architecture in place.
I can't recommend working with the LMAX library enough if you are interested in low latency in the JVM. Lots of people have driven to the same/similar place but LMAX was the first to just let everyone see it.
I'm probably thinking of algos more generally and generically. For example, whatever the class of trades, I think that what you are doing is making a stochastic forecast of future price movements. So maybe there's a good separation possible between your algorithms that create a distribution and algos that determine how to trade once you have that distribution. But saying this probably reveals a bias I have to think statistically.
And I totally agree the majority of the system build thats important is the boring stuff. An algorithm should be able to tell if the market feed has died or the market shuts down unexpectedly, right? And know what to do next.
And good to hear that someone thinks the costs are much less than popular opinion suggests.
I worked on a framework for managing stock and options positions based on feedback methods developed by Stafford Beer. It doesn't do the algorithmic/transaction part, but rather focuses on providing reporting and alerting for large hierarchies of institutions, accounts, stock and options positions. The proprietary algorithmic code could be added by subclassing the code that I wrote. It's called the Viable System Agent. It's written in Smalltalk, tested under both Squeak and Pharo. It's licensed under the BSD license. It can be found at:
197 comments
[ 7.0 ms ] story [ 307 ms ] threadAs you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from.
Long term investing isn't affected by HFT, except in providing more liquidity at the point you want to exit any position. If you're planning on holding for 5+ years you're competing against relatively few people and can do good research. Any shorter timeframe and you're competing against millions, against computers and against unknowables that will distort the price in the short term.
Why would short term investing be bad, and how long do you think HFT algorithms hold their "investments"?
I hope you're not just trying to blow smoke up my ass here.
Short term investing is "bad" because you're not investing in the future success of the company you're trading on the basis of what you think the market is going to do. To the extent to which that is ever knowable it is unlikely you have the skill, experience and data to be able to beat the large number of professionals doing it.
> The HFT trader
You say it like it's a person making trades.
> I've benefited because I can buy/sell at a price close to the quoted exchange price, which didn't use to be the case.
What's a "quoted exchange price" if it's not the price you actually pay?
> Short term investing is "bad" because you're not investing in the future success of the company you're trading on the basis of what you think the market is going to do.
Isn't this what HFT is all about?
Trading on insider information is bad, of course. But if you're on a level playing field, I don't see anything wrong with profiting from something you correctly predicted short-term.
The quoted exchange price is an average of the buy/sell offers available, there is no guarantee if you try and trade that you will pay that amount. Especially if you are shifting large blocks of stock.
There is nothing wrong with "short term predicting" but it isn't investing. It is often a very easy way to lose money, "bad" doesn't just mean morally so. It comes down to time horizons, how much you want to make, and how many people are also trying to predict that event. You can correctly predict that event and still not make money as it is already priced in.
The actual price you have to pay is based on how much you want to buy and how much other people are willing to sell at what price.
People who want to buy give a "bid" price and people who want to sell give an "ask" price. Whenever there is an ask price that is lower than a bid price, a sale takes place. This results in the lowest ask price always being higher than the highest bid price. The quoted exchange price will be somewhere in between the two prices.
If you want to sell right away, the most you can get is the highest bid price on the books. If you want to buy right away, the cheapest you can get it is the lowest bid price on the books. This means that you have to pay more than the quoted exchange price to buy and receive less than the quoted exchange price to sell.
The mean of the bid and the ask, generally. For a daily close price it can be a fifteen-or-so minute average thereof.
And managers turn over long-term investor portfolios quickly. The average holding period for an SP500 company shareholder is 100 days.
And every time a trade happens you run the risk of getting clipped by the faster guys who see you coming.
I think it is natural, but fallacious, to apply line-of-sight properties to trading. If you plan to trade, then there are two ways that another participant can "see you coming": 1. If you don't have direct market access, your order gets routed through a broker. The broker sees your order before it hits the market, and if he jumps in line ahead of you then that is front-running, and a Bad Thing. Your broker can get in a lot of trouble for this sort of thing. 2. If you are trying to move a large position by sending multiple orders to the market (one after another), then all market participants have the chance to react to the first order. It's really a game to try to move a lot of inventory at once, without tipping your hand to anyone else in the room. Thems the breaks. Nobody else gets to see anyone's order before the matching engine has already processed it, so there's no way to jump "ahead" of it.
OTOH, maybe your long-term investor is trying to time the market: wait for a signal intraday, and pick that moment to send an order. In that case, if it is a good intraday signal then it is likely that someone else will compete. It is unlikely for a long-term trader to have spent as much on infrastructure as a HF trader, so the juicy signals will result in missed executions that _look_ like front-running.
1. VWAP based trading of large positions, which creates assymetric momentum effects in volume and price (and which is then somewhat forecastable)
2. Not then recognizing that you are forecastable (as a result of playing the VWAP game). The extra information that someone who looks at intraday price relationships has over someone who doesn't. If you wade into the middle of a market that is short-run forecastable (eg it's trending downwards to a new level and the market maker/HFT guys are battling their battles), and you don't check as to whether it's short-run forecastable, then you're probably the patzy at the poker table.
I just don't agree that HFT is a scourge. It's an ecological shift (neither good or evil) and longer-horizon investors need to evolve.
This makes this a very worthwhile project: whether or not anything comes of it (I considered writing technical analysis trading software for the exact same reason).
Some documentation about the patterns and techniques you used would likely benefit the community a tiny bit more. Not everyone understands Haskell (or, forbid, has the time to peruse large codebases).
Best of luck to you!
Do you really think any Joe Schmoe off the street could just grab your open source HFT and start making money with it? If not, how does your project benefit anyone?
Are you working in the industry, but not in HFT? Maybe this is project is just practise for getting a job in HFT? I imagine that's where programmers get the fattest paychecks in the world. Only a fraction of what the sociopaths running the show get, of course, but good money nevertheless.
> I just don't agree that HFT is a scourge.
Good thing you're not at all biased.
I am biased in thinking that monitoring the market and trading at low latency isn't fundamentally a bad thing, and shouldn't be taxed because others are too lazy to do the same thing. I certainly think that hft has been used by evil people to front-run unknowing third-parties, often in collaboration with middle men and women who turn a blind eye to the morality of their business models.
Yes, but for US equities this is purely economical: _anyone_ can buy colocated rackspace, direct market data feeds, and direct connectivity. The average Joe does not want to do this any more than he wants to build his own tennis-shoe factory.
> As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from.
Good thing you're not stating opinion as fact.
What HFT does is segment the market. It's probably best described as a form of arbitrage. Arbitrage is necessary, but in a good market hopefully it is pushed to some form of equilibrium.
HFT has an impact on the trading market. However, that doesn't automatically extend to investment.
I for one think latency arb is one of the bigger net wins for hft. As a market participant, each venue I have to maintain a presence at is a cost to me. I'm willing to pay the latency arb shops their cut to provide me price consistency because for my models it is much cheaper to do so than to continually reevaluate and redeploy to every possible venue. It frees me to shop for venues that provide the best features and fees.
As for quote stuffing, you are absolutely right it is awful. That's why almost every venue out there has taken or is taking steps to curtail it. They did so because their customers agree with you.
Think of HFT's as of part of the financial network infrastructure. Where exchanges play role of nodes and HFT companies role of links/queues/buffers.
That is a loaded term. "Latency arb" as you described it is HFT keeping all protected exchanges synchronized, and it is a good thing. It means that everyone else can ignore the 13 exchanges, and send their orders to the market with the most competitive pricing for connectivity. "Latency" arb" as described in most literature critical of HFT is the specific practice of submitting and canceling non-bona-fide quotes to an exchange with the intent of slowing down the matching engine. If you can slow down the matching engine that most other participants are using, you can effectively delay the public response your actions on the other 12 exchanges. A trading strategy that operates on the basis of a DoS attack on one exchange is definitely problematic.
Yes. Sometimes it is referred to as latency arbitrage. Hence why people _nominally_ talking about that subject may, in fact, be talking about different things.
> Nearly every venue has either enacted or is enacting policies and procedures to either prevent it or severely penalize it, because that is what all of their customers want.
Regarding penalties to prevent or penalize it, I disagree. NASDAQ's policy uses a ratio weighted by distance from the top-of-book. There is no penalty for excessive order submit-cancel loops at the top-of-book. Their matching engine also operates in a fashion which specifically encourages cancel/resubmit loops at the top-of-book, in that they accept and subsequently display limit orders at a different price than submitted. If they were serious about preventing quote-stuffing, they could fix this simply by rejecting those orders. Presumably they either don't care (because their system doesn't get bogged down), or there is pressure from some big customers to keep the status quo here.
So I guess if you were a shop that didn't mind playing with fire quoting top of book, and you never actually wanted to market make on NASDAQ, you could still quote stuff them.
I stand by my statement that venues continue to enact penalties to discourage quote stuffing and is such is not nearly the problem people make it out to be.
As a market participant, if the NASDAQ is not providing you with an execution platform to your liking (whether due to laggy matching or anything else) you are free to choose another venue and thanks to latency arb shops you are probably not going to pay much of a price premium to do it.
The biggest problem in all discussions of HFT/algo trading especially when related to internet forums and expose reporting is people using incorrect terms. We don't let people get away with it in other technical settings because it leads to unnecessary strife. I think the same thing applies to electronic trading.
I'd point out that a lot of the things that are going on in HFT, are rehashes of old trading scams, and either are or would be illegal if there was any adult supervision. I was tempted to say something about the SEC being left behind the technology, and doesn't understand it. But for that to be true, and have this crap go on for so long, they either need to be complete fools, or they benefit from the status quo.
>and it's a game only the biggest and best-connected players benefit from.
That's definitely true in the case of latency arbitrage. But here again. It's the exchanges themselves that have decided that some players get an advantage over others. As an example, latency arbitrage, and a whole host of other problems could be sorted out by putting traders on exchange-hosted virtual machines. http://www.dailyfinance.com/2010/06/05/rigged-market-latency...
If not virtual machines, there are other ways to fix these problems, but first you need to get the players to acknowledge that there is a problem.
http://www.zerohedge.com/article/do-it-yourself-latency-arbi...
Nanex has done a lot of good analysis of HFT, some of which is published here:
nanex.net/FlashCrash/OngoingResearch.html
- They only use consolidated feeds for US equities, never direct market-data feeds. The consolidated feed necessarily contains less information that direct feeds (to satisfy more stringent bandwidth requirements), which masks some "interesting" effects of how the exchanges publish their data.
- They disregard that the CME feed publishes a fixed depth-of-book, and whenever they look at total liquidity in the book it can appear to flicker when deep levels fall "out" of the back of the book, even if liquidity is actually improving with the presence of a new inside level.
- They make a big deal about wholesaler matching only occurring when the consolidated book is not locked. Their rationale is that subpenny prices are always wholesalers, and (erronously) therefore a lack of subpenny-priced trades must mean a lack of wholesale matching.
These mistakes sound believable, but they do not hold up to any of scrutiny. Use their site to find interesting events, but be very careful about taking their conclusions at face value.
Edit: bullet-list formatting
Most trading platforms are primarily loss leaders for the 'professional' version and otherwise attached to a non-open source business plan.
That said, I've worked on several systems and never seen any of them that would allow a faster market participant to see your order flow without your permission. This is what front running is.
People who don't understand the term seem to think seeing market data before other participants is front running it's not. There have always been and will always be some traders that see (and can react to) market data faster than everyone else. In the era of electronic trading this information latency cost is going down, not going up.
Edit: Ah. It now works only with JS on when putting the icon-name into <i> and not in the class. Sorry for that, seems to be new in Bootstrap.
It'll get you up and running with IB in no time.
https://github.com/brotchie/ib-zmq
The annoying thing about the IB API is that there's no framing; that is, you can't simply consume the message types you are interested in without parsing the entirety of every variable length messages.
ib-zmq resolves this annoyance by parsing incoming messages and placing them individually into ZeroMQ message frames.
I also wrote an alternative to the IBrokers R package with a much nicer interface using this ZeroMQ adapter. It parses most IB API messages, but hasn't been used in production yet.
https://github.com/brotchie/r-zerotws
However, it is impossible to build an open source HFT system especially because it will be open source. The other sharks will know how you trade and they will only need to play against you by sending opposite orders every time. That is what happens when the source code of a big US bank were stolen. So it should be more appropriate to say it will be a fast trading system. The other reason is you can't build a perfect HFT trading system because it's a moving target. It is the war in the micro seconde market. The sharks will send you wrong orders just to disrupt your system or they will manipulate the market. And finally there is a hardware problem. To do a HFT trading system you need to rent your own severs inside the market, use FPGA cards, etc.
In conclusion, I only say there is a wrong goal to say "I will build the best open source HFT trading system" rather than "I will build the best open source fast trading system". This project is interesting (and there is Haskell cool!) so I will stay connected to see how it evolves.
For 99% of the "system" there's no particular reason you'd have to connect to the NYSE and trade real stocks for real money. Looks like they're feeding from iqfeed, but I think it would be huge fun to create an imaginary competition exchange, convince about 100 algo writers to compete, and shove them up against each other purely for the fun of it, see who's a better algo writer.
With modern virtualization, it should be pretty easy to distribute a pack of images, including practice data, to replicate an entire financial system on a small scale in your basement, then once you think you have a decent algo, upload your image to the competition league. If your hft codewars league wants limits for storage or cpu cycles, virtualization is a pretty easy way to implement it.
Superficially the most obvious thing to do is totally make everything up, from top to bottom, but it would probably work just as well to use real live market data and overlay on top of it. Just as air traffic control simulation has very slowly moved over the decades from purely random to sorta realistic to actual airport data.
This could be quite a bit of fun.
Hm, I wonder if there is an interest to make this a more open event :)
Yes. I bet there is.
You're right in that the HF bit doesn't really fit but 'fast trading' doesn't have the same ring to it.
Edit - I love the project by the way. As others commented already, there's a really strong argument to be made in favor of open sourcing the infrastructure side of this. And that appears to be where most of your effort is focused (e.g. open infrastructure, private trading algorithms).
You're pretty right about good trading algorithms but that doesn't necessarily apply to good traders or good trading systems. A good system - hft or otherwise - needs to be flexible and evolve to stay in touch with an evolving ecology. And to do this, it needs to keep track of many different trading ideas and algorithms, each of which may have its day in the sun.
doing it in go and then using cgo where necessary will get you pretty close to C speed.
Conspicuously you don't address Haskell at all. Then again, who cares what might actually be best for OP? There's an advocacy bingo card in play!
Our platform is fast and could be used for some slower strategies. If we replaced pieces of it with C we could be competitive but probably never less then 10 microseconds wire to wire.
https://github.com/fmstephe/matching_engine
But if I can get a reasonable level of performance out of the standard Go queues then I will just use those in the meantime.
I _was_ using the ITCH format for messages and switched away from it. I may revisit that decision in the future.
I agree that it would be really interesting to look at the messaging protocols of real exchanges, and I have made a quick stab at it. But I am focused on getting something very simple working right now.
If you wanted to play around with ITCH files the executable generated in the itch directory will run any ITCH file through the matching engine sort of like a debugger, allowing you to step forwards and backwards in time and see how the internal state of the matching engine changes.
I work in a hedge fund in London at the moment and there's a massive shift away from R towards python in the algorithmic shops that I know about here. There's also the great work that quantopian are doing on their backtesting framework zipline [1]
[1] https://github.com/quantopian/zipline
The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.
But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that.
I've had good success so far with using R as a fast column database, solving big data hassles, but there may be better solutions in that space.
The way we do our monitoring/UI is with zeromq/protocol buffers as an external surface to the trading system (basically a PUB socket) and a very thin bridging and translation layer to WebSockets and JSON in python. That way you get to use d3 or anything else on the front end. We use backbone, knockback and d3 with coffeescript.
The main benefit we've found with python as the algo language is that it allows for stat programming with pandas, but also OO or functional programming for the algo logic. This smoothes the transition from research to production, just as you're describing with R -> haskell, but you can stay in one language.
I think one of the biggest potential wins with parallelization is if you can assume all positions are closed overnight, most often true for HFT. That way, you can simulate all the trading days in a test range in parallel. This is quite similar to the parallel processing we do to handle the large number of concurrent backtests running at quantopian. We did all of that with python, but I'd be fascinated to see it done with haskell.
How would you handle the position sizing part of the algo if you're testing all days in parallel? Wouldn't the trade size depend on the all of the previous day's PNL?
It is a Lisp, semantically, with a MATLAB-like syntax (including all the linear algebra sugar), and a LLVM back end.
The stated goal of the language is to give to scienctific programmers the convenience of high level languages for prototyping, yet the speed of low level ones. It has multiple dispatch, based on a powerful type system, homoiconicity and macros (optionally hygienic).
The package collection is growing steadily [1]. There's also a mailing list dedicated to doing statistics with Julia[2].
I looks like a good fit for your project.
[0] http://julialang.org
[1] http://docs.julialang.org/en/release-0.1/packages/packagelis...
[2] https://groups.google.com/forum/?fromgroups=#!forum/julia-st...
mmap is a package that let me create a fast db from scratch in a day. ggplot, lattice and all the charting routines are a joy to work with.
I'm interested in random forests today, and I bet there's a package that links to the standard libraries whatever they are. This is also probably true for python, but Julia you'd have to build the API.
(as in, I spent part of an afternoon end of last week skimming the entire code base).
This is troubling, because when you don't design your language to make it "manageable" to do static analysis / optimization on, it be becomes DIFFICULT to add that capability later. Julia is about as dynamic as JS or Scheme or Lua, plus it has all those runtime type tag cases in code... but whereas JS has relatively unbounded manpower (relatively!) to make a fast JIT, whereas lua is a small enough language that we have the amazing impressive LuaJIT, whereas Scheme is designed with very thoughtful semantics / specifications in mind, Julia lacks.
more succintly, Julia lacks a clear enough thoughtful choice in static/dynamic semantics for the pre LLVM side to have an easy optimization story given a small sized core team, such optimization engineering will either take a long time to develop, or will require some commercial entity to sink serious capital into writing a good JIT / optimizer. LLVM is a good backend, but it can't leverage the semantics of the input language, just the semantics of the LLVM bitcode you provide. You really really can't punt on optimization on the before LLVM side.
also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) )
Let me repeat: Julia doesn't have clear static semantics / phase distinction, and doesn't really seem to do anything beyond method specialization before passing code on to LLVM. This means it can't get compelling performance compared with any other LLVM backed language.
LLVM is amazing, and i use ghc -fllvm exclusively, and while theres some really really awesome optimizations that LLVM does for haskell code (for example), a HUGE amount of high level code performance is really depending on the language (any language!) doing optimizations before passing things onto the LLVM.
The fact is that the performance of Julia is already quite good. Julia is semantically much closer to C than Haskell is, and it needs fewer optimisations to get reasonable speed.
The necessity of static analysis for performance is a myth that's been debunked for a while now. From the language semantics, I think that the runtime model of Julia must be about as complex as Lua's, including the LuaJIT FFI (swap metatables with type-based multiple dispatch).
As you said, though, whereas LuaJIT has its own tailored bytecode, Julia compiles to LLVM IR, and potentially useful information is lost in the translation process. LLVM-Lua is reportedly 2-3x slower than LuaJIT. I suppose that type specialized traces and immutable types help somewhat.
An area that clearly needs improvement is vectorised code. Currently, an new vector/matrix is allocated for each intermediate step. Bad. This is being worked on.
> also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) )
You're conflating type system with static type checking. Julia is dynamically typed, but its programming model (the UI of sorts) is built around the type system and multiple dispatch, it is one of its main features of the language, actually.
The first point is that Julia already has excellent performance on a par with most compiled languages, including, e.g. Haskell, whether they are using LLVM or not. Straight-forward Julia code is typically within a factor of two of C. That's shown in the microbenchmarks on Julia's web site [http://julialang.org/], but, of course, that's not entirely convincing because, you know, they're microbenchmarks and we wrote them. However, similar performance is consistently found in real-world applications by other people. You don't have to take my word for it – here's what Tim Holy [http://holylab.wustl.edu] has to say: https://groups.google.com/d/msg/julia-users/eQTYBxTnVEs/LDAv.... Iain Dunning and Miles Lubin also found it to be well within a factor of 2 of highly optimized C++ code when implementing realistic linear programming codes in pure Julia: https://github.com/JuliaLang/julia-tutorial/blob/master/Nume.... The benchmarks appear on page 7 of their presentation.
This statement about Julia's high-level optimizations is entirely wrong:
> i've looked at how it compiles to llvm, and it basically punts any optimization to the llvm side, aside from the most basic tracing jit method monomorphization/specialization.
Julia does no tracing at all, so it's definitely not a tracing JIT. A relatively small (but growing) very crucial amount of high-level optimization is performed on the Julia AST before generating LLVM code. In particular a dynamic dataflow-based type inference pass is done on the Julia AST. Since Julia is homoiconic, this type inference pass can be implemented in Julia itself, which may be why you missed it: https://github.com/JuliaLang/julia/blob/master/base/inferenc.... Don't be fooled by the briefness of the code – Jeff's dynamic type inference algorithm is one of the most sophisticated to be found anywhere; see http://arxiv.org/pdf/1209.5145v1.pdf for a more extensive explanation. It's also highly effective: 60% of the time it determines the exact type of an expression and most of the expressions which cannot be concretely typed are not performance critical [see section 5.2 of the same paper]. You are correct that we leave machine code generation to LLVM – after all, that's what it's for – but without all that type information, there's no way we could coax LLVM into generating good machine code. Other very important optimization passes done on the Julia AST include aggressive method inlining and elimination of tuple allocation.
> more succintly, Julia lacks a clear enough thoughtful choice in static/dynamic semantics for the pre LLVM side to have an easy optimization story given a small sized core team, such optimization engineering will either take a long time to develop, or will require some commercial entity to sink serious capital into writing a good JIT / optimizer.
There is a very clear and thoughtful choice in static vs. dynamic semantics in Julia: all semantics are dynamic; there are no static semantics at all. If you think about your code executing fully dynamically, that is exactly how it will behave. Of course, to get good performance, the system figures out when your code is actually quite static, but you never have to think about the distinction. And again, Julia already has excellent performance, and we have accomplish...
I think we've had 1-2 interactions where we've not quite gotten along, but I might have just been misinterpreting (or mixing up julia devs)
It would help if this explanation was a bit more prominent.
For example, one of the biggest challenges in dynamic type inference is getting the types of fields of mutable heap-allocated objects. We avoid this debacle by letting you declare field types, which also has benefits for code clarity and specifying memory layout. Everybody wins. But once you are writing types, you need fairly flexible types, to avoid being stuck with only Int and Any (everybody's favorites). We then follow the implications of this as far as we can.
As for "all those runtime type tag cases", the key is that they form a lattice, which feeds nicely into dataflow analysis. The lattice-theoretic properties of a language's universe of objects typically do not get enough attention, especially in dynamic languages, where it is actually most needed. Typically the lattices are either trivial (for example, scheme's fixed set of types), or highly uncooperative. You want to hit a sweet spot where you can compute greatest lower bounds that are actually somewhat interesting.
Is Cython or PyPy useful, or do you use Python more for prototyping, and rewrite some portions in C?
[1] https://github.com/numba/numba [2] https://code.google.com/p/numexpr/
WRT to the problem of deploying prototyping/research code, there are the following unsolved problems that I'm aware of:
* Going from matrix operations over the whole timeseries (taking care to avoid problems with your algo looking ahead) for speed in a research setting to deploying to an environment that streams updates to the timeseries one at a time. I think that this is an area that haskell has the potential to excel at, given it's strong guarantees on structure.
* Concurrency. The options in python all suck to some degree - especially if you have to interact with C libraries or extensions. I don't think it will ever make sense to build your real time market date in python. Again here haskell has an advantage.
However the python ecosystem seems to be almost perfect for researchers:
* Excellent and flexible data slurping/munging/transforming.
* numpy, scipy, pandas, theano, scikits... 'nuff said
* ipython
* Cross platform
HTH
exactly my thought. Algo guys get stuck in matrix land because that's where their tools take them. Whereas this came out in R last week: http://cran.r-project.org/web/packages/stream/index.html
python is a better toolset than R maybe, but the R problem domain seems broader in the last few months anyway.
My interest is in monitoring thousands of algorithms in real time directly within the messaging environment, and before the data hits a database. That type of concurrency is where haskell can muscle up and do the job.
Looks like a match made in heaven :-)
[1] http://blaze.pydata.org/
If you're interested in advice on how to bridge the two worlds let me know, there's a lot of of upcoming technology ( LLVM, Blaze, pipes, zeromq, cloud-haskell ) that could be very useful.
2. I agree with you comment that it is easier to think about concurrency in Haskell than in something like C++, however you can't really compete with C/C++ in Haskell. Not even with cgo (Go packages that call C code), not with OCaml or any other higher level beasts that promise the speed. Fortran would be the only one faster for the "algo" part of your initiative. But again, if this is just an exercise, Haskell and others (I prefer Clojure for example :) will do just fine.
3. Would make sense to split the "platform" in two (very different) parts: "Quantitative Analysis" (a collection of tools and rules) and "Technical Glue to Read and Stream". Each can/should be divided further of course, but the two above are essential yet very different for a true "HTF Platform".
I'm preparing some speed tests between C++ and haskell on an identical block of processing so stay tuned! You might be surprised - haskell is way ahead of clojure on compiler smarts.
The split you suggest is exactly what I think is wrong with the way things get done right now. I'd like to integrate the quant inside the read and stream - now that's potentially a large speed up that might compensate a tight budget.
I see the financial world as a necessary evil, and I think some of its parts have benefits for the rest of the world. HTF is not one of those parts; I see it as a pure burden.
You could have made an open source Patent Trolling system and I would feel the same.
http://www.nyxdata.com/capacity
BTW, many trading firms do use haskell in Trading, but thats all privately held.
To everyone saying that the cost in these systems is the cost of co-locating servers or fpga cards, you're wrong. You can get hosting deals/leases on those kinds of things for the same cost as high-end web hosting. The cost of running these systems are 2 fold: 1 paying the employees (because your competitors can pay them a lot) and 2 having deep enough pockets to survive the bad days/weeks/months. These are the same costs that have always existed in the trading space and have nothing to do with electronic trading. In fact, electronic trading has lowered the information asymmetry and made it easier for new participants to be involved in the markets.
This reminds me of the LMAX financial trading platform where they started with a concurrent model but ended up using a single-thread "...that will process 6 million orders per second..."
http://martinfowler.com/articles/lmax.html
If what you are interested in is complex decision making then it may make sense to use a different sort of messaging technology than LMAX, but you won't be getting into anything remotely "low latency". Nothing wrong with that, just needs to be a known expectation.
Again there is nothing wrong with that. It's just better to state it up front as an expectation and/or a goal than to assume you are going to be able to pivot easily once you have an architecture in place.
I'm probably thinking of algos more generally and generically. For example, whatever the class of trades, I think that what you are doing is making a stochastic forecast of future price movements. So maybe there's a good separation possible between your algorithms that create a distribution and algos that determine how to trade once you have that distribution. But saying this probably reveals a bias I have to think statistically. And I totally agree the majority of the system build thats important is the boring stuff. An algorithm should be able to tell if the market feed has died or the market shuts down unexpectedly, right? And know what to do next. And good to hear that someone thinks the costs are much less than popular opinion suggests.
Can anyone recommend any resources (books, tutorials, etc..)?
http://www.amazon.com/Dark-Pools-High-Speed-I-Financial/dp/0...
http://home.rbcarleton.com/rbc/software/smalltalk/VSA/
Look at the RBC-VSA-Portfolio category. That's where all the stock/option code is.