Happy to answer any questions about this. It's been a side project that turned into a full blown obsession. There is nothing too secret about the system since it's more about having a solid platform that you can plug your strategies into. I'd probably even open source it but I'd have to clean up all my hacks :)
I've been really wanting to use Go, but as you say, much of the community is Python due to the data analysis strengths. To the detriment of the other things Python does do poorly.
Can you give some thoughts with your experimentation on the following from a Go perspective.
1. Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc.
- certain forks tailored to real time rather than historical (eg: no re-compute on ticks)
2. Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB
3. If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)
I had not looked at Mojo... thanks for the pointer.
In the past, I've had issues with library compatibility with compiled derivative languages of interpreted ones (eg: Crystal of Ruby, etc, etc).
Know if Mojo directly uses existing Python ecosystem?
I'm excited for Mojo, but it'll be years before it's ready for prime time as a general purpose language. It's not even available to download yet AFAIK...
> Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks)
I've been basically, just manually coding the algorithms from python into Go. ChatGPT is amazing at this. I really only just about 4 so it was a one time thing.
> Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB
Yeah, I ran into issues and then was like what would be the fastest, then just went in-memory. I download all raw trades/quotes each night and store then into gob+lz4 compressed files. Then for backtesting and stuff I can load these in and build the aggrogate bars on the fly.
> If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)
Yeah, I have a historical lookup table that I build nightly too. This gives me a reference point when I'm doing % change calculations and stuff. I should probably have mentioned that.
I'd be careful using ChatGPT for this. I tried the same recently to bootstrap spot yields from par, which is a fairly common code problem with hundreds of examples online. ChatGPT couldn't do it. It produced code that looked right, but would fail my tests abysmally. I ended up writing it by hand. I hope you are validating their code with known data.
Yeah, I'm heavily testing everything it writes. So, I'm very sure it's correct or get it close enough and then code it. I've seen ChatGPT add imports, functions, etc that don't even exist in Go. What I've found so useful though, is not even coding, but asking it how to solve problems, and then having it code things up. With Google, you need to pretty much already know how to solve the problem then go looking for answers. With ChatGPT you can ask it how it would solve this problem. For algorithms and stuff it have been unreal. Thanks for the warning though. You're totally right.
Awesome write up. I have a similar project in Go myself, although I just use minute bar data instead of realtime ticks.
Can you share your approach for plugging in various strategies? I quickly learned that having a pluggable strategy system is tricky as it could span across multiple layers of the system.
Also, with backtesting, are you storing and replaying all the quote/tick data? or just using the historical aggregates?
I wish this was better, but honestly, I'm just hard coding them right into the BUY loop. So, I need to restart the app anytime I want to change something. That's why I build this logic to dump and reload the state into a gob file (go memory dump essentially). Ideally, you'd have some type of format to write out your algorithm, and then have a way of hot loading it or something. But, I don't change these enough to really do that yet.
For back testing, I download all raw trades and quotes, and put them into 1 file sorted by time (1 file per day). Then, I compress them using lz4. This allows me to sort of replay the entire market and build up all my intraday backtesting from the source. This took me a long time to figure out and build but has been so worth it. So, I have an off-line script that basically, loops through these files, and replays the market, and makes simulated trades, and then spits out what would have happened. There is a GUI for that too so you can go in an explore the trades and see what triggered the buy and sell. I have seen nothing that goes backtesting for intraday like this.
This is super inefficient but I'm just building the aggregates on the fly. I could probably cache them somewhere but it takes maybe 4-5 minutes to replay a days worth of trades/quotes and build all this so I haven't bothered yet.
Count me in the group of solo algo trading developers (Go and Python and Tradingview Pinescript and other sruff). Planning to send you an email but maybe we can form a group/discord or something online.
"Tech Trader is a fully autonomous trading system live with no human intervention or updates, now for over 10 years. It is unique from conventional algorithmic systems, not only because it actually is fully automated, but because it takes a "human" approach to markets. It is not quant. It is not stat-arb. It is not high frequency. It is a program that looks at stocks the same way a person does but with the cold discipline and infinite attention span of a machine. It is analogous to having a thousand independent traders each focusing on a single stock, as opposed to a single quant manager trying to make sense of a thousand datapoints. A person doesn't think through stats, correlations, or complex math models when trading, and neither does Tech Trader. Tech Trader leverages technology to do what human traders do at scale rather than approach markets from the point of view of an academic, mathematician, or scientist.
Since its launch in Dec 2012, Tech Trader has been trading live capital completely on its own, fully automated in the truest sense with no human input, no tweaking, no updates. It is, for all intents and purposes, an autonomous hedge fund, one of the first to truly trade unsupervised for years on end. Whereas many "automated" or "AI" funds may have a hundred scientists providing the actual intelligence behind the curtain, the creator of Tech Trader consists of just one person - a self-taught individual going by the gaming moniker pftq, who created the system at age 21 and has long moved on to other interests. "
No way. Not even close. That would be the dream though. Taxes are insane in Canada. Like 50% insane. Housing and kids just eat money. Maybe if I was renting and didn't have a family but that's not happening.
You connect to IB's TWS API... If you execute your trade there only, wouldn't it be an option to fetch the price in real-time from the IB API and not having to use polygon.io at all?
I have no idea: does IB send the price feed in real time? (they certainly send the data in real-time to TWS as it constantly updates right? But is the order book available through their API?)
Basically and even though I know this was published on polygon.io's blog, would that work by only using IB / TWS's API?
Yeah, I initially tried that actually. They have very low resolution data and do not provide raw trades/quotes (something like a message every 250ms), you cannot watch the entire market so you're stuck with like 100 tickers [1] vs 5500 tickers, and fetching historical data is very cumbersome. You can basically pull down pre-aggregated candlestick data and I wanted to base things off the raw trades/quotes. I actually probably spend a month or two trying to make this work. So, that's why I'm using polygon.io in that you can stream trades/quotes for the entire stock market in real-time, without any caps, which is amazing!
I've been going through this journey myself. I started learning on Tradingview, then bought Build Alpha to discover how to test strategies. I chose Portfolio 123 for my automated factor trading but had been working towards creating a program/basket trading system like yours that can act on intraday data.
I moved to long-term investment until I could build a simulator capable of verifying the correctness of my investment strategies using fuzzy testing ideas stolen from TiggerBettle.
I have almost two years of polygon quotes and trades for the whole market captured with a monotonic timestamp to be able to replay the data –and test the handling of polygon socket glitches.
I'm focusing initially on capturing the data in a way that allows fast replay and aggregation, similar to what Kafka can do with topics but in-process using zig and custom memory-mapped data structures. My idea is to be able to generate signals like VIX (once I add options data), ETFs, and indexes and hopefully be faster at doing so than others :), please HN folks, call me out here if I'm being too naive.
This has been a three-year learning process for me. I have been a retail investor for +10 years, but over the last three years, I've gone deep into learning algo-trading, drank del Prado Kool-aid, and read numerous trading and investment books.
I'm now focusing on my technical chops to build the engine to build order books for individual stocks, baskets, and indexes with realistic market prices. I aim to develop a system that can get as close to the market price in the next dollar bar as possible.
This has been a very lonely journey, and after reading the responses to this post, I'd love to connect with others on a similar path. Sending you an email!
I've used ATR bots for years, and would love to hear your thoughts on how what you're building delineates itself as a distinct product strategy beyond just the programming language GO.
I've used wonderbit, zigz, and some of the others.
I've been algo trading as a side project for a few years now, but all through TD Ameritrade. I get all my quotes and whatnot with them, and it's free with my accounts. I also get other data elsewhere, which is free as well.
I was curious in getting a websocket setup with Polygon after seeing your post, but I noticed it's $29 for first package with the websocket feature. Then I noticed that the package with real-time data is the Advanced package at $200 a month. I'll admit, real-time with unlimited API calls sounds pretty sweet, as my strats rely on to the second data. However, I don't know if I can justify $200 as my algos run no more than 78% win rates with a normal bankroll funding it all. I'm looking to save wherever I can while I build these things and get a passive income stream rolling in.
I would love to know which package you were using, as I didn't see it in the time I quickly read your post? Also, any pro/cons to that specific package? Any and all other details are welcome, and if you would rather respond to my email, it's in my profile. Thanks!
Yeah, personally, it's been a game changer just being able to load so much more into your field of view. Having two side-by-side makes it really easy to look at other parts of your code and make changes. I'm loving it! I'll give the H method a try for a few days and see how it goes.
Eh, honestly only thing horizontal is good for is media, any documents/code is better in narrower vertical columns.
Honestly my perfect screen for work would probably had something like 1:1 ratio, so I can have 2 nice columns of code that are tall enough that can be split horizontally if needed while still being useful.
Fascinating application of the language and a terrific write-up. I would presume a GC language would normally be a disqualifying factor in real-time trading, but I think I'm coughing up some premature optimization, especially with what looks like a pretty beefy rig. Congratulations though, this is spectacular.
Yeah, I never really looked a GC. Most of my trades take at least 500ms+ round trip. It's more in the latency to the broker and getting confirmations that I've been trying to tighten up. At least that's what I'm seeing right now.
I guess my presumption was that algorithmic trading was a very tight feedback loop, with as many controlled variables (i.e., GC) as possible, so I think it just subverted some of my misplaced expectations.
What's cool about this is that you can look at the IB TWS client and see things happening in real-time. So, it acts as sort of a sanity check. I know they have that gateway too but personally I like to look at the client all the time too. My workflow is to run the bot and the TWS client side-by-side and watch it make trades, see how things are moving, etc.
Excellent write-up! Is there somewhere where I can read up on all the trading jargon?
Also I wonder how can there be changes in the price of a stock after market if the exchange has closed? Isn’t the whole point that trades need to happen for stocks to get a certain value?
There's after market trading on venues. But because most of the liquidity has "gone home" you can't transact as much at those prices and those prices are based on relatively few/small transactions.
Yeah, this is where a system like this shines. In that you can build a tool, like top, but for the stock market. You have all the real-time data. So, you can look at where the party is happening at. Just by looking at the trade flows. This is where building your own system absolutely kills anything else.
https://www.investopedia.com/ is probably the best resource and chatgpt. I honestly, just google stock related terms and all roads lead to investopedia.
This is really interesting. Have your strategies out performed buy and holding index funds, or are you mainly just doing this with a small amount of capital to learn how the markets work? I have always wanted to try algorithmic trading to learn about it, but I have always read it is a fools errand to think you will beat just buy and hold.
1. I think buying and holding is more probable to have a higher return
2. Sometimes you get a cool api and think wow this would be fun, and next thing you know you've lost thousands on boneheaded trades.
I did something similar during the pandemic with Rust with the Polygon API (and instead of interactive brokers, I used tradier). Eventually I learned I actually had more fun building the thing than actually trying to beat the market.
I have won and lots thousands for sure. Haha. When stocks were on a rip the bot was making lots of money just because everything was going way up. Then, in 2022 when everything went way down, like tons of tech stocks, my bot sucked. So, I really need to add shorting or something. I'm still exploring things on the strategy side.
Yeah, right now I'm only set up to buy stocks. I haven't tried to short anything yet. I want to get to this eventually since it would be nice to make money when the market goes down too.
I'm sure you know this, but for others reading this who are novices at finance / trading, like I am - the gotcha here is that while the strategy may be symmetric, the risk is not - when buying a stock, there is a floor to how much money you can lose (the price you paid for the stock), while with short-selling, there is no such floor, since the price can rise to any amount and increase your losses to infinite. I believe that traders will use hedges to account for this, however, these hedges will eat into your profits if they are not exercised (but may save your bacon if they are!).
Yeah, I haven't explored this yet. The unlimited risk thing bothers me thought. Which probably sounds funny since this whole things is risky as hell. But, I was more looking into options or something to hedge but honestly everything takes so much time to learn about, test, and then do. So, I just wanted to focus on this and then expand.
Yes, 100%. Everything looks amazing while the markets are going up. Sometimes, I've just shut the entire thing down when there is Fed news or the markets are taking a dump. That's a legit strategy too. Only run it while the markets are going up.
This is going to sound crazy given all the scams out there. But I was interested in testing the idea of small compounding returns. Like, could you get a daily 0.5% compounding return. Sure, you could go all in on TSLA for example and get a 1% daily return. But, could you do that with automation, using lots of small bets, across the entire market. You can, but there is a scale issue here. In that you need to make exponentially large bets as your bank roll increases. So, it's capped. Well, that's what I've seen. So, yes you can beat it but only with smaller amounts of money.
That makes sense to me and lines up with what buffet says about there are alot of people on wall street who can average 50% returns with 100k but once it gets into the millions it is much harder to find alpha.
32. It’s easy to make 50% on a million, but much
more difficult on larger amounts
WARREN BUFFETT: Station 9. We’re just about — yeah, we’ve got time
for a couple more.
AUDIENCE MEMBER: My name is John Dorso (phonetic), and I’m from New York. Mr. Buffett, you’ve said that you could return 50 percent per annum if you were managing a one-million-dollar portfolio. What type of strategy would you use? Would you invest in cigar butts, i.e., average businesses at very cheap prices? Or would it be some type of arbitrage strategy? Thank you.
WARREN BUFFETT: It might well be the arbitrage strategy, but in a very different, perhaps, way than customary arbitrages, a lot of it. One way or another, I can assure you, if Charlie was working with a million, or I was working with a million, we would find a way to make that with essentially no risk, not using a lot of leverage or anything of the sort. But you change the one million to a hundred million and that 50 goes down like a rock. There are little fringe inefficiencies that people don’t spot and you do get opportunities occasionally to do, but they don’t really have any applicability to Berkshire. Charlie?
CHARLIE MUNGER: Well, I agree totally. It’s just you used to say that large amounts of money, they develop their own anchors. It gets harder and harder. I’ve just seen genius after genius with a great record and pretty soon they’ve got 30 billion and two floors of young men and away goes the good record. That’s just the way it works. It’s hard as the money goes up.
WARREN BUFFETT: When Charlie was a lawyer, initially, I mean, you were developing a couple of real estate projects. I mean, if you really want to make a million dollars — or 50 percent on a million — and you’re willing to work at it — that’s doable. But it just has no applicability to managing huge sums. Wish it did, but it doesn’t.
CHARLIE MUNGER: Yeah. Lee Louley (phonetic), using nothing but the float on his student loans, had a million dollars, practically, shortly after he graduated as a total scholarship student. He found just a few things to do and did them.
Yes, 100%. Like I said in the article there. I once lost like 40% in a few minutes in the pre-market because my app didn't have the correct sell logic. Honestly, I don't even know if I had the correct logic that would have even saved me, because the market was falling so quickly I probably could not have even for the fills I wanted. So, I've moved to breaking things into tons of small bets and really trying to manage risk.
Yeah, you could look at it like that too. But you need to have a platform before you can even do something like this. So, that's why I framed it like that.
Yeah, I'm not saying this is low risk. It's more about trying to build a system to be in the right place at the right time. It is massively risky. I was sort of hesitant to even write that comment because you constantly see all these youtube day traders selling courses on making 1% daily. If you try that you'll lose money extremely quickly.
This comment made me think of the tax advantage of buy-and-hold versus lots of small bets. Buffett also talks about this in his Berkshire shareholder letters.[0]
>I went from not knowing how to solve a problem, blindly googling around and reading books, to just telling ChatGPT the problem, and then asking how it would solve it, then asking it to code the solution. This is absolutely insane and has easily 3x my productivity.
Very interesting. We can facetiously say that ChatGPT is using you as a medium between setting up algorithmic trading in Go!
Regarding open sourcing: don't be afraid to show your flaws; it doesn't have to be lonely if it is open source.
Obviously whatever trading bot you're running separate from the actual trading engine itself is somewhat proprietary, but it would be great for the community to get more of this type of software in the hands of other hackers.
Quantopian / Robinhood tried and failed, and the numerous clones since then have been somewhat sub par.
You might find https://www.quantconnect.com/ interesting! They offer a platform for quickly developing and backtesting trading strategies. They have a good community and overall do a great job. Give it a shot!
I worked in HFT for a while. I find the whole space fascinating. I'm glad OP found similar thrills.
"This aspect, the platform itself, seems to be often overlooked in most discussions. Many conversations revolve around strategies (mean reversion, trend following, linear regression, etc.), and backtesting, without fully addressing the practical mechanics or logistics of strategy implementation, particularly in the context of live, intraday trading."
I'm glad you had fun, OP, but also I think I can shed some light on why most people discuss strategy.
Trading is a perfect storm of ridiculously high tech, ridiculously complicated, ridiculously regulated (Not over-regulated, mind you, this isn't a value judgement. But the amount of regulation is extremely high.), and ridiculously competitive.
But that said, it's the last bit that drives it all. Since it's so competitive, even though building an order entry system, and a risk system, and a position-tracking system, etc is a huge accomplishment (again, congrats OP!), it's table-stakes to even dip your toes in the pool here. Trading shops can attract top talent and robust, bespoke trading systems are basically cost of entry.
So people talk about strategy because everyone already has the table-stakes stuff and are now trying to make money with it.
It doesn't help, too, that lots of market participants aren't even playing the same game. In HFT, we operated on trades with alphas that lasted a few seconds, where races to entry/exit were battled in shaving nanoseconds off FPGAs being able to shoot out orders and microseconds off wireless networks flying market data around new jersey. Meanwhile, banks are more concerned with elections and geopolitics than they are about the weather in Carteret. (Rain = no microwave network for the day). And then there's a million strategies in the middle with alphas that last from hours to weeks.
So it makes it really hard to even speak the same language to each other when talking in common forums.
YES, 100%! I honestly, cannot agree more. I've read pretty much every book that I could get my hands on and no one talks about the platform. This makes so much sense. Thank you for your perspective. It seems like so much of this stuff is silod off into each company and no one is talking about it.
HFTs are definitely playing a completely different game. I was reading about the exchange architectures and how things are actually wired. I'm getting my data from SIPs while HTFs are directly connected to the exchanges [1]. I'm transacting in seconds and they, like you said, are transacting in microseconds, so there is no comparison. Which, in a way is actually nice in that I'm not really competing with them. Or, maybe I am but I can still make some money. haha.
Not only are the platforms table stakes they are the more straightforward part to build. Even at the bleeding edge of latency you can usually work your way to the limits of your platform budget without having to find anything novel.
The strategies though are where the discovery is. There are a few strategies that are well known and still profitable but those are largely consolidated to the biggest firms. For everything else it’s a discovery process. And done strategies are only profitable for very short regimes.
I miss it sometimes too, but so much has been consolidated it’s largely a big firm world now.
More straightforward, heh, sure. But still damnably complicated. Which just goes to show how much money and how much engineering talent is invested in this world that these things are so taken for granted.
Essentially yes. When I worked in algo trading, it never bothered me that we were extracting profits from the markets, nor that we served little social good. It felt like a step up from where I’d been before (being told that we were making the world a better place, when every engineer knew otherwise.) At least we weren’t making things obviously worse.
What did bother me, and was acknowledged by my coworkers, was how much top talent was being pulled away from productive tasks to essentially wank around. Not just technologists either, but all sorts of mathematicians and scientists were drawn to the flame.
We as a society have managed to allocate so many of the “best and brightest” to either fintech wankery or placing ads in front of eyeballs. It’s enough to make one want to give up on capitalism, except that everything else appears to be worse.
Seems like every person I've ever met in M&A has ended up hating life because of their career choice. They seem to burn out in 5 years or so after salting away a million bucks or 3. Was that your feeling doing the HFT thing?
The jobs in algo trading are very interesting for technically - mathéamtically inclined people. It’s really one of those fields where you have a direct impact on the results of your - measurable in additional dollars made.
There is 0 social impact. That’s the downside of course - but hey, how many jobs out there are really having any kind of positive social impact ? Not 0, but close to it.
That impact is not made by HFT - finding the right risk premia for different investments is very valuable but that is a signal measured over days/weeks/months because actual capital investment decisions take that long.
Intraday financial games are zero-sum. What HFTs gain, they leech away from mutual funds and pension funds and retail investors and market makers who operate over a longer horizon.
I work on very interesting technical problems with very smart colleagues for excellent pay. All my career progression is in compensation, so I can remain an IC forever and no one thinks that's a negative. I'm subject to no politics whatsoever, and there's very little politics in the company as a whole. The work I do every day has a direct material impact on the company and I'm rewarded proportionally to my impact. My WLB is so-so, but it's better than it was in grad school so I'll take it.
Regarding social impact, the world does have some demand for liquidity and price discovery. Providing those services is both essential and extremely difficult. It's definitely not the most social good I could be doing with my talents, but I think it's weakly positive.
What would rising levels of being lifted out of poverty suggest? The equality gap doesn’t count for much if you starve or freeze to death, and poverty is declining world wide.
The worst way to measure it is inequality. How the top person is doing relative to the bottom person isn't important. How the bottom person is doing now compared to the bottom person 100 years ago is.
sorry, i don't believe inequality is rising, at least in ways that matter.
in the last 50 years, a substantial number of people were pulled out of poverty.
Obviously this would mean that countries that previously relied on subsidised labour from those third world countries would have to start paying up. I'm not suggesting that it is a zero sum game however.
Ads are tremendously useful, just ask any business owner.
The problem with ads is not that they're useless, it's that it is an industry prone to scams and grift. (Because doing advertising right requires all sorts of actual science, and ain't nobody got time for that when there's money to be made.)
>We as a society have managed to allocate so many of the “best and brightest” to either fintech wankery or placing ads in front of eyeballs
It's nothing to do with "we as a society". I'm a quant trader and know many others, and the vast majority are in the industry because we care about making money not some leftist save the world crap. Even if socialists managed to completely destroy the financial market, we'd just find another way to make money without trying to save the world (e.g. like the mostly corrupt officials in Russia and China when they were communist). "Society" can't change human nature; even mass brainwashing on the scale attempted by Maoist China failed.
I.e. the best and brightest will always work where they want to work, not where leftists like you want to "allocate" them.
I think you just illustrated the problem. You have complete awareness of what you do, and you somehow manage to see it as a battle against "leftists" rather than selfish greed. I'm cool with that, but it precludes you from having valid opinions on anything related to improving the world for the majority of people. Please accept this and refrain from any social commentary. Your comment above demonstrates that you don't actually operate in society.
> Please accept this and refrain from any social commentary. Your comment above demonstrates that you don't actually operate in society.
Cut the fascist bullshit. Everyone can comment about how society should be run, even people that are very successful financially.
If you try to exclude greed from the design of your societal system, it will immediately fail. In large numbers, economics shows us that altruistic people wash out of the model and everyone operates in their own self-interest (greed).
Self interests have externalities. The pleasure you gain from improving someone's life is in no way comparable to the pleasure you gain from making other people suffer. It's a fallacy to say "everyone operates in their own self-interest". It's usually parroted by anti-social people in order to justify anti-social behaviour.
> Cut the fascist bullshit. Everyone can comment about how society should be run, even people that are very successful financially.
It's not financial success that precludes someone from commenting on how society is run, it's blatant, deliberate anti-social actions. This is the foundation of a social contract. It's the same reason you'll get locked up if you go around punching people in the face.
Also, I don't think you know what "fascist" means.
You’re suggesting that people can’t participate in societal planning because they disagree with your causes. You just dress it up in overly dramatic phrases like “anti-social” and “selfish greed” to make inhumanely oppressing them palatable. This was bog standard behavior of fascism in Italy.
It doesn’t seem like fascism to you because you’ve done some mental gymnastics to pretend you’re not just crushing intellectual opposition, but that’s all it is in the end. Op is not punching anyone in the face. He/she is just blatantly unsupportive of socialism.
Focusing solely on getting rich to the detriment of other people is the antithesis of "societal planning". I'm not sure why you're defending this behavior. Do you really believe that people who strive for profit over the well-being of other people are in any way being "social"? Do you see them as the necessary negative force for positive action to be possible? Do you really believe that humans require adversity to succeed, and that those who choose to be adverse agents are engaging in some self-sacrificial martyrdom so society can benefit from their negative influence?
Please, I want to know; what benefits do the actions of the op have for "societal planning"? The person he was replying to was at least honest in admitting that this behavior "served little social good".
Once again, I think you'd do well to actually read what fascism actually stood for. The propaganda of fascism had a lot more to say about nationalism, "shared history" and the good of corporations than it did to say about things being "anti-social" or even mentioning "selfish greed".
I’m a lot less leftist than you apparently assume. I’m a landlord, among other things. My problem with our system is that the things which “pay” are not things which have much benefit to society. I’ve got zero fault with you for choosing to make money. Didn’t I say I did the same thing and loved it? I just question the system that drives the behavior.
Largely, though I receive 1 or 2 job specs every week for start ups with the keywords 'hft' and 'low latency'. Admittedly there's going to be duplication there if you read them closely.
I think it's a bit of a myth that (ignoring FPGAs) that writing a low-latency software trading system is a time/cost expensive process. Anecdata = I worked at two firms where we did a rewrite from scratch with teams of 5-6 people and traded in the market within 3 months. I'd argue a senior dev that's been around the block a few times could achieve similar when you remove corporate politics, and bikeshedding over design.
The big firm part is paying for multiple quants at $200k++ to come up with strategies and historic market data access for trading models. Small firms are getting backing as long as the co-founders are 'ex-CxO from MegaCorp'.
> I think it's a bit of a myth that (ignoring FPGAs) that writing a low-latency software trading system is a time/cost expensive process.
This depends a lot on the complexity of the trading system and the trading venue specifics. A system to trade single stocks or futures can be built, certified and running in 3 months. A system for options market making will take a lot longer.
Yes, stocks and futures was exactly what I was talking about. Though we also hit the market with options models within another couple of months.
The big costs for small firms are historic data (if you don't have any), colo, distance to exchange, and number of connections.
From the number of job specs I see, it feels like the HFT/low latency market place is healthy enough that there are always new firms appearing. It's competitive, so it's hardly surprising that if someone has new ideas they'll find a backer.
It's quite a statement. You're almost saying capitalism and efficient markets are pointless. Maybe they are, but I think it's nothing like crypto.
In the old days before HFT, you weren't sure you'd get the best price. You'd have to rely on a broker to make sure that happens, but as a retail trader you generally got a worse price/out of date price.
Nowadays with HFT you can get pretty much the best price anywhere. Those <1ms HFT traders make that happen. It's the efficient market hypothesis in effect, made possible by HFT.
Now is it pointless to shave off even more ns in the all out war to grab a piece of the order flow? For retail and institutional investors, at a certain point, yes it's completely pointless.
But it's also just pure capitalism at work. HFT firms compete against each other, and the competition is about speed to provide the best price and volume. If you try to regulate with something like enforced delays, then what do you compete on instead?
> You're almost saying capitalism and efficient markets are pointless.
Capitalism necessitates efficient markets (and efficient markets necessitate HFTs) so any criticism of HFTs is a direct criticism of capitalism as well. I mean this isn't really a problem that's specific to HFTs -- there are just a lot of jobs that we can perceive as providing no value or even negative value (jobs that are possible specifically due to capitalism e.g. payday loans, 2008 style trading, certain scams or predatory practices etc.)
I think we can all agree that this is a flaw, and we're not criticizing capitalism to replace it with something else, but rather just recognizing this as a problem. At the end of the day, a useless job is a useless job even if it exists solely because of capitalism.
>You're almost saying capitalism and efficient markets are pointless. Maybe they are, but I think it's nothing like crypto.
People are saying this because, HFT sounds similar to 'crypto mining'.
That's people with best infrastructure, the 'big-guys' -- win.
While leaving out the retail investors as broiler chicken, pumped with 'drugs' (by influencers) to spend more on imaginary assets, so that they can be used for 'food' by these 'big-guys'.
There are different influencers for retail investors vs crypto.
In retail investing there are promises of 'retirement paradise', actual tax deductions, the Jim Cramer-like people (at least what I heard in US)
For crypto investing the influencer are different, the geography is wider.
A promise to participate in markets if you do not live the country that has adopted US/UK-based financial services.
- - -
By the way, I think the markets will still have liquidity if there is a rule to wait, for say, 30 min before a stock that was just recently bought -- can be sold (unless by a clear fat finger mistake)
This rule will cause the HFTs to stop existing in the current form.
So are retail investors truly no different from the people gambling their money on the lottery at the 7/11 every night?
Do any regular pedestrian middle class retail investors EVER actually earn enough to retire on? Or do anything meaningful with? If so.. what is THEIR secret sauce, since it's not HFT...
a)Members of political elite that get insider trading stock tips.
(illegal of course). The number of folks in usa congress and senate that become
'very lucky' investors after they join the rank, has to be amazing
b)Lucky
c) everybody else -- that looses.
Overtime, I would say last 30 years, the amount of 'influencing' retail investors to trap them into unreasonable actions had gradually increased.
So the percentage of folks going becoming the victims of the charade, will become higher.
Certainly if you concentrate on ( b ) you create the plausible deniability defense for the manipulators
You are profiting off workers as a middle man in the economy by doing HFT, and trying the justify it by some vague concept of the "correct price". You are producing nothing of value, merely taking away value before someone else notices it is there.
HFT provides a great deal of liquidity and efficient pricing in markets that are adapted to it. It provides a real service, allowing people to transact without using a large bank or broker efficiently on an open market. The fact you can click buy and it buys on almost any stock is likely due to a HFT on the other side. That may not mean much to you directly but it does provide a lot of utility in markets. The biggest gripe people bring about HFT is it during high volatility HFTs usually pull out of the market at a time when people really value the increased liquidity. I think some of the more advanced HFT firms though have moved into longer time range trades which helps provide more liquidity in those markets.
Depends on their strategy but a lot of strategies depend on some sort of price discovery which requires having a probable estimate of what the current price should be and some form of market making around the level. In a very dislocated market the price is unknowable and it becomes gambling, and generally market making strategies are explicitly not about gambling but about facilitating trades around the “true” price.
Nonsense - what could be more beneficial to society than providing imaginary "liquidity" by interposing yourself between legitimate buyers and sellers?
It's a bit like stock brokers - and why wouldn't we want stock brokers to operate at drastically faster-than-human timescales, because we all know the value of a company changes every nanosecond! And "flash crashes" create opportunities for investors to make huge amounts of money!
And just as crypto has poured money into GPU companies (providing opportunities for enterprising secondary market resellers of same) HFT has poured money into networking companies.
Short of creating the great firewall or helping governments slurp up all the traffic on the internet, what could be a more beneficial application of network technology?
HFT (like finance in general) has also gobbled up lots of tech grads, making it easier to find jobs at tech companies.
> Nonsense - what could be more beneficial to society than providing imaginary "liquidity" by interposing yourself between legitimate buyers and sellers?
Now when I say the same thing about index funds people get all huffy
It's similar, though in theory you are paying the index fund to act as your agent, and the fee structure is observable.
Regarding liquidity, I imagine it's possible that the index fund might allow you to withdraw money faster than you could sell on your own, but realistically I kind of doubt it.
The classic theory of longer term investing is that you are providing capital to businesses and rewarded for that. That is a little bit different from being a middle man. My personal opinion though is that lending money/investing capital and capitalism in general is power. Money quantifies the potential of making other humans do things for you, it allows you to excercise power over fellow humans. Inheritance means you can inherit that power from your parents. It's not all that different from feudalism.
It may seem so at the first glance, but such trading if carried out by many independent providers actually increases the liquidity if the market for everyone else.
Likewise with crypto. Personally I don't think that if you're calling an API over an Internet it matters if your trading bot is written in go or python (mine was in python). Use the language you're most comfortable in. The network and trade submission/execution at your broker will be 10x slower than your bot anyway. Unless the size of your operation approaches the size where you can get direct market access which seems to be reserved for big forms only.
"increases the liquidity if the market for everyone else"
That's just it. The whole premise is pretty absurd. The market, the actors, everything. It's so far removed from literally anything remotely human. It's the financial equivalent of an infinite sea of AI bots producing CVs and research papers which are only being evaluated and read by other bots.
If you step away from it all for a second, what the hell is the endgame of this whole hustle.
I mean, efficient allocation of capital is the overarching goal. HFT is either a facilitator of that or a perverse byproduct, depending on who you ask.
At any point in time yes, this is an accurate description. When you integrate over time it’s not. At the end of the line it’s people betting with advanced tools for execution of their bets. Behind the mathematical beauty of risk free and arbitrage, there is a reality of someone, somewhere making a bet at end of a line of contracts. And there is also someone making the opposite bet. Markets are not disseminating information but beliefs as a proxy for information, and as such they are much more human than you would think. The end game is a network of beliefs, livelihoods, politics and petty or grand aspirations, the same mess as in any organisation. The only reason you question the futility of this one specifically is that it’s a clean room abstraction of the futility embedded everywhere. I would call it futility as a service.
Your analogy isn't applicable here. What the OP was trying to get at is that even an individual who doesn't know anything about markets, HFT, liquidity, etc can still benefit from high liquidity from HFT (since it allows for transactions to occur sooner and quicker). In the AI example, the implication is that the final product isn't benefiting consumers.
It was always the only end game. AI on quantum computers is the ultimate computing technology — basically the final research technology in our made up little game of Civilization.
Anytime you create technology that sufficiently replicates the creators, you end up with the spirit of the creator embodied in the technology. So, of course digital brains are going to do weird things like crossword puzzles and sodoku at scale, because it's the same kind of useless shit we do to entertain ourselves.
You get into this conversation whenever you dive hard into cyberpunk, which is so many, many things. Even the Internet itself started out that way. The endgame started as a game, and it will end up being a game, played by our creations as odd mirrors of their creators.
I think there's a lot of people who subscribe to doing the same thing to save the planet, of which I have a keen interest. Solarpunk is the name of that movement, and it also has similar crazy ideas.
We're inventing digital brains. It's literally an architecture designed to be removed from being anything remotely human as it's a mimic or replacement technology for intellectual capacity.
If you really take a step back, the endgame is crazier than just bots producing content merely for other bots to consume (which basically describes the vast majority of scientific papers these days too, ironically). When you take that absurdity and multiply it by tens of thousands in terms of efficiency, the whole system we're building looks WILD and almost inconceivably strange to the way we do things now.
I know this is only tangentially related, so I appreciate your understanding that I already understood that and wrote this anyways. :)
There’s a very nice book about this subject called “Dark Pools”. I found it very enjoyable, you certainly seem to have the knowledge to enjoy it even more :-)
I'm in the medium frequency trading world. (calculate lots of market conditions, send at most a few orders a minute) I'd like to understand the HFT world a little better.
Is it basically "do what MFT does, but faster", or is there any specific advantage like getting into an order queue with priority?
also having worked in the space: HFT should not exist. Break up the day into segments and have a single crossing. Do it every five minutes or whatever for sufficient timeliness. the millisecond race does not make anything better for anybody except the people doing the trading.
I think you would still end up with races, they'd just move around. IEX's "speed bump" was just marketing and didn't really matter to actual HFTs.
Plus, some of it is unavoidable if you don't have a single unified exchange. Where there's latency, there's inefficiencies, and where there's inefficiency, there's profit to be made. And competition among exchanges is healthy for the ecosystem, so I don't think we'd want to consolidate.
And lastly, HFT has consolidated so much that I don't think it's worth worrying about. Virtu literally switched sides and make most of their money on order execution. Industry-wide, HFT revenues are down like 80% over the last 5-8 years. Between wholesaling/PFOF taking non-toxic order flow off the lit exchanges, and banks finally wising up on not being pants-on-head about their order execution, it's literally just sharks in the pool now, there's not even any water.
Single point of failure is a simple justification right off the top of my head. What happens if you have a critical failure that brings down an exchange for several days?
As a counter I had some VW shares I couldn’t access for 6 months because they were moved to another exchange. 20 calls to the broker to find out what had even happened. Eventually I was able to sell them.
Well, your first problem was investing in VW (KIDDING!). But seriously, I usually stick to self-serve stock purchases. Was this a situation where you had to contact a human to make the buy in the first place, or was it that you bought it yourself, then VW changed up their infrastructure and inadvertently locked you out?
Self serve, one one of those trading sites. No human contact, until the site wouldn’t let me trade the shares. I think the exchange changed up things and I am buying from another country so communication issues etc.
I checked my email history on it. There isn't much but from what I can tell the exchange was changed that they were traded on but there was also a change in back-end broker and there was a delay of them moving the shares to the new back-end broker. So maybe the change of exchange wasn't the actual reason, just a coincidence.
I assumed that some exchange provides enough volume that without it there would be worries that you aren't getting true price discovery. Isn't that a concern with low trading volume? Not that that would necessarily be true, but there would be enough concern to cause some people not to trade. And others to want to trade specifically with those kind of nervous people wouldn't because the people they expected to profit by were staying put. And then it would cascade.
Also, "exchange competition" is a thing in US stocks, but not, for example, in the futures' market. And there's HFTs in futures too, so having a single exchange wouldn't "remove" HFTs (not that you'd want to).
Running the matching engine at discrete times doesn't eliminate the speed race, as that's mainly about jockeying for queue position. You could put in an auction instead, but that's just changing the rules of the game rather than eliminating the game.
But really, the key thing is that liquidity provision actually is a service -- market makers are basically selling insurance. There's money to be made doing his, so people will compete to do so. That's what the millisecond race is about. Faster trading -> less risk for the MM -> less capital needed -> lower profits acceptable. Contra what you're claiming, the race is cutting into profits, not raising them.
If you want to reduce market-maker profits, crack down on payment for order flow, and let everyone compete for a chance to trade against it.
He’s referring to the fact that the HFT “provides liquidity” as in getting a share off your hands and flipping it to someone else that is not you a moment later. This way you didn’t have to wait 3 seconds or 1 minute or whatever to sell your shares because someone knew they could flip them (alpha) on a short notice so they participated in a transaction with you. He’s assuming that without the HFT, the transactions wouldn’t be as frequent and you’d have to either a) wait for the fill on your trade , or b) lower your ask price if selling under time deadline. Analogously for buy orders.
The only parties with semi-global visibility are prime brokers by definition; they see every position of everyone who custodies with them.
The parties with global visibility of the US market are TRF (trade reporting facility) and those are the only parties who can sort of evaluate the HFT claims without bias or vested interest. Most of the studies in the field have some sort of an angle or vested interest so it’s hard to evaluate the veracity of the claims one way or another.
A counterpoint to HFT is that stock markets existed before the the advent of computers and they had runs, panics and blow ups just like regular markets do now.
I don’t have time to reply properly until a few hours from now.
In the meantime what I can say very simply in the hope that someone else knowledgeable can contribute earlier.
It’s simply an empirical fact that the costs of intermediation to the system are the lowest they have ever been. The US and other global electronic markets are incredibly efficient and deliver unmatched liquidity, information efficiency, and the lowest costs to the entire capital markets than at anytime in history. That march forward is continuous and brutally competitive
There are many many way to see this and measure it, any serious quantitative analysis, by professionals, for instance trained in econometrics and with access to the raw data, like those at say the Fed, or those operating exchanges, as examples.
Your dismissals seem to be very biased and you’re not allowing the possibility of being wrong.
As a practitioner who worked on the systems you’re citing I can tell you’re wrong on many fronts.
As far as costs are concerned, you’re not answering the most relevant question that applies to most users of this forum: costs to retail traders, and are only tackling the institutional side of things. Nobody from retail concerns themselves with costs of intermediation because those costs are irrelevant to retail. Not at the order volumes that don’t even incur slippage.
Plus, fees are only part of the game. I can give you an NBBO improvement now to be compliant with the regulation, but I am not guaranteeing anything in terms of price in the next second. See where this is going?
Edit: by the way, what you’re repeating is what I jokingly call a “party line”. Especially if you look at the hard cold data. For example people often cite narrower spreads etc. but even with Interactive Brokers you can get MPID displayed on the NMS aggregated full depth order book and see who quotes how much and at what levels and the spreads don’t add up to the half of the myths people keep repeating. It’s easy to hide the real numbers in the _averages_ and various other statistics though.
I’m a practitioner and no, the costs to retail traders is included and analyzed.
The most vocal critics of HFT are very often previously practitioners who are upset when their strategies and models becomes obsolete are are outwitted by even more efficient operators.
Give any example of where you are going?
I will say RegNMS and NBBO regulations are actually preventing even further efficiencies. Dark pools and off exchange matching or internalization are complex topic that are easy to misunderstand. There are absolutely bad actors to be found in the system. This is true in any system. But on aggregate the system is continuously reducing costs and improving efficiency.
Intermediation costs are a friction on the real economy and capital markets and they will always exist, but on aggregate they are dropping for all users, institutional and retail.
In regards to party line, it’s absurd, you can simply take the aggregate income, not profits, of all the top HFT operators, their income is their counterparties costs, and when attributed per market, this number is continuously dropping on aggregate. If you look at an individual firm, you can see it’s income growing, however that will come from 2 dimensions, either expanding their operations to other markets, or taking share from a competitor. However if you sum all profits across all HFT operators on a single market complex, say US equities, on longer economic timescales, this number is continuously dropping. Obviously during periods of market volatility this number can increase, but the trend over years, will be always downward.
HFT is a absolutely brutally competitive industry.
I’d be interested in what you saw working on such systems seem “wrong” or “unfair”. My guess is you don’t understand that all is fair in war and that includes HFT, as long as it is legal.
I am not disagreeing with you that the costs are dropping and that participants benefit from that. In fact I agree. Your projection that I find the field unfair is also unfounded. I love the field and find it immensely interesting. I just don’t take mythology surrounding it at face value because I often found it to be a) outdated b) full of mythology but no hard data.
What I am saying is that the thesis that the costs are dropping due to HFT style strategies has not been proven. Majority of the HFT tend to be market makers which tend to help with liquidity but not all are. I agree that liquidity helps offload or acquire large stakes.
As for NBBO and the effect a regulation has on markets you seem to be extremely US centric but if you go across the ocean and find out that Europe has no concept of NBBO at all, and that a retail person trading experience is equivalent for getting a different price on Amazon depending on which web browser they use you could imagine how that would make an average retail person feel.
Another example of the regulation is trying to move a large stake outside regular hours when NMS is suspended. Why do you think OTC block trades are pre-arranged at a fixed price? You can even look them up in relevant reporting facilities.
As for dark pools, ATS and internalizers there’s nothing difficult about them. I don’t think you’re doing anyone any favors by obfuscating an extremely simple concepts. Those market participants with their specific mechanics that are learnable. What’s complex is how to devise strategies and how to rely on the liquidity sources to get your desired fills and desired rates. You’re describing bread and butter of trading at an institutional level and the fact it seems difficult can maybe be attributed to the fact that it is opaque, doesn’t enjoy public communities, e.g. very little blog posts exist on the subject and the knowledge is sort of centralized to a specialized corners of the industry. But it is learnable without great difficulty if you have access to the resources.
What’s surprising to me is that a lot of people can talk about abstract concepts in computer science and then don’t connect facts that bridge into a separate discipline. As an example I could refer to consistency that people love to rave about but somehow forget the concept the moment capital market is introduced.
And yes, I’m familiar with the no trade theorem and relevant academic concepts but they are all models that often don’t translate to real life due to extremely limiting assumptions that don’t enjoy any connection to reality. I think it’s a problem with the community in this particular industry to take academic market models at face value. Often you will find that looking at a model carefully with assumptions that mirror real world, your performance gets better.
It’s true I’m US centric. What I do know about European equities market structure actually points to one specific problem that actually we also have some of in the US. I’d suggest you look at clearing house collateral regulations, especially around ETF transactions, creation and redemption, and posting of OTC positions. That part of the system, which btw isn’t HFT, is very very shady.
I agree it’s all quite understandable if people had the resources available. I would argue part of the reason it is opaque is the fears of operators that too much public attention can bring a lynch mob. This business can literally be one of the most profitable activities on the planet, but contrary to populist sentiment this is not evidence of wrong doing, it’s simply because it’s so automated and scalable and all electronic.
Back to ATS and dark liquidity, this isn’t as trivial to explain as you suggest, it requires understanding of flow toxicity and information theory to be understood completely. You must know that in markets, transactions can be beneficial to both parties as they have different objective functions and timescales.
What evidence can be presented that supports the idea HFT operators are extracting increasing costs on the system? I’m not aware of any.
Yes, toxic flows and adverse selection are the magical buzzwords that get thrown around a lot. I’m familiar with glosten-milgrom model and private information and all related paradoxes.
What you’re failing to disclose is that there is an easy way out of solving the adverse selection problem. For example you could buy “uninformed flow”. I think that you could agree that for the touted sophistication of the field you would expect something … more sophisticated?
Again, don’t get me wrong. I love the field but I think it’s stagnant in certain aspects and I like to have a sober view of it.
EDIT: As a thought experiment, envision a setup where the trading strategies compete on the basis of the strategy itself, with a single global market with a single API that takes bids and offers in rounds and anyone who wants to is allowed to participate for free with no fees whatsoever. For the sake of example, suppose it’s a government owned and operated project just like the GPS (for which you don’t have to pay a subscription) in your phone. No market access fees, no market data fees, no preferential latency treatment. A perfect coding competition playground.
How many current market participants do you think would survive in such an environment and if your answer is different than the current number, why?
While I fully believe you are an experienced practitioner however I feel your thought experiment actually shows you don’t understand the underlying nature of capital markets in capitalism, the relationship between risk and liquidity, and the fractal nature of timescales of participants. I’m fortunate enough to have both a background in HFT but prior to that a background in many other aspects of capital markets and the broader financial system, which gives be a better perspective than the typical HFT strategist or engineer.
It’s too late in the day for me today to attempt to write a coherent and accurate explanation of exactly why I say that. However I will say you don’t understand this isn’t a zero sum game and it’s an incorrect argument to claim HFT is an arms race in a game like a war, which is a negative sum game.
You start sounding a bit like chatgpt with the “it is I who have the sole power of understanding” and self contradicting arguments. I never mentioned zero sum games or anything of the sort. Please don’t ascribe your thoughts to mine. I did helpfully screenshot your comment invoking war and claiming HFT is like it. I can jog your memory if you’d like. Now you’re claiming the opposite. Your arguments start becoming internally inconsistent.
Your risk argument doesn’t hold water because liquidity is not the only risk. There are at least 20+ I can think of from the top of my head without even trying.
HFT is war but the system as a whole isn’t. My impression of our interactions on HN is you have a valuable insight that perhaps I can even agree with. However unfortunately I don’t know what it is yet as you haven’t explained properly what your hypothesis of negative effects or externalities HFT operators have.
HN isn’t the best forum for this type of ongoing discussion. Engage me on reddit if you like. u/alchemist1e9
By the way, your description of profits dropping for market participants reminds of the concept you may find interesting.
Here’s an example of P&L dynamics:
At time t, your throw an unbiased coin. If it comes out heads your wealth multiplies by 0.6 with probability 0.5, or if it comes up tails, your wealth multiplies by 1.5 with probability 0.5.
Now you could simulate this process and take two averages. One is an ensemble average, averaging over many trajectories (of many participants) at a pre-defined time step t.
The other average is a time average (what happens to a a single trajectory picked at random over time). You may find the result interesting and close to what you’ve just described. This result is because of the process being non-ergodic; but it exhibits a few “winner takes all” and has nothing to do with the properties of the winners. It is a purely random property. An illuminating exercise. There are quite a few other results like this one from the field of stochastic processes that relate volatility bounds to your expected P&L, etc.
If you know more than others, that's great—please share some of what you know, so the rest of us can learn—or else don't post. Sneers and putdowns only make everything worse.
Yes, both. Basically, I learned early that I needed a backup power supply. I have a couple that power my computer, monitors, and the internet router. For the network, that one is really tricky. In that, you're screwed. I've only had this happen once and then I needed to manually sell all my positions. Luckily, I didn't lose any money.
QQ for everybody since OP is using IB API which is notoriously bad that many wrappers have been written for it (ib-insync).
This is a general question, I'm wondering is there any good framework/wrappers out there that one can learn from to code up a complex trading application?
Like dealing with all the asynchronous nature of process/submitting trading and quotes messages.
Yeah, I ended up taking https://github.com/gofinance/ib and rewrote my own wrapper. This took a long time but has been stable since. I'm basically only doing buy lmt, sell lmt, cancel, and updates orders though. So, the logic is pretty simple. Catching all the return messages and structuring them correctly took tons of debugging, trail, and error. Basically, mapping the messages into the correct orders for state tracking.
I use ib-insync with asyncio and it is surprisingly easy, the most difficult is order management. Jason Brownlee’s SuperFastPython was the most approachable asyncio explanation I found.
I've been building a bot with Typescript because it is what I know and Python because of the tools available.
It is an overwhelming lonely endeavor. With all my other projects, I've always worked on teams, although they have always been very small teams and most of my work was autonomous -- still there was the occasional meeting and stand-ups. I've been working on this for six months and thought about bringing a friend onboard for no other reason than to not be alone.
Myself included. One issue I've run into with the public Discords is there are just too many people and therefore too much noise. A smaller non-public group would be ideal.
How can you hope to compete with the pros? You're not going to be faster than them and unless you have several PhDs in maths it's unlikely you'll come up with a better algorithm.
The only way I can think of is to get data that they can't.
Not everything has to be super fast. I was using quantopian to trade based mostly off fundamentals data on a monthly rebalance basis successfully. At least enough to beat the sp500 for a few years- I stopped because I went to a hedge fund that prevented me from trading that frequently and in most small caps. If you want an idea take a look at the piotroski f score and factor based investing. Aqr for example is still largely factor based.
There are plenty of strategies that can be profitable on a small scale but which just don't work as you scale up the capital or leverage. Such strategies can be simultaneously profitable for a small operation and not worth the bother for most larger trading shops.
This is just one example, but at a small scale you might be placing buy orders with let’s say $1k or $10k. This is insignificant compared to the total amount of money being traded, and will not affect the stock price.
However if you start increasing scale to $1mm or $10mm, your buy or sell orders begin to actually move the stock price itself. You might not be able to successfully sell $10mm of stock without dropping the price, signaling others to sell, further dropping the price, cutting into your own profits.
For the point I'm trying to make, the particulars of the strategy don't matter. What matters is the fact that the larger and/or more numerous your orders are, the more likely you are to move the price, and any such price movements will necessarily be disadvantageous to you.
I.e., if you're buying, the larger your buy order is (well, assuming a visible buy order) the more likely it is that liquidity-adding sellers will increase the price of their sell orders. Also makes it less likely that liquidity-taking sellers will want to trade against your large buy order, because (like everyone else) they'll tend to interpret your large buy order as a sign that the price is likely to increase, so not as good a time to sell.
You could of course use hidden orders to avoid some of those disadvantages, but hidden orders have their own set of tradeoffs too.
This article is well-timed as I was about to use options monitoring as an excuse to get into some other technologies. Probably not going to make a ton of progress as a side project, but I'm always up for a chat!
Really funny coincidence that I am seeing Interactive Brokers mentioned in this good article. Story + rant time, feel free to skip if you are not interested how one guy gambled and lost. It's also a tentative call for partnership if somebody is interested. And a call for chat if anyone has any interest in the topic.
(It's also kind of off-topic, my apologies for that. To me it seems semi-related but would agree with mods' assessment if it doesn't match mine.)
---
I recently "broke up" with some extremely toxic "investors" that wanted me to do a fully parallel trading demo -- meaning it receives ticks for N instruments (I successfully got to little less than 300) and trades with each of them depending on strategy. All in real-time.
I got very far but the open-source libraries for Interactive Brokers are quite low quality in general and it was very hard and slow to progress (one of them couldn't even post orders, another used Mutex-es for "parallelism" which was of course not parallel at all, another one seemed to work well but only worked on servers of older versions compared to those I have access to, etc). I also had to gather code from separate places and assemble my own Frankenstein as I went along.
Eventually I muscled through but by that time I have drained all my savings, my tax fund and even got a new loan. And the liaison + the investors of course refused to acknowledge the demo was basically 90% done (couldn't do full parallel trading due to defects of the IBKR libraries I have used and I used like 5 of them, and was in the process of repairing 2 of them to unlock the said full parallelism). They refused to send a small pre-funding wire (we're talking something small like $30k - $50k, not millions; for the work that was done, namely months of professional Rust programming work, that's a -75% discount if we look at market rates).
They had all the proof and paper trail they needed to see that I was very close but I had to stop because I was literally about to be unable to pay rent and bills. They still did not concede. Obviously I picked a job and dumped them but I still have some regrets because the door is technically still open (they have not cut my access to the IB Gateway servers that they own), but other factors like now-ruined health are seriously getting in the way as well. Not to mention completely shattered trust.
They insist "if you just finish the demo we'll give you money" and used every gaslighting technique I knew about (and many I didn't know about, so I learned a lot about gaslighting from them, lol) to try and coerce me to keep working for free with zero guarantees of funding -- but I no longer trust such rich investors to fulfill a promise without a binding and legally enforceable contract; I am in Eastern Europe, they are in the USA, even if they sign contract and violate it I practically cannot do anything to them i.e. I can't afford to travel and sue. Also they insist to get access to the source code but swear to everything that's holy that they will not run away with it and never give me a penny. Which is exactly what I think would happen.
I had to draw the line at one point. To me it was red flags all around.
---
I made many concessions and I will not make another one until they do. I'll be finding a new job soon again since the previous contract was agreed upon to be for several months, I helped a team accelerate bootstrapping a business-critical product (and we did that successfully). And then maybe, just maybe, after I settle a bit, I might work an hour or two on this again during some evenings. Maybe. And that won't be used to provide the demo to these toxic investors -- I'll use it to have a proven implementation that I can pitch to other people. These guys don't deserve the time of day from me.
I have quite a lot of good code (mostly Rust, but also some Elixir and Golang -- I experimented a lot) that interfaces with Interactive Brokers...
How did you get into this kind of shady arrangement? This is not how it works if you go to a reputable prop firm. Also sometimes they already have infra setup for you to build on.
A liaison of these "investors" found me somehow. I can only hypothesize it was through HN.
And yes I was tempted. I also don't have any trading credentials. I'm simply a senior programmer with an eye for details who always wanted to try his hand at algorithmic trading. So yeah, I got hooked. :(
It all boils down to what do you know in terms of languages and how good a quality of libraries you can find.
IBKR is kind of an elitistic VIP club, not just anyone can gain access. Thus you won't find a lot of libraries. There's a good number of them but the quality is not great.
I always wanted to learn OCaml by the way but in my current life and career phase I still can't justify the time and energy expenditure, and I know it will be significant.
Yeah, I've been messing around with this for 4+ years so. Got to see COVID nose dive, meme stocks, massive build up, market taking a dump, bank collapses, and now a bull run again. I have no idea if it was always like this but it has been in insane.
I find this article to be absolutely pointless and clickbaity. It can be boiled down to: "you can use poligon.io for market data but algotrading is difficult anyway so there is not much to share yet".
Hey, I'm sorry you feel that way and I wrote it. Personally, I wanted to share the high-level structure of how you'd build your own system. I would have loved to have seen something like this when I went down this rabbit hole in that I needed to figure it all out myself. What do you think would have made it better? I'd be happy to roll that back in.
One of the most misunderstood concepts about algorithmic trading is that speed is not a critical factor for the majority of systems. My systems like https://grizzlybulls.com/models/vix-ta-macro-mp-extreme have been crushing the market with live trading for 3+ years and yet trade on a frequency of only once ever 18 trading days on average (some a little more often, some even less), and are set up to only generate signals around hourly breakpoints.
The last 18 months have been weaker than the first given the enormous structural shift in the market in this high inflation and rapidly rising interest rates environment, but we've still managed to deliver a return of +14.11% since the site launched in Jan 2022 compared to -7.83% for the SPX. We've managed to do it without any use of leverage and also with lower drawdowns as well of -16.48% vs. -27.57% for the SPX over that time frame.
The models vary greatly in the number of external data sources they pull from to compute signals. The simplest ones like https://grizzlybulls.com/models/ta-mr-basic and https://grizzlybulls.com/models/ta-trend-basic only use a technical analysis calculations on the raw price and volume data of SPX on a few different timeframes. The most complex pulls data from dozens of external sources as it incorporates macroeconomics, monetary policy, various market sentiment sources, yield curve, valuation data, fundamentals, earnings estimates, etc.
It's also worth noting that every additional data source adds some risk of that data source being down or publishing inaccurate data during real-time signal calculation which can cause inaccurate signals, so in order to justify that risk, the external source must meaningfully contribute to alpha or better risk-adjusted returns.
None of them involve a human element in real-time. However, they are occasionally updated as new data comes in, but any updates only apply going forward so as to preserve the live trading history accurately (live trading start date varies by model from mid 2020 to jan 2022 with 2009 - 2020 being purely backtest for all models).
Your chart including what looks like backtesting data starting 2009 looks a bit disingenuous. It’s a totally random starting point. It’s obviously not your actual performance, if you say you started 3 years ago. It’s made to look like your model actually outperformed the market significantly.
Kudos to you if you really sit on an untapped gold mine, but imho, there are some red flags that makes me not buy in.
Thanks for the feedback. I try to call out in several places in the app that 2009 - 2020 is backtest only, but perhaps I need to make it more clear. No one should ever expect a model to trade as well in live trading as it backtests, and that's been true of nearly all our models (save the anomaly of TA-MR-Basic). However, there's more than enough room in the returns and drawdowns to underperform the backtest while still producing significant alpha, which is what we've seen with 4/7 models and especially with the top two.
However, the April 2009 start date is not actually random--it's the first start date for which intraday futures data is available for more than just front month contract. Several derivative indicators of the VIX futures curve are the most foundational to all the VIX-based models, and they simply cannot be processed without it. The VIX futures were only created in 2004, and I've scoured the internet for intraday data for more than just front month (can't create the curve if you only have front month data), and the earliest it can be found is April 2009.
Go itself, or any other language won't give you too much of an advantage. What gives you advantage is trading algo, which is always hard to find. I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different.
I could use Visual Basic, and it would be better than Go, Rust, or whatever it is out there, given the algo and strategy are flawless. Language is just a tool. It's great you used Go, but I think the title is a bit misleading - people think of it as some kind of advantage. It isn't.
And for HFT trading a language with Garbage Collector is not a great choice IMO.
Hey, I write this. I did program this in Go. The reason is that it's my go to programming language. Using it is actually a disadvantage in that the industry uses C++ and Python but that's just what I know how to program in. I wasn't trying to be misleading. Go does work really well for taking in data, doing something with it, and then calling a remote API. So, it actually works really well but if you wanted to get a job based off this it probably wouldn't help you.
Go is great for network heavy apps like algo trading!
I used Go to write trading algos that would find small windows of triangle arbitrages in crypto exchanges. Made me some money but the risk of a big loss made me stop pursing crypto trade and it required too much time and attention. It's a full time job from my experience. Reasons I could lose big at any given time if I scaled up the stakes:
- Exchanges temporarily pausing some specific crypto trading for N reasons (happened very often) while I'm in the middle of the arbitrage.
- Getting caught in the middle of a pump and dump event (also frequent)
- Any algo mistake that would perform excessives trades in succession would incur huge losses because crypto exchanges charge %.
Also, most cryptos have too low volatility. Every time I tried to scale my bot would start interferring oo much with the market. And it wasn't even much money.
It was my (admittedly ignorant) impression that C++ was preferred over garbage-collected languages because you'd be more likely to avoid GC pauses. Yes, of course I know that Go's is super efficient, but if you're doing a million trades a day even a tiny percentage of slowdowns that would be otherwise preventable could be a career-limiting move. Am I just behind the times?
I think you're right! Go will never beat C++ in this aspect.
I used Go because that's what I knew and for the non-professional trading I was aiming, C++ wouldn't have made much difference. My bottlenecks were network (1s+ per trade roundtrip) and chaotic unreliable crypto markets.
Just note that Java is used in HTF, but it's a different beast than our average CRUD Java. For example this article states:
"Essentially, we use a contrived form of Java that avoids all the Java constructs that make things go slow. We only use the constructs that are fast and efficient, and we avoid all the garbage."
Yep, you can with GOGC=off but then I would have to learn about how to generate less garbage in Go for long running processes. Which is good knowledge even for non-trading applications. I just didn't have the energy and time to try it.
Maybe? Every language one would have to work at to make performant, and generally pre allocation is what often happens to do so. allocation is probably more expensive with a gc’d language like go than C++, but both would want to avoid it.
I read like 5 or so books about trading, mostly the classic ones, and tried to toy around with backtracking some self-made strategies using MetaTrader: https://www.metatrader4.com/en/trading-platform
First week was funny. I kept thinking I found goldmines with fine-tunned stochastic lagging indicators, only to realize that I overfitted and an algo that made me rich when tested against a certain period of time would make me go broke when applied to another period.
I tried applying some defense against local maximum, but to no avail.
Best I could do is tie after paying exchange fees. And that took me endless weekends toying around with code, reading books and articles.
My hunch is that algos that don't rely on external signals (i.e. news) need extreme technical edge like High Frequency Trading have with their ultra low latency and premium exchange data feeds.
And I wasn't ready to dive into news sentiment analysis and other external indicators.
You overfitted. That can happen due to human optimisation as well as computer optimisation. After you have experimented on some historical data for a while it becomes less useful.
> What gives you advantage is trading algo, which is always hard to find.
At the end it is necessary to make a decision whether to buy or sell (and how much), which will compete with other decisions made based on some logic. Developing such a logic (strategy) manually is of course quite difficult. I developed an intelligent trading bot which derives its trading strategy from historic data:
> I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different.
It is a typical situation. The whole problem is to develop a strategy which works for future (unseen) data. Even backtesting algorithms should be designed in such a way that future experience (data) does not leak to the past.
>And for HFT trading a language with Garbage Collector is not a great choice IMO.
Depends. I've heard of places that use Java by creating a huge heap, minimising allocations to almost zero, and just restarting the app periodically before it can fill up. You can achieve very good performance doing this and you don't have to worry about memory bugs.
I love finding other people who enjoy the pure feedback loop of code, strategy, and money that trading provides for me. I’ve been working on timing and correlations between stocks and indexes as a source of alpha and having some success. Now trying to automate the process and absolutely loving the work along the way. Not trying to play market maker and not jumping between stocks. Instead honing in on an accurate model that monetizes a few really well. Some tools I find useful: TradingView and Pinescript indicators and strategies, excel models with exported data for backtesting, Python and Go for the backend machine learning, ChatGPT for faster iteration on new code. Would love to talk shop if you guys are interested: trading @ dianazink.com
Are there any examples out there ofactual algotrading strategies that made money in a real market, say within the past 10 years? I know these are closely guarded secrets, but figured someone may have published a strategy that ceased being profitable.
310 comments
[ 2.9 ms ] story [ 337 ms ] threadI've been really wanting to use Go, but as you say, much of the community is Python due to the data analysis strengths. To the detriment of the other things Python does do poorly.
Can you give some thoughts with your experimentation on the following from a Go perspective.
1. Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks)
2. Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB
3. If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)
I had not looked at Mojo... thanks for the pointer. In the past, I've had issues with library compatibility with compiled derivative languages of interpreted ones (eg: Crystal of Ruby, etc, etc).
Know if Mojo directly uses existing Python ecosystem?
I've been using Polars, etc.
They say they do, including libs with C bindings.
I've been basically, just manually coding the algorithms from python into Go. ChatGPT is amazing at this. I really only just about 4 so it was a one time thing.
> Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB
Yeah, I ran into issues and then was like what would be the fastest, then just went in-memory. I download all raw trades/quotes each night and store then into gob+lz4 compressed files. Then for backtesting and stuff I can load these in and build the aggrogate bars on the fly.
> If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)
Yeah, I have a historical lookup table that I build nightly too. This gives me a reference point when I'm doing % change calculations and stuff. I should probably have mentioned that.
Can you share your approach for plugging in various strategies? I quickly learned that having a pluggable strategy system is tricky as it could span across multiple layers of the system.
Also, with backtesting, are you storing and replaying all the quote/tick data? or just using the historical aggregates?
For back testing, I download all raw trades and quotes, and put them into 1 file sorted by time (1 file per day). Then, I compress them using lz4. This allows me to sort of replay the entire market and build up all my intraday backtesting from the source. This took me a long time to figure out and build but has been so worth it. So, I have an off-line script that basically, loops through these files, and replays the market, and makes simulated trades, and then spits out what would have happened. There is a GUI for that too so you can go in an explore the trades and see what triggered the buy and sell. I have seen nothing that goes backtesting for intraday like this.
This is super inefficient but I'm just building the aggregates on the fly. I could probably cache them somewhere but it takes maybe 4-5 minutes to replay a days worth of trades/quotes and build all this so I haven't bothered yet.
I’d like to play with something like this but I couldn’t be bothered to build it all from scratch.
I've also got a few questions regarding the way you manage memory, so I might ask you a couple !
Excerpt from his site:
"Tech Trader is a fully autonomous trading system live with no human intervention or updates, now for over 10 years. It is unique from conventional algorithmic systems, not only because it actually is fully automated, but because it takes a "human" approach to markets. It is not quant. It is not stat-arb. It is not high frequency. It is a program that looks at stocks the same way a person does but with the cold discipline and infinite attention span of a machine. It is analogous to having a thousand independent traders each focusing on a single stock, as opposed to a single quant manager trying to make sense of a thousand datapoints. A person doesn't think through stats, correlations, or complex math models when trading, and neither does Tech Trader. Tech Trader leverages technology to do what human traders do at scale rather than approach markets from the point of view of an academic, mathematician, or scientist.
Since its launch in Dec 2012, Tech Trader has been trading live capital completely on its own, fully automated in the truest sense with no human input, no tweaking, no updates. It is, for all intents and purposes, an autonomous hedge fund, one of the first to truly trade unsupervised for years on end. Whereas many "automated" or "AI" funds may have a hundred scientists providing the actual intelligence behind the curtain, the creator of Tech Trader consists of just one person - a self-taught individual going by the gaming moniker pftq, who created the system at age 21 and has long moved on to other interests. "
I call bullshit. Reads more like a scam for trading signals.
https://www.sec.gov/Archives/edgar/data/1653903/000165390318...
I can't say any more about its validity. Ask your financial planner if techtrader is right for you :)
I have no idea: does IB send the price feed in real time? (they certainly send the data in real-time to TWS as it constantly updates right? But is the order book available through their API?)
Basically and even though I know this was published on polygon.io's blog, would that work by only using IB / TWS's API?
[1] https://www.interactivebrokers.com/en/?f=%2Fen%2Fgeneral%2Fe...
I've been going through this journey myself. I started learning on Tradingview, then bought Build Alpha to discover how to test strategies. I chose Portfolio 123 for my automated factor trading but had been working towards creating a program/basket trading system like yours that can act on intraday data.
I moved to long-term investment until I could build a simulator capable of verifying the correctness of my investment strategies using fuzzy testing ideas stolen from TiggerBettle.
I have almost two years of polygon quotes and trades for the whole market captured with a monotonic timestamp to be able to replay the data –and test the handling of polygon socket glitches.
I'm focusing initially on capturing the data in a way that allows fast replay and aggregation, similar to what Kafka can do with topics but in-process using zig and custom memory-mapped data structures. My idea is to be able to generate signals like VIX (once I add options data), ETFs, and indexes and hopefully be faster at doing so than others :), please HN folks, call me out here if I'm being too naive.
This has been a three-year learning process for me. I have been a retail investor for +10 years, but over the last three years, I've gone deep into learning algo-trading, drank del Prado Kool-aid, and read numerous trading and investment books.
I'm now focusing on my technical chops to build the engine to build order books for individual stocks, baskets, and indexes with realistic market prices. I aim to develop a system that can get as close to the market price in the next dollar bar as possible.
This has been a very lonely journey, and after reading the responses to this post, I'd love to connect with others on a similar path. Sending you an email!
I've used ATR bots for years, and would love to hear your thoughts on how what you're building delineates itself as a distinct product strategy beyond just the programming language GO.
I've used wonderbit, zigz, and some of the others.
Cool project.
I was curious in getting a websocket setup with Polygon after seeing your post, but I noticed it's $29 for first package with the websocket feature. Then I noticed that the package with real-time data is the Advanced package at $200 a month. I'll admit, real-time with unlimited API calls sounds pretty sweet, as my strats rely on to the second data. However, I don't know if I can justify $200 as my algos run no more than 78% win rates with a normal bankroll funding it all. I'm looking to save wherever I can while I build these things and get a passive income stream rolling in.
I would love to know which package you were using, as I didn't see it in the time I quickly read your post? Also, any pro/cons to that specific package? Any and all other details are welcome, and if you would rather respond to my email, it's in my profile. Thanks!
It works better for videos and code, IMO. You still have two huge portrait fields for documentation/webpages/etc, or 8 portrait-oriented quarters.
I usually run 8 portrait-shaped windows quartered on the sides, then two side by side on the middle, which are each approximately square.
I don't like my editor window getting too tall.
> 3 screens like that, put the middle (primary) one in landscape, so the array is like a big H.
Honestly my perfect screen for work would probably had something like 1:1 ratio, so I can have 2 nice columns of code that are tall enough that can be split horizontally if needed while still being useful.
Mostly because you'll never be able to respond to each tick as by the time the tick gets to you the market has moved.
Use the language that you know and can work with the fastest. For retail trading GC vs non GC will never matter at all.
What's cool about this is that you can look at the IB TWS client and see things happening in real-time. So, it acts as sort of a sanity check. I know they have that gateway too but personally I like to look at the client all the time too. My workflow is to run the bot and the TWS client side-by-side and watch it make trades, see how things are moving, etc.
Also I wonder how can there be changes in the price of a stock after market if the exchange has closed? Isn’t the whole point that trades need to happen for stocks to get a certain value?
2. Sometimes you get a cool api and think wow this would be fun, and next thing you know you've lost thousands on boneheaded trades.
I did something similar during the pandemic with Rust with the Polygon API (and instead of interactive brokers, I used tradier). Eventually I learned I actually had more fun building the thing than actually trying to beat the market.
Or do you believe there are more fundamental changes needed so your app can trade in shorting as well?
After a bit of digging, I found this Q&A from the 2019 shareholder meeting: https://www.youtube.com/watch?v=geRIJQJXRVo&t=17980s
And the meeting minutes in PDF form (see page 120, question #32): https://s3.amazonaws.com/static.contentres.com/media/documen...
The text from that document...
32. It’s easy to make 50% on a million, but much more difficult on larger amounts
WARREN BUFFETT: Station 9. We’re just about — yeah, we’ve got time for a couple more.
AUDIENCE MEMBER: My name is John Dorso (phonetic), and I’m from New York. Mr. Buffett, you’ve said that you could return 50 percent per annum if you were managing a one-million-dollar portfolio. What type of strategy would you use? Would you invest in cigar butts, i.e., average businesses at very cheap prices? Or would it be some type of arbitrage strategy? Thank you.
WARREN BUFFETT: It might well be the arbitrage strategy, but in a very different, perhaps, way than customary arbitrages, a lot of it. One way or another, I can assure you, if Charlie was working with a million, or I was working with a million, we would find a way to make that with essentially no risk, not using a lot of leverage or anything of the sort. But you change the one million to a hundred million and that 50 goes down like a rock. There are little fringe inefficiencies that people don’t spot and you do get opportunities occasionally to do, but they don’t really have any applicability to Berkshire. Charlie?
CHARLIE MUNGER: Well, I agree totally. It’s just you used to say that large amounts of money, they develop their own anchors. It gets harder and harder. I’ve just seen genius after genius with a great record and pretty soon they’ve got 30 billion and two floors of young men and away goes the good record. That’s just the way it works. It’s hard as the money goes up.
WARREN BUFFETT: When Charlie was a lawyer, initially, I mean, you were developing a couple of real estate projects. I mean, if you really want to make a million dollars — or 50 percent on a million — and you’re willing to work at it — that’s doable. But it just has no applicability to managing huge sums. Wish it did, but it doesn’t.
CHARLIE MUNGER: Yeah. Lee Louley (phonetic), using nothing but the float on his student loans, had a million dollars, practically, shortly after he graduated as a total scholarship student. He found just a few things to do and did them.
[0]: https://en.wikipedia.org/wiki/Li_Lu
It seems that this is the key to your approach. How is this part achieved?
Gaining 0.5%/day 60% of the time and breaking even 39% of the time looks great until you run into the 1% of the time where you lose 50%.
1.005^260 = 366%
and
1.01^260 = 1329%
In case anyone sees "0.5%" _daily_ and thinks low risk.
[0]: Link to my (non-monetized and WIP) blog where I keep a collection of excerpts: https://sileret.com/projects/warren-buffet-shareholder-lette...
You might want returns that aren't correlated just to an index - this is a major reason to look to invest in (say) a hedge fund.
Very interesting. We can facetiously say that ChatGPT is using you as a medium between setting up algorithmic trading in Go!
Obviously whatever trading bot you're running separate from the actual trading engine itself is somewhat proprietary, but it would be great for the community to get more of this type of software in the hands of other hackers.
Quantopian / Robinhood tried and failed, and the numerous clones since then have been somewhat sub par.
You don't have to use the system I am building, but it's worth thinking about that design.
"This aspect, the platform itself, seems to be often overlooked in most discussions. Many conversations revolve around strategies (mean reversion, trend following, linear regression, etc.), and backtesting, without fully addressing the practical mechanics or logistics of strategy implementation, particularly in the context of live, intraday trading."
I'm glad you had fun, OP, but also I think I can shed some light on why most people discuss strategy.
Trading is a perfect storm of ridiculously high tech, ridiculously complicated, ridiculously regulated (Not over-regulated, mind you, this isn't a value judgement. But the amount of regulation is extremely high.), and ridiculously competitive.
But that said, it's the last bit that drives it all. Since it's so competitive, even though building an order entry system, and a risk system, and a position-tracking system, etc is a huge accomplishment (again, congrats OP!), it's table-stakes to even dip your toes in the pool here. Trading shops can attract top talent and robust, bespoke trading systems are basically cost of entry.
So people talk about strategy because everyone already has the table-stakes stuff and are now trying to make money with it.
It doesn't help, too, that lots of market participants aren't even playing the same game. In HFT, we operated on trades with alphas that lasted a few seconds, where races to entry/exit were battled in shaving nanoseconds off FPGAs being able to shoot out orders and microseconds off wireless networks flying market data around new jersey. Meanwhile, banks are more concerned with elections and geopolitics than they are about the weather in Carteret. (Rain = no microwave network for the day). And then there's a million strategies in the middle with alphas that last from hours to weeks.
So it makes it really hard to even speak the same language to each other when talking in common forums.
It's a fun world. I miss it sometimes.
HFTs are definitely playing a completely different game. I was reading about the exchange architectures and how things are actually wired. I'm getting my data from SIPs while HTFs are directly connected to the exchanges [1]. I'm transacting in seconds and they, like you said, are transacting in microseconds, so there is no comparison. Which, in a way is actually nice in that I'm not really competing with them. Or, maybe I am but I can still make some money. haha.
Cheers and thanks for the awesome comment!
[1] https://www.researchgate.net/figure/Latencies-in-the-Electro...
The strategies though are where the discovery is. There are a few strategies that are well known and still profitable but those are largely consolidated to the biggest firms. For everything else it’s a discovery process. And done strategies are only profitable for very short regimes.
I miss it sometimes too, but so much has been consolidated it’s largely a big firm world now.
More straightforward, heh, sure. But still damnably complicated. Which just goes to show how much money and how much engineering talent is invested in this world that these things are so taken for granted.
What did bother me, and was acknowledged by my coworkers, was how much top talent was being pulled away from productive tasks to essentially wank around. Not just technologists either, but all sorts of mathematicians and scientists were drawn to the flame.
We as a society have managed to allocate so many of the “best and brightest” to either fintech wankery or placing ads in front of eyeballs. It’s enough to make one want to give up on capitalism, except that everything else appears to be worse.
There is 0 social impact. That’s the downside of course - but hey, how many jobs out there are really having any kind of positive social impact ? Not 0, but close to it.
Intraday financial games are zero-sum. What HFTs gain, they leech away from mutual funds and pension funds and retail investors and market makers who operate over a longer horizon.
Regarding social impact, the world does have some demand for liquidity and price discovery. Providing those services is both essential and extremely difficult. It's definitely not the most social good I could be doing with my talents, but I think it's weakly positive.
Those things have driven a load of proprietary and open source tech that helps everyone else.
That depends on how you measure it.
Obviously this would mean that countries that previously relied on subsidised labour from those third world countries would have to start paying up. I'm not suggesting that it is a zero sum game however.
The problem with ads is not that they're useless, it's that it is an industry prone to scams and grift. (Because doing advertising right requires all sorts of actual science, and ain't nobody got time for that when there's money to be made.)
t. Worked for 20 years and adtech.
It's nothing to do with "we as a society". I'm a quant trader and know many others, and the vast majority are in the industry because we care about making money not some leftist save the world crap. Even if socialists managed to completely destroy the financial market, we'd just find another way to make money without trying to save the world (e.g. like the mostly corrupt officials in Russia and China when they were communist). "Society" can't change human nature; even mass brainwashing on the scale attempted by Maoist China failed.
I.e. the best and brightest will always work where they want to work, not where leftists like you want to "allocate" them.
Cut the fascist bullshit. Everyone can comment about how society should be run, even people that are very successful financially.
If you try to exclude greed from the design of your societal system, it will immediately fail. In large numbers, economics shows us that altruistic people wash out of the model and everyone operates in their own self-interest (greed).
> Cut the fascist bullshit. Everyone can comment about how society should be run, even people that are very successful financially.
It's not financial success that precludes someone from commenting on how society is run, it's blatant, deliberate anti-social actions. This is the foundation of a social contract. It's the same reason you'll get locked up if you go around punching people in the face.
Also, I don't think you know what "fascist" means.
It doesn’t seem like fascism to you because you’ve done some mental gymnastics to pretend you’re not just crushing intellectual opposition, but that’s all it is in the end. Op is not punching anyone in the face. He/she is just blatantly unsupportive of socialism.
Please, I want to know; what benefits do the actions of the op have for "societal planning"? The person he was replying to was at least honest in admitting that this behavior "served little social good".
Once again, I think you'd do well to actually read what fascism actually stood for. The propaganda of fascism had a lot more to say about nationalism, "shared history" and the good of corporations than it did to say about things being "anti-social" or even mentioning "selfish greed".
Largely, though I receive 1 or 2 job specs every week for start ups with the keywords 'hft' and 'low latency'. Admittedly there's going to be duplication there if you read them closely.
I think it's a bit of a myth that (ignoring FPGAs) that writing a low-latency software trading system is a time/cost expensive process. Anecdata = I worked at two firms where we did a rewrite from scratch with teams of 5-6 people and traded in the market within 3 months. I'd argue a senior dev that's been around the block a few times could achieve similar when you remove corporate politics, and bikeshedding over design.
The big firm part is paying for multiple quants at $200k++ to come up with strategies and historic market data access for trading models. Small firms are getting backing as long as the co-founders are 'ex-CxO from MegaCorp'.
This depends a lot on the complexity of the trading system and the trading venue specifics. A system to trade single stocks or futures can be built, certified and running in 3 months. A system for options market making will take a lot longer.
The big costs for small firms are historic data (if you don't have any), colo, distance to exchange, and number of connections.
From the number of job specs I see, it feels like the HFT/low latency market place is healthy enough that there are always new firms appearing. It's competitive, so it's hardly surprising that if someone has new ideas they'll find a backer.
In the old days before HFT, you weren't sure you'd get the best price. You'd have to rely on a broker to make sure that happens, but as a retail trader you generally got a worse price/out of date price.
Nowadays with HFT you can get pretty much the best price anywhere. Those <1ms HFT traders make that happen. It's the efficient market hypothesis in effect, made possible by HFT.
Now is it pointless to shave off even more ns in the all out war to grab a piece of the order flow? For retail and institutional investors, at a certain point, yes it's completely pointless.
But it's also just pure capitalism at work. HFT firms compete against each other, and the competition is about speed to provide the best price and volume. If you try to regulate with something like enforced delays, then what do you compete on instead?
Capitalism necessitates efficient markets (and efficient markets necessitate HFTs) so any criticism of HFTs is a direct criticism of capitalism as well. I mean this isn't really a problem that's specific to HFTs -- there are just a lot of jobs that we can perceive as providing no value or even negative value (jobs that are possible specifically due to capitalism e.g. payday loans, 2008 style trading, certain scams or predatory practices etc.)
I think we can all agree that this is a flaw, and we're not criticizing capitalism to replace it with something else, but rather just recognizing this as a problem. At the end of the day, a useless job is a useless job even if it exists solely because of capitalism.
People are saying this because, HFT sounds similar to 'crypto mining'. That's people with best infrastructure, the 'big-guys' -- win. While leaving out the retail investors as broiler chicken, pumped with 'drugs' (by influencers) to spend more on imaginary assets, so that they can be used for 'food' by these 'big-guys'.
There are different influencers for retail investors vs crypto. In retail investing there are promises of 'retirement paradise', actual tax deductions, the Jim Cramer-like people (at least what I heard in US)
For crypto investing the influencer are different, the geography is wider. A promise to participate in markets if you do not live the country that has adopted US/UK-based financial services.
- - - By the way, I think the markets will still have liquidity if there is a rule to wait, for say, 30 min before a stock that was just recently bought -- can be sold (unless by a clear fat finger mistake)
This rule will cause the HFTs to stop existing in the current form.
Do any regular pedestrian middle class retail investors EVER actually earn enough to retire on? Or do anything meaningful with? If so.. what is THEIR secret sauce, since it's not HFT...
a)Members of political elite that get insider trading stock tips. (illegal of course). The number of folks in usa congress and senate that become 'very lucky' investors after they join the rank, has to be amazing
b)Lucky
c) everybody else -- that looses.
Overtime, I would say last 30 years, the amount of 'influencing' retail investors to trap them into unreasonable actions had gradually increased.
So the percentage of folks going becoming the victims of the charade, will become higher.
Certainly if you concentrate on ( b ) you create the plausible deniability defense for the manipulators
You are profiting off workers as a middle man in the economy by doing HFT, and trying the justify it by some vague concept of the "correct price". You are producing nothing of value, merely taking away value before someone else notices it is there.
Our entire economy runs on "middlemen". Convenience has huge value to most people.
It's a bit like stock brokers - and why wouldn't we want stock brokers to operate at drastically faster-than-human timescales, because we all know the value of a company changes every nanosecond! And "flash crashes" create opportunities for investors to make huge amounts of money!
And just as crypto has poured money into GPU companies (providing opportunities for enterprising secondary market resellers of same) HFT has poured money into networking companies.
Short of creating the great firewall or helping governments slurp up all the traffic on the internet, what could be a more beneficial application of network technology?
HFT (like finance in general) has also gobbled up lots of tech grads, making it easier to find jobs at tech companies.
Now when I say the same thing about index funds people get all huffy
Regarding liquidity, I imagine it's possible that the index fund might allow you to withdraw money faster than you could sell on your own, but realistically I kind of doubt it.
Likewise with crypto. Personally I don't think that if you're calling an API over an Internet it matters if your trading bot is written in go or python (mine was in python). Use the language you're most comfortable in. The network and trade submission/execution at your broker will be 10x slower than your bot anyway. Unless the size of your operation approaches the size where you can get direct market access which seems to be reserved for big forms only.
That's just it. The whole premise is pretty absurd. The market, the actors, everything. It's so far removed from literally anything remotely human. It's the financial equivalent of an infinite sea of AI bots producing CVs and research papers which are only being evaluated and read by other bots.
If you step away from it all for a second, what the hell is the endgame of this whole hustle.
Anytime you create technology that sufficiently replicates the creators, you end up with the spirit of the creator embodied in the technology. So, of course digital brains are going to do weird things like crossword puzzles and sodoku at scale, because it's the same kind of useless shit we do to entertain ourselves.
You get into this conversation whenever you dive hard into cyberpunk, which is so many, many things. Even the Internet itself started out that way. The endgame started as a game, and it will end up being a game, played by our creations as odd mirrors of their creators.
I think there's a lot of people who subscribe to doing the same thing to save the planet, of which I have a keen interest. Solarpunk is the name of that movement, and it also has similar crazy ideas.
We're inventing digital brains. It's literally an architecture designed to be removed from being anything remotely human as it's a mimic or replacement technology for intellectual capacity.
If you really take a step back, the endgame is crazier than just bots producing content merely for other bots to consume (which basically describes the vast majority of scientific papers these days too, ironically). When you take that absurdity and multiply it by tens of thousands in terms of efficiency, the whole system we're building looks WILD and almost inconceivably strange to the way we do things now.
I know this is only tangentially related, so I appreciate your understanding that I already understood that and wrote this anyways. :)
Is it basically "do what MFT does, but faster", or is there any specific advantage like getting into an order queue with priority?
Plus, some of it is unavoidable if you don't have a single unified exchange. Where there's latency, there's inefficiencies, and where there's inefficiency, there's profit to be made. And competition among exchanges is healthy for the ecosystem, so I don't think we'd want to consolidate.
And lastly, HFT has consolidated so much that I don't think it's worth worrying about. Virtu literally switched sides and make most of their money on order execution. Industry-wide, HFT revenues are down like 80% over the last 5-8 years. Between wholesaling/PFOF taking non-toxic order flow off the lit exchanges, and banks finally wising up on not being pants-on-head about their order execution, it's literally just sharks in the pool now, there's not even any water.
Shitty situation either way.
Shares are not held at an exchange.
Besides, when has an exchange gone done for hours, let alone days? Absent intentional breaks in trading as speedbumps.
Why would you assume that.
Every exchange is a valid place to trade and the sip ensures you always have the correct NBBO
Also, "exchange competition" is a thing in US stocks, but not, for example, in the futures' market. And there's HFTs in futures too, so having a single exchange wouldn't "remove" HFTs (not that you'd want to).
That's not true according to their filings. In their filing for 2022 Q4, I see 185m from market making, and 89m from execution.
It could be all run once a day if it was just to give ability to raise funds via investment
But really, the key thing is that liquidity provision actually is a service -- market makers are basically selling insurance. There's money to be made doing his, so people will compete to do so. That's what the millisecond race is about. Faster trading -> less risk for the MM -> less capital needed -> lower profits acceptable. Contra what you're claiming, the race is cutting into profits, not raising them.
If you want to reduce market-maker profits, crack down on payment for order flow, and let everyone compete for a chance to trade against it.
Can you explain what that is?
> To understand this requires understanding markets very deeply
Apparently you aren't the only person on this forum who does. Are we just supposed to take your word for it?
The only parties with semi-global visibility are prime brokers by definition; they see every position of everyone who custodies with them.
The parties with global visibility of the US market are TRF (trade reporting facility) and those are the only parties who can sort of evaluate the HFT claims without bias or vested interest. Most of the studies in the field have some sort of an angle or vested interest so it’s hard to evaluate the veracity of the claims one way or another.
A counterpoint to HFT is that stock markets existed before the the advent of computers and they had runs, panics and blow ups just like regular markets do now.
In the meantime what I can say very simply in the hope that someone else knowledgeable can contribute earlier.
It’s simply an empirical fact that the costs of intermediation to the system are the lowest they have ever been. The US and other global electronic markets are incredibly efficient and deliver unmatched liquidity, information efficiency, and the lowest costs to the entire capital markets than at anytime in history. That march forward is continuous and brutally competitive
There are many many way to see this and measure it, any serious quantitative analysis, by professionals, for instance trained in econometrics and with access to the raw data, like those at say the Fed, or those operating exchanges, as examples.
As a practitioner who worked on the systems you’re citing I can tell you’re wrong on many fronts.
As far as costs are concerned, you’re not answering the most relevant question that applies to most users of this forum: costs to retail traders, and are only tackling the institutional side of things. Nobody from retail concerns themselves with costs of intermediation because those costs are irrelevant to retail. Not at the order volumes that don’t even incur slippage.
Plus, fees are only part of the game. I can give you an NBBO improvement now to be compliant with the regulation, but I am not guaranteeing anything in terms of price in the next second. See where this is going?
Edit: by the way, what you’re repeating is what I jokingly call a “party line”. Especially if you look at the hard cold data. For example people often cite narrower spreads etc. but even with Interactive Brokers you can get MPID displayed on the NMS aggregated full depth order book and see who quotes how much and at what levels and the spreads don’t add up to the half of the myths people keep repeating. It’s easy to hide the real numbers in the _averages_ and various other statistics though.
The most vocal critics of HFT are very often previously practitioners who are upset when their strategies and models becomes obsolete are are outwitted by even more efficient operators.
Give any example of where you are going?
I will say RegNMS and NBBO regulations are actually preventing even further efficiencies. Dark pools and off exchange matching or internalization are complex topic that are easy to misunderstand. There are absolutely bad actors to be found in the system. This is true in any system. But on aggregate the system is continuously reducing costs and improving efficiency.
Intermediation costs are a friction on the real economy and capital markets and they will always exist, but on aggregate they are dropping for all users, institutional and retail.
In regards to party line, it’s absurd, you can simply take the aggregate income, not profits, of all the top HFT operators, their income is their counterparties costs, and when attributed per market, this number is continuously dropping on aggregate. If you look at an individual firm, you can see it’s income growing, however that will come from 2 dimensions, either expanding their operations to other markets, or taking share from a competitor. However if you sum all profits across all HFT operators on a single market complex, say US equities, on longer economic timescales, this number is continuously dropping. Obviously during periods of market volatility this number can increase, but the trend over years, will be always downward.
HFT is a absolutely brutally competitive industry.
I’d be interested in what you saw working on such systems seem “wrong” or “unfair”. My guess is you don’t understand that all is fair in war and that includes HFT, as long as it is legal.
What I am saying is that the thesis that the costs are dropping due to HFT style strategies has not been proven. Majority of the HFT tend to be market makers which tend to help with liquidity but not all are. I agree that liquidity helps offload or acquire large stakes.
As for NBBO and the effect a regulation has on markets you seem to be extremely US centric but if you go across the ocean and find out that Europe has no concept of NBBO at all, and that a retail person trading experience is equivalent for getting a different price on Amazon depending on which web browser they use you could imagine how that would make an average retail person feel.
Another example of the regulation is trying to move a large stake outside regular hours when NMS is suspended. Why do you think OTC block trades are pre-arranged at a fixed price? You can even look them up in relevant reporting facilities.
As for dark pools, ATS and internalizers there’s nothing difficult about them. I don’t think you’re doing anyone any favors by obfuscating an extremely simple concepts. Those market participants with their specific mechanics that are learnable. What’s complex is how to devise strategies and how to rely on the liquidity sources to get your desired fills and desired rates. You’re describing bread and butter of trading at an institutional level and the fact it seems difficult can maybe be attributed to the fact that it is opaque, doesn’t enjoy public communities, e.g. very little blog posts exist on the subject and the knowledge is sort of centralized to a specialized corners of the industry. But it is learnable without great difficulty if you have access to the resources.
What’s surprising to me is that a lot of people can talk about abstract concepts in computer science and then don’t connect facts that bridge into a separate discipline. As an example I could refer to consistency that people love to rave about but somehow forget the concept the moment capital market is introduced.
And yes, I’m familiar with the no trade theorem and relevant academic concepts but they are all models that often don’t translate to real life due to extremely limiting assumptions that don’t enjoy any connection to reality. I think it’s a problem with the community in this particular industry to take academic market models at face value. Often you will find that looking at a model carefully with assumptions that mirror real world, your performance gets better.
I agree it’s all quite understandable if people had the resources available. I would argue part of the reason it is opaque is the fears of operators that too much public attention can bring a lynch mob. This business can literally be one of the most profitable activities on the planet, but contrary to populist sentiment this is not evidence of wrong doing, it’s simply because it’s so automated and scalable and all electronic.
Back to ATS and dark liquidity, this isn’t as trivial to explain as you suggest, it requires understanding of flow toxicity and information theory to be understood completely. You must know that in markets, transactions can be beneficial to both parties as they have different objective functions and timescales.
What evidence can be presented that supports the idea HFT operators are extracting increasing costs on the system? I’m not aware of any.
What you’re failing to disclose is that there is an easy way out of solving the adverse selection problem. For example you could buy “uninformed flow”. I think that you could agree that for the touted sophistication of the field you would expect something … more sophisticated?
Again, don’t get me wrong. I love the field but I think it’s stagnant in certain aspects and I like to have a sober view of it.
EDIT: As a thought experiment, envision a setup where the trading strategies compete on the basis of the strategy itself, with a single global market with a single API that takes bids and offers in rounds and anyone who wants to is allowed to participate for free with no fees whatsoever. For the sake of example, suppose it’s a government owned and operated project just like the GPS (for which you don’t have to pay a subscription) in your phone. No market access fees, no market data fees, no preferential latency treatment. A perfect coding competition playground.
How many current market participants do you think would survive in such an environment and if your answer is different than the current number, why?
It’s too late in the day for me today to attempt to write a coherent and accurate explanation of exactly why I say that. However I will say you don’t understand this isn’t a zero sum game and it’s an incorrect argument to claim HFT is an arms race in a game like a war, which is a negative sum game.
Your risk argument doesn’t hold water because liquidity is not the only risk. There are at least 20+ I can think of from the top of my head without even trying.
I’ll rest my case here.
HN isn’t the best forum for this type of ongoing discussion. Engage me on reddit if you like. u/alchemist1e9
Here’s an example of P&L dynamics:
At time t, your throw an unbiased coin. If it comes out heads your wealth multiplies by 0.6 with probability 0.5, or if it comes up tails, your wealth multiplies by 1.5 with probability 0.5.
Now you could simulate this process and take two averages. One is an ensemble average, averaging over many trajectories (of many participants) at a pre-defined time step t.
The other average is a time average (what happens to a a single trajectory picked at random over time). You may find the result interesting and close to what you’ve just described. This result is because of the process being non-ergodic; but it exhibits a few “winner takes all” and has nothing to do with the properties of the winners. It is a purely random property. An illuminating exercise. There are quite a few other results like this one from the field of stochastic processes that relate volatility bounds to your expected P&L, etc.
It’s unfortunate that calling out other’s bias is off limits in society too much as it’s often the primary reason people advocate their position.
HFT is not anything like what critics claim and those critics have severe biases.
https://news.ycombinator.com/newsguidelines.html
If you know more than others, that's great—please share some of what you know, so the rest of us can learn—or else don't post. Sneers and putdowns only make everything worse.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
instead of order entry + continuous matching: order entry is allowed, but no matching (like pre-open phase)
then after a random period of time the auction algo runs on the entered orders
then repeat the entire thing every second
This is a general question, I'm wondering is there any good framework/wrappers out there that one can learn from to code up a complex trading application?
Like dealing with all the asynchronous nature of process/submitting trading and quotes messages.
It is an overwhelming lonely endeavor. With all my other projects, I've always worked on teams, although they have always been very small teams and most of my work was autonomous -- still there was the occasional meeting and stand-ups. I've been working on this for six months and thought about bringing a friend onboard for no other reason than to not be alone.
We all know how to load up OHLCV data and do basic math on it. Where does that gain us any edge, you know?
I would be interested in some kind of regular meetup as well to discuss new techniques and to analyze publicly disclosed strategies, etc.
I recently posted this article on HN... There is so much more knowledge to share:
https://news.ycombinator.com/item?id=36344587
The only way I can think of is to get data that they can't.
However if you start increasing scale to $1mm or $10mm, your buy or sell orders begin to actually move the stock price itself. You might not be able to successfully sell $10mm of stock without dropping the price, signaling others to sell, further dropping the price, cutting into your own profits.
I.e., if you're buying, the larger your buy order is (well, assuming a visible buy order) the more likely it is that liquidity-adding sellers will increase the price of their sell orders. Also makes it less likely that liquidity-taking sellers will want to trade against your large buy order, because (like everyone else) they'll tend to interpret your large buy order as a sign that the price is likely to increase, so not as good a time to sell.
You could of course use hidden orders to avoid some of those disadvantages, but hidden orders have their own set of tradeoffs too.
(It's also kind of off-topic, my apologies for that. To me it seems semi-related but would agree with mods' assessment if it doesn't match mine.)
---
I recently "broke up" with some extremely toxic "investors" that wanted me to do a fully parallel trading demo -- meaning it receives ticks for N instruments (I successfully got to little less than 300) and trades with each of them depending on strategy. All in real-time.
I got very far but the open-source libraries for Interactive Brokers are quite low quality in general and it was very hard and slow to progress (one of them couldn't even post orders, another used Mutex-es for "parallelism" which was of course not parallel at all, another one seemed to work well but only worked on servers of older versions compared to those I have access to, etc). I also had to gather code from separate places and assemble my own Frankenstein as I went along.
Eventually I muscled through but by that time I have drained all my savings, my tax fund and even got a new loan. And the liaison + the investors of course refused to acknowledge the demo was basically 90% done (couldn't do full parallel trading due to defects of the IBKR libraries I have used and I used like 5 of them, and was in the process of repairing 2 of them to unlock the said full parallelism). They refused to send a small pre-funding wire (we're talking something small like $30k - $50k, not millions; for the work that was done, namely months of professional Rust programming work, that's a -75% discount if we look at market rates).
They had all the proof and paper trail they needed to see that I was very close but I had to stop because I was literally about to be unable to pay rent and bills. They still did not concede. Obviously I picked a job and dumped them but I still have some regrets because the door is technically still open (they have not cut my access to the IB Gateway servers that they own), but other factors like now-ruined health are seriously getting in the way as well. Not to mention completely shattered trust.
They insist "if you just finish the demo we'll give you money" and used every gaslighting technique I knew about (and many I didn't know about, so I learned a lot about gaslighting from them, lol) to try and coerce me to keep working for free with zero guarantees of funding -- but I no longer trust such rich investors to fulfill a promise without a binding and legally enforceable contract; I am in Eastern Europe, they are in the USA, even if they sign contract and violate it I practically cannot do anything to them i.e. I can't afford to travel and sue. Also they insist to get access to the source code but swear to everything that's holy that they will not run away with it and never give me a penny. Which is exactly what I think would happen.
I had to draw the line at one point. To me it was red flags all around.
---
I made many concessions and I will not make another one until they do. I'll be finding a new job soon again since the previous contract was agreed upon to be for several months, I helped a team accelerate bootstrapping a business-critical product (and we did that successfully). And then maybe, just maybe, after I settle a bit, I might work an hour or two on this again during some evenings. Maybe. And that won't be used to provide the demo to these toxic investors -- I'll use it to have a proven implementation that I can pitch to other people. These guys don't deserve the time of day from me.
I have quite a lot of good code (mostly Rust, but also some Elixir and Golang -- I experimented a lot) that interfaces with Interactive Brokers...
And yes I was tempted. I also don't have any trading credentials. I'm simply a senior programmer with an eye for details who always wanted to try his hand at algorithmic trading. So yeah, I got hooked. :(
IBKR is kind of an elitistic VIP club, not just anyone can gain access. Thus you won't find a lot of libraries. There's a good number of them but the quality is not great.
I always wanted to learn OCaml by the way but in my current life and career phase I still can't justify the time and energy expenditure, and I know it will be significant.
The last 18 months have been weaker than the first given the enormous structural shift in the market in this high inflation and rapidly rising interest rates environment, but we've still managed to deliver a return of +14.11% since the site launched in Jan 2022 compared to -7.83% for the SPX. We've managed to do it without any use of leverage and also with lower drawdowns as well of -16.48% vs. -27.57% for the SPX over that time frame.
It's also worth noting that every additional data source adds some risk of that data source being down or publishing inaccurate data during real-time signal calculation which can cause inaccurate signals, so in order to justify that risk, the external source must meaningfully contribute to alpha or better risk-adjusted returns.
None of them involve a human element in real-time. However, they are occasionally updated as new data comes in, but any updates only apply going forward so as to preserve the live trading history accurately (live trading start date varies by model from mid 2020 to jan 2022 with 2009 - 2020 being purely backtest for all models).
Am I right to doubt that something this simple generates any alpha whatsoever?
Kudos to you if you really sit on an untapped gold mine, but imho, there are some red flags that makes me not buy in.
However, the April 2009 start date is not actually random--it's the first start date for which intraday futures data is available for more than just front month contract. Several derivative indicators of the VIX futures curve are the most foundational to all the VIX-based models, and they simply cannot be processed without it. The VIX futures were only created in 2004, and I've scoured the internet for intraday data for more than just front month (can't create the curve if you only have front month data), and the earliest it can be found is April 2009.
I could use Visual Basic, and it would be better than Go, Rust, or whatever it is out there, given the algo and strategy are flawless. Language is just a tool. It's great you used Go, but I think the title is a bit misleading - people think of it as some kind of advantage. It isn't.
And for HFT trading a language with Garbage Collector is not a great choice IMO.
I used Go to write trading algos that would find small windows of triangle arbitrages in crypto exchanges. Made me some money but the risk of a big loss made me stop pursing crypto trade and it required too much time and attention. It's a full time job from my experience. Reasons I could lose big at any given time if I scaled up the stakes:
- Exchanges temporarily pausing some specific crypto trading for N reasons (happened very often) while I'm in the middle of the arbitrage.
- Getting caught in the middle of a pump and dump event (also frequent)
- Any algo mistake that would perform excessives trades in succession would incur huge losses because crypto exchanges charge %.
Also, most cryptos have too low volatility. Every time I tried to scale my bot would start interferring oo much with the market. And it wasn't even much money.
I used Go because that's what I knew and for the non-professional trading I was aiming, C++ wouldn't have made much difference. My bottlenecks were network (1s+ per trade roundtrip) and chaotic unreliable crypto markets.
Just note that Java is used in HTF, but it's a different beast than our average CRUD Java. For example this article states:
"Essentially, we use a contrived form of Java that avoids all the Java constructs that make things go slow. We only use the constructs that are fast and efficient, and we avoid all the garbage."
https://www.efinancialcareers.co.uk/news/2020/11/low-latency...
I read like 5 or so books about trading, mostly the classic ones, and tried to toy around with backtracking some self-made strategies using MetaTrader: https://www.metatrader4.com/en/trading-platform
First week was funny. I kept thinking I found goldmines with fine-tunned stochastic lagging indicators, only to realize that I overfitted and an algo that made me rich when tested against a certain period of time would make me go broke when applied to another period.
I tried applying some defense against local maximum, but to no avail.
Best I could do is tie after paying exchange fees. And that took me endless weekends toying around with code, reading books and articles.
My hunch is that algos that don't rely on external signals (i.e. news) need extreme technical edge like High Frequency Trading have with their ultra low latency and premium exchange data feeds.
And I wasn't ready to dive into news sentiment analysis and other external indicators.
Therefore the title was actually on point.
At the end it is necessary to make a decision whether to buy or sell (and how much), which will compete with other decisions made based on some logic. Developing such a logic (strategy) manually is of course quite difficult. I developed an intelligent trading bot which derives its trading strategy from historic data:
https://github.com/asavinov/intelligent-trading-bot
Currently it works for cryptocurrencies but can be applied to other markets:
https://t.me/intelligent_trading_signals
> I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different.
It is a typical situation. The whole problem is to develop a strategy which works for future (unseen) data. Even backtesting algorithms should be designed in such a way that future experience (data) does not leak to the past.
Depends. I've heard of places that use Java by creating a huge heap, minimising allocations to almost zero, and just restarting the app periodically before it can fill up. You can achieve very good performance doing this and you don't have to worry about memory bugs.
There are places that use Java and just preallocate all the memory they need at startup. Jane Street famously uses OCAML