This. What is the point of using a higher level, garbage collected language instead of C++ or Rust if you're going to spend the same amount of time fighting against the VM or the Garbage collector to make it actually work the way you want it to? It feels like it's just for the sake of avoiding any kind of exposure to a newer language.
I can think of a couple of reasons; Java code is portable across OS'es; also while this code reads like C++, it might be part of a larger system with less performance-critical idiomatic Java.
I'm not sure that portability is a good reason to use Java in this case.
Firstly, these trading firms probably have tight control over their server architectures. Secondly, once you start using low-level Java idioms in order to optimize your code, you've started approaching the point where differences in performance between JVM implementations across platforms become noticeable.
Unlikely any time soon - most HFTs will have massive, battle-tested C++ code bases already. The data ownership model in HFT is usually very clear - receive packet, pass through some processing pipeline, emit packet, on the same thread. If you want to send data into another part of the system (eg. because different threads handle different securities), posting a message onto their event queue works well.
The GC isn't the feature you want though, it's the JIT compiler.
JIT compilers optimise based on the actual runtime characteristics of the running code, not the best guesses that a static compiler has to make. The effect can be significant. It's no surprise that the Java leaders in Techempower benchmarks match or surpass the C++ ones.
You can use profile-guided compilation with an AOT compiler, too-- Or you could just add optimization hints to the performance-critical parts of the code. Either way, the compiler doesn't need to "guess"
Also one of the advantages of JIT is that it can take advantage of whatever dedicated instruction is available on the current CPU, while an AOT compiler (discounting runtime dispatching) need to target the lowest common denominator.
In this case though you would know exactly the target hardware.
On the other hand, the JIT has to balance the runtime of its optimization passes with the expected improvement in execution speed of the code. An AOT compiler can afford much more expensive optimizations.
I've talked with HFT firms that spent effort fighting the JIT because they cared more about the latency of rarely reached code paths than the common path.
Java doesn't beat C++ in raw computational benchmarks. The JIT compiler is a marvel, but it doesn't beat peer AOT compilers.
The Techempower benchmarks mostly show how good the entrant's web-serving stack is. Java has a couple of decades of multiple skilled, well-established groups competing to build the fastest web stack. C++ has Boost.Beast, which, although its developers are smart, wise, widely respected, sexy, and moderators of a Slack i frequent, can't compete in terms of resources.
The point is development time. Even in time critical applications the majority of the code is not time critical, you only optimize the small part that really matters.
With Java vs C++, the experience seems similar. In C++, memory allocation is slow, so you worry about memory allocation up front. In Java, GC is a pain so you worry about memory allocation up front. As Daniel points out, there are a few extra tricks available to native developers (eg vector instructions), so there's a small factor speed-up in best-practice C++ vs best-practice Java, but this is closing (eg the JIT increasingly uses vector instructions). Overwhelmingly the difference between a slow system and a fast system is appropriate choices of algorithms and architecture, not the language.
>In C++, memory allocation is slow, so you worry about memory allocation up front. In Java, GC is a pain so you worry about memory allocation up front...
This.
A lot of people seem to be willfully dismissive of this reality when they are fanboying for their favorite languages.
Ask for memory in C++, you'll take a HUGE hit getting it. Ask for memory in Java, and you'll get it lightning fast, but you'll take a huge hit getting rid of it. Either way, you should carefully plan memory usage up front.
Basically, one road leads to death and despair, the other leads to disease and destruction...
True but these approaches in both C++ and Java start to converge: Arena allocation vs Pooling/unsafe. The question is, how annoying is your performance insensitive code?
Pareto's law. You optimize the few parts that matter speedwise, and the rest you can do in ordinary "higher level, garbage collected" fashion. In C++ you can't do that. This is the standard argument for Lisp type-optimization (DECLAIM, THE, etc.) as well.
Many people are saying that cpp benefits are cancelled by all the pointer/template voodoo required to get them.
Now I'd be happy to see high perf low latency talks about rust, go, ocaml, haskell, sbcl, whatever. I believe limit-cases like these are always very very instructive.
As a very long time Haskell user, I wonder how much latency management would be controlling for the lazy evaluation thunks being evaluated at certain deterministic times.
In my gut I would not choose Haskell for my first choice for a low latency language. I would definitely have looked at Rust before Haskell.
Haskell abstracts away order of operations. Both high performance and low latency benefit from the ability to be very specific about as much as possible and Haskell take the opposite approach. This may be good for data flow and throughput, but as soon as you need to control explicit order, state and resource usage Haskell is going to be working against you.
It seems to me that part of high-performance computing is optimizing memory layout (so you can vectorize, and to avoid cache misses). I have a hard time seeing that even in Java; I have a really hard time seeing that in Haskell. Am I missing something?
As is usual in Haskell for this kind of question, you can theoretically capture the information at the type level, and there are good odds somebody is working in a GHC extension for that.
But on practice, I don't think you are missing anything.
There are people with good results for high throughput Haskell (worse than C, not by much, with code almost as complex as C). But for latency, the GC alone kills it.
It sounds like D is in a good spot for these usages. It's fast, and you can nicely control the usage of GC - GC only gets called on allocations, and you can use @nogc attributes to statically confirm there are no allocations in the hot loop.
At least for Rust, the raw compiled code performance isn't quite there yet.
You can pick languages other than C++ or Java, but for a specialized and bespoke hardware/software combination like HFT, you'll be at a disadvantage vs other teams.
In my experience, or at least something I've seen at 3 different firms now, there's also an issue around the kind of people that pick exotic languages in terms of pragmatism. Often times the Scala/Haskell/whatever else side of the codebase is beautiful and slow moving while the hacky C++ side actually works and is agile.
Compared to the typical functional language applications, Rust is different in that it's mainly being battle tested in the kind of Firefox performance issues that are very interesting to an HFT programmer and Rust easily interfaces with C.
I'm not sure if it's still an issue since I don't use Rust daily; but one recommendation I'd make concerning array bounds checking is to do something like electric-fence and allow the compiler to be told to allocate arrays to end at the page boundary such that the hardware interrupts when you go off the edge.
If the compiler restricts such an array to only being indexed with an unsigned integer, then it can be safe but forego bounds checking altogether since the lower bound is always 0 and the upper bound check is hardware accelerated.
Yeah, a lot of those would be faster now, because SIMD was stabilized. That's the source of the discrepancy on n-body, for example. I don't think anyone has bothered to really submit new implementations.
Most bounds checks can be hoisted out of the compiler can’t figure it out on its own via asserts. In general it’s not a huge problem in real-world code.
The advantage over C++ isn't about the language itself. If your datastores all have functioning java connections, and your data modeling is already all done in java, it can be worth the pain to do performance critical code in java. I would rather do it in C++, but sometimes the most rational path forward is ugly.
Rust's future is far from certain. I would feel irresponsible using it on a project that wasn't self contained.
Have you watched the talk? The point is that it requires less effort overall, and if you need to go significantly beyond that, C++ can't save you anyway and you'll use FPGA.
Only theoretically. It is not possible to implement a GC as efficient as Java GCs for C++. Pointer arithmetic, placement new, and other low level features make it hard to determine pointers from non-pointers and to move things in memory.
"It is not possible" claims are very hard to substantiate.
Java is just someone's C++ program.
> "Pointer arithmetic, placement new, and other low level features make it hard to determine pointers from non-pointers and to move things in memory."
Real applications in a garbage collected language are mixtures of the managed code, plus unmanaged components (like foreign libraries).
Garbage collected C++ can work in much the same way: there is a framework for managed code that uses GC, and then there are unsafe parts that are analogous to foreign code. (Except, the managed code can much more easily and efficiently interact with this code than the typical FFI.)
There is more than one way to integrate garbage collection into C++, in any case.
100%. I've been going to their meetups for a few years now.
If there's anyone in London who is interested in Java, give it a go. The meetups have a variety of topics, covering many skill levels. You can go sit at the back and slink off if you want, or sometimes they have pizza/beer and you can chat afterwards.
That always amazed me. Why people keep torturing themselves applying Java to the usecase where Java is clearly not a right technology. I've watched very interesting talk by LMAX people and half of it was about how to overcome garbage collection gaps, latency, etc.
The most important factor is how well can your language be optimized. HFT is about winner takes all. If the rust optimizer is even slightly worse than your competitors language you will make nothing. Development speed might get your a faster algorithm for a few weeks but your competition will notice you making that money and will catch up despite the slower pace of development and then their faster language will make the difference and you lose all future trades.
llvm is not known as the best optimizer though. (but benchmarks tend to lie and llvm is always pretty close). There are also subtle areas where the front end can generate code that the backend cannot optimizer as well (though given equal effort I'd expect this advantage to go to newer languages that are design for modern optimizers - but effort is not equal with C++ getting for more love)
Yes, my point is mostly informative; we get benefits from using their optimizer, and all the time and resources others pour in, so it's not solely about what the Rust team does. Your point about front-end optimizations is true though; we generally try to copy what clang does, but there's stuff we can do better as well. And also implementing some of our own front-end optimizations.
I've researched this some a few months back. I don't blame you for saying that Java is wrong for HFT. My first reaction was the same, but there's much more to it.
Java has a few things going for it in HFT. The obvious pluses are it's mature and memory safe. What's less obvious is that you can make it low-latency. It takes a lot of work, but it's doable, at which point you have all the nice things: mature ecosystem, speed, latency, safety. It takes a lot of work, because Java was always oriented towards server use cases, as in high-throughput, not low-latency. That's changing by the way, there are two new GC engines coming out that are low-latency oriented. Also, there's been third-party JVMs with low-latency guarantees for quite a while.
Of course, what's between the lines is that there isn't any easy answers for HFT people. You either choose mature safety and do gc gymnastics (because everything is throughput oriented), or you choose manual memory management, which is its own gymnastics.
Anyway, that's my take. I welcome input and contradictions.
Mostly I agree, but there is one factor that makes C++ (or any native compiled language) better than java for HFT: the ability to lie to the optimizer about what the hot path is. in HFT you have thousands of no trades for every trade, so the java optimizer will optimize the no-trade code path as more likely, then when the trade happens java pays a CPU branch prediction miss penalty at the only time low latency matters.
You have to have good algorithms optimized to the max for this to matter though.
Unless you write HFT code, or follow talks by those writing HFT code you probably wouldn't. When you write HFT code you have to look at profiles and think about cache misses until branch missed become something to consider. If you don't come up with that idea someone else will and they will beat you to every trade and put you out of business.
I was with you until you mentioned branch prediction... isn't branch prediction a hardware feature? How do you trick the HW branch predictor into predicting the unlikely case?
Yes it is a hardware feature. However the hardware can be given hints as to which branch is more likely. This is generally documented by the manufacturer, in one of those technical documents aimed at compiler writers.
With profile guided optimization it is possible for the compiler to have much better information about branches than the CPU can guess. Java applies profile guided optimization in real time, with C++ it much more complex to apply.
> However the hardware can be given hints as to which branch is more likely.
I don't think that is the case for modern (last 8 years or so) Intel processors. For example, I'm under the impression that gcc's __builtin_expect only affects the layout of the generated code. However I'd love to learn something new here; do you have a source or any additional info you could share?
My understanding is the layout is the hint. Modern CPUs would not want to use cache space for any hints, not to mention all the silicon to decode the hint.
Compiler writers are smart people and can layout code to give the CPU the right hints, so long as they know what the CPU does. CPU manufactures want the compilers writers to do this as the compiler is a significant factor in making one CPU faster than the competition in benchmarks.
It seems to me that it's misleading to call that a hint. Rather, certain code paths create hard-to-predict branches, and others create predictable branches. A hint would be some kind of metadata that says "you'll want to predict this branch this way based on your algorithm, but please don't!"
To me, that section says that you can emit machine code that will allow the branch predictor to do a better job, not that you can control what the branch predictor does.
The hints are purely for the compiler. When branch probabilities are available (either via heuristics, annotations or profile data), it will optimize hot paths differently from cold paths. For example it might be more aggressive with inlinig or vice versa optimize for size. Also will attempt to put cold code in separate pages so that it doesn't get pollute the cache. Also non taken braches are marginally "faster" than taken so it is worth, when possible to put hot code in the non taken branch.
I'm not a compiler writer, I'm sure there is more.
I think you're discussing two different phases of optimization. PGO and Java's JIT use branch information to emit different machine code. Hardware branch prediction takes machine code and determines which branches in the machine code are taken, and speculates based on that information. There's an underlying pattern that both follow, but they're very different.
PGO and JIT both use their information to change the machine code to suggest which branch is more likely.
PGO and JIT also do a lot of other things that are unrelated to this discussion. Some of those things can have a much larger gain than branch prediction.
The cpu still needs to load code in via instruction cacheline fetches. For every instruction fetch, that core isn't doing much.
The compiler alleviates this somewhat by putting the hot path right under the branch instruction so that the fetch that grabs the branch also grabs the start of the hot path as part of the same cacheline.
It sounds minimal, but if that fetch is swapped out of L2 cache due to long periods of inactivity, it can take upwards of 100ns, which starts to add up in HFT.
So, if I want to manipulate the market I need to work out (or poison) the hot path and have a system that's tailored to being faster on profitable - if colder - paths?
If you have a low-latency trading component written in Java, a common trick is to continuously bombard it with ‘fake’ inputs to keep the desired code paths nice and hot.
The fake inputs should be virtually indistinguishable from real ones that you would normally act on. The more subtle the difference, the better, e.g., just flip the sign on the timestamp field.
You can use that subtle difference to pick whether the order goes out to the real exchange or a fake exchange. The decision needs to avoid actual branching instructions, though, or the JVM will likely optimize out the ‘real’ hot path, and you’ll fall back to interpreted mode when an actionable ‘real’ event comes in. I usually use a branchless selector to index into an array ([0] goes to a real socket, [1] goes to a black hole socket).
You can also use this technique to make sure you can respond to very rare events quickly. For example, you may want to respond to news signals from Bloomberg. Actionable news is rare, so if you want to keep your news parsing/analysis code warmed up and in the cache, it needs to constantly be reacting to warmup data.
Thankfully, no (knocks on wood). But you have to base your design around the idea that real and fake trades are indistinguishable until the last possible moment. That critical requirement needs to always be in your mind.
I would never try to bolt those kinds of optimizations onto an existing system. It’d be too easy to miss something.
I have been out of the HFT space for many years now, but your conclusions make sense. Here is my own take on the subject from my short tenure at an HFT firm: https://news.ycombinator.com/item?id=12053159
One important thing to know is that allocating memory in C or C++ has high and/or unpredictable latency relative to the target latency of HFT code. So for critical paths, you end up needing to do the same kind of pre-allocation tricks in C/C++ that you would need to do in Java. There is some benefit to being able to write more natural code in the non-critical paths, but that has to be weighed against the overall development advantages that lead people to choose Java over C/C++ in other industries.
A very good point. Most of my GC whispering is done in .NET, which also has value types, but I was guessing that escape analysis in Java would solve most of the problems in this regard. Is that not actually the case?
In trading, correctness is still a lot more important than latency. Using Java instead is about risk avoidance, which is why you also see people using OCaml.
The reason Java has been so successful is that avoiding memory errors is far more important than being fast. Once you have removed the possibility of those errors, being fast is still valuable, which is how we end up with people spending so much time optimizing their Java instead of switching to C.
It's more likely that people will switch to OCaml or Rust than ever go back to unmanaged memory. First secure, then correct, then fast.
In case you haven't watched the talk, people don't spend so much time optimizing Java; on the contrary, Java is often faster than C/C++ without optimization effort. It does have a lower ceiling, but as he says, if you need to be faster than Java (for the uses he talks about), then you're better off going FPGA than another language, because that's the only way to gain a very significant boost.
Huh? People don't have to put in that effort because there are companies who have invested crazy amounts of man hours and time and money into engineering the JVM to be what it is today. Java is only as fast as it is because there are people who've put in that effort. Java's reputation for being slow isn't unwarranted - it just isn't true anymore.
I think what s/he meant is that app developers don’t have to spend time on optimization (because - as you point out - so many smart people have spent so much time making the JVM as fast as possible).
Most developers of Java I've encountered don't know the first thing about optimization and manage to make some really suboptimal stuff despite the supposed ease of achieving near-best performance.
I spent a fair amount of time actually benchmarking and iterating on code to make java programs perform better in college, because thankfully they considered that a useful skill. What I see in practice in industry is architectural patterns that supposedly support maintainability at a great cost to performance, with the concrete parts being written by people who don't have any idea what the performance characteristics of the code they're writing are and why.
I think Java has a lot of merit, but it also has an "ecosystem" rife with bloated, dogmatic, hyper-abstracted code that was never really given a thought about optimality and requires purchasing big servers with lots of RAM.
As good at it is, and yes, it was and is good in the grand scheme; I'm disappointed that Java still has so much inertia, and that development and adoption of alternatives is so slow going. I feel as if there is a "it's good enough, why change anything" attitude that stems from lack of understanding that there are still big juicy low hanging fruit to be found and incorporated into our tools, to ultimately make software better AND easier to write/maintain. The line about "lots of smart people have been optimizing the JVM for a long time" seems like a go-to defense of what I see as stagnation.
Java has about 10M developers, who do everything from developing banking systems, GMail, Netflix, to people writing avionics, air-trafic and weapons control, and manufacturing control. Even 1% of Java developers is more than many language's entire ecosystems.
> I think Java has a lot of merit, but it also has an "ecosystem" rife with bloated, dogmatic, hyper-abstracted code that was never really given a thought about optimality and requires purchasing big servers with lots of RAM.
Again, it's really hard to generalize with such a diverse ecosystem. 1% of Java developers is still ~100K developers.
> that stems from lack of understanding that there are still big juicy low hanging fruit to be found and incorporated into our tools
I'm not sure there are, but if so -- go ahead and prove it.
> The line about "lots of smart people have been optimizing the JVM for a long time" seems like a go-to defense of what I see as stagnation.
Well, as one of the many people working on OpenJDK, I won't comment on that :)
Getting good performance out of Java is not always easy. One often has to structure code imperfectly to get efficient branching (Java offers only vtable dispatch or switch on integers/enums). It also boxes nearly all values leading to cache inefficient pointer chasing. These problems can be avoided, but you'll be working against the language.
I am "getting these things" from over a decade of working with Java. There are many good things about Java, especially when compared to C++. But the bad things will never get fixed if we pretend everything is perfect.
I did say vtable dispatch was efficient, but I might not want to split my logic across classes. A chain of if-else is a linear scan and not efficient. Switch in its current form is too limited.
It is well known that Java has no value types (there is a JSR). Even for a primitive type, the collection libraries need to box all values in order to abstract the element type. Every HFT user on Wall Street has had to write their own collections library using custom code generation to work around this.
>Java offers only vtable dispatch or switch on integers/enums
The JVM has complete knowledge of the class hierarchy. If a method has not been replaced in a subclass then no vtable entry has to be generated and the JVM just uses a static call.
My point was that often one has to structure code across classes to get efficient branching. Any details regarding optimisation for virtual dispatch doesn't change this fact.
> It's more likely that people will switch to OCaml or Rust than ever go back to unmanaged memory. First secure, then correct, then fast.
Interestingly enough, that's a philosophy that's valuable and applicable in other skill sets as well. Woodworking/machining comes to mind and other trades.
Safety/security first, then precision, then speed once you got the other 2 working.
Use memory mapped files to "talk" between processes.
Use flyweights for the messages. Avoid strings in the messages wherever possible.
Use single-threaded processes wherever possible.
Isolate cpus.
Pin to isolated cpu.
Turn off hyperthreading.
Use some form of object pooling.
The last point is the only Java/GC language specific thing.
Having used Chronicle I can confirm that you can see single digit microsecond latency (or better), and reduce 99th percentile jitter with significant analysis.
HPC software like MPI has been using these kind of tricks for years to achieve sub-microsecond (often as small as 0.2-0.3 us) latency for inter-process communication.
117 comments
[ 2.6 ms ] story [ 161 ms ] threadFirstly, these trading firms probably have tight control over their server architectures. Secondly, once you start using low-level Java idioms in order to optimize your code, you've started approaching the point where differences in performance between JVM implementations across platforms become noticeable.
Rewriting some hot path parts of the stack in Verilog is a usual thing, and rewriting some other parts in Coq is the future.
https://www.janestreet.com/tech-talks/ocaml-all-the-way-down...
https://deepspec.org/entry/Project/Kami http://conal.net/blog/posts/haskell-to-hardware-via-cccs
Whereas 90% of the time might be spent in 10% of the code image, no 10% chunk of the image can ever be found that contributes to 90% of its bloat.
1% of the image contributes 1% to the bloat, 10% to 10% and so on.
:)
JIT compilers optimise based on the actual runtime characteristics of the running code, not the best guesses that a static compiler has to make. The effect can be significant. It's no surprise that the Java leaders in Techempower benchmarks match or surpass the C++ ones.
In this case though you would know exactly the target hardware.
The Techempower benchmarks mostly show how good the entrant's web-serving stack is. Java has a couple of decades of multiple skilled, well-established groups competing to build the fastest web stack. C++ has Boost.Beast, which, although its developers are smart, wise, widely respected, sexy, and moderators of a Slack i frequent, can't compete in terms of resources.
There are a few AOT-compiling JVMs out there, and it's not the case that they reliably outperform the JIT-based JVMs like HotSpot.
The significant differences are in Java mandating array bounds checks, runtime checking for null, no undefined behaviour, etc.
This.
A lot of people seem to be willfully dismissive of this reality when they are fanboying for their favorite languages.
Ask for memory in C++, you'll take a HUGE hit getting it. Ask for memory in Java, and you'll get it lightning fast, but you'll take a huge hit getting rid of it. Either way, you should carefully plan memory usage up front.
Basically, one road leads to death and despair, the other leads to disease and destruction...
pray you choose wisely.
While it is hard to beat existing general purpose allocators in all the scenarios, it is easy for specific use cases.
What?
Now I'd be happy to see high perf low latency talks about rust, go, ocaml, haskell, sbcl, whatever. I believe limit-cases like these are always very very instructive.
In my gut I would not choose Haskell for my first choice for a low latency language. I would definitely have looked at Rust before Haskell.
I always love optim talks whatever the language (granted its not a clusterfk interpreter).
But on practice, I don't think you are missing anything.
You can pick languages other than C++ or Java, but for a specialized and bespoke hardware/software combination like HFT, you'll be at a disadvantage vs other teams.
In my experience, or at least something I've seen at 3 different firms now, there's also an issue around the kind of people that pick exotic languages in terms of pragmatism. Often times the Scala/Haskell/whatever else side of the codebase is beautiful and slow moving while the hacky C++ side actually works and is agile.
Compared to the typical functional language applications, Rust is different in that it's mainly being battle tested in the kind of Firefox performance issues that are very interesting to an HFT programmer and Rust easily interfaces with C.
I'm not sure if it's still an issue since I don't use Rust daily; but one recommendation I'd make concerning array bounds checking is to do something like electric-fence and allow the compiler to be told to allocate arrays to end at the page boundary such that the hardware interrupts when you go off the edge.
If the compiler restricts such an array to only being indexed with an unsigned integer, then it can be safe but forego bounds checking altogether since the lower bound is always 0 and the upper bound check is hardware accelerated.
Yeah, a lot of those would be faster now, because SIMD was stabilized. That's the source of the discrepancy on n-body, for example. I don't think anyone has bothered to really submit new implementations.
Most bounds checks can be hoisted out of the compiler can’t figure it out on its own via asserts. In general it’s not a huge problem in real-world code.
You made that claim before, and I showed you the Rust n-body program "vector instructions using SIMD" contributed 5 months ago.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Edit: the correct URL is https://benchmarksgame-team.pages.debian.net/benchmarksgame/....
It shows 0.48 seconds, and 2 seconds, but https://benchmarksgame-team.pages.debian.net/benchmarksgame/... shows 13.27. Why isn't that one on the main page?
HN have now corrected the URL, as I asked.
> It shows 0.48 seconds, and 2 seconds
It shows 3 different workloads:
> … but … shows 13.27…Which we can see is the time for `n-body #2` at the largest workload:
> Why isn't that one on the main page?I don't know which you consider to be the "main page".
We can see that measurements for `n-body #2` are shown on both:
andRust's future is far from certain. I would feel irresponsible using it on a project that wasn't self contained.
http://www.stroustrup.com/bs_faq.html#garbage-collection
Java is just someone's C++ program.
> "Pointer arithmetic, placement new, and other low level features make it hard to determine pointers from non-pointers and to move things in memory."
Real applications in a garbage collected language are mixtures of the managed code, plus unmanaged components (like foreign libraries).
Garbage collected C++ can work in much the same way: there is a framework for managed code that uses GC, and then there are unsafe parts that are analogous to foreign code. (Except, the managed code can much more easily and efficiently interact with this code than the typical FFI.)
There is more than one way to integrate garbage collection into C++, in any case.
Not all garbage-collection schemes move things in memory; only copying and/or compacting garbage collectors do.
https://youtu.be/BD9cRbxWQx8?t=1025
If there's anyone in London who is interested in Java, give it a go. The meetups have a variety of topics, covering many skill levels. You can go sit at the back and slink off if you want, or sometimes they have pizza/beer and you can chat afterwards.
Java has a few things going for it in HFT. The obvious pluses are it's mature and memory safe. What's less obvious is that you can make it low-latency. It takes a lot of work, but it's doable, at which point you have all the nice things: mature ecosystem, speed, latency, safety. It takes a lot of work, because Java was always oriented towards server use cases, as in high-throughput, not low-latency. That's changing by the way, there are two new GC engines coming out that are low-latency oriented. Also, there's been third-party JVMs with low-latency guarantees for quite a while.
Of course, what's between the lines is that there isn't any easy answers for HFT people. You either choose mature safety and do gc gymnastics (because everything is throughput oriented), or you choose manual memory management, which is its own gymnastics.
Anyway, that's my take. I welcome input and contradictions.
You have to have good algorithms optimized to the max for this to matter though.
With profile guided optimization it is possible for the compiler to have much better information about branches than the CPU can guess. Java applies profile guided optimization in real time, with C++ it much more complex to apply.
I don't think that is the case for modern (last 8 years or so) Intel processors. For example, I'm under the impression that gcc's __builtin_expect only affects the layout of the generated code. However I'd love to learn something new here; do you have a source or any additional info you could share?
Compiler writers are smart people and can layout code to give the CPU the right hints, so long as they know what the CPU does. CPU manufactures want the compilers writers to do this as the compiler is a significant factor in making one CPU faster than the competition in benchmarks.
The hints are purely for the compiler. When branch probabilities are available (either via heuristics, annotations or profile data), it will optimize hot paths differently from cold paths. For example it might be more aggressive with inlinig or vice versa optimize for size. Also will attempt to put cold code in separate pages so that it doesn't get pollute the cache. Also non taken braches are marginally "faster" than taken so it is worth, when possible to put hot code in the non taken branch.
I'm not a compiler writer, I'm sure there is more.
PGO and JIT also do a lot of other things that are unrelated to this discussion. Some of those things can have a much larger gain than branch prediction.
The compiler alleviates this somewhat by putting the hot path right under the branch instruction so that the fetch that grabs the branch also grabs the start of the hot path as part of the same cacheline.
It sounds minimal, but if that fetch is swapped out of L2 cache due to long periods of inactivity, it can take upwards of 100ns, which starts to add up in HFT.
If you have a low-latency trading component written in Java, a common trick is to continuously bombard it with ‘fake’ inputs to keep the desired code paths nice and hot.
The fake inputs should be virtually indistinguishable from real ones that you would normally act on. The more subtle the difference, the better, e.g., just flip the sign on the timestamp field.
You can use that subtle difference to pick whether the order goes out to the real exchange or a fake exchange. The decision needs to avoid actual branching instructions, though, or the JVM will likely optimize out the ‘real’ hot path, and you’ll fall back to interpreted mode when an actionable ‘real’ event comes in. I usually use a branchless selector to index into an array ([0] goes to a real socket, [1] goes to a black hole socket).
You can also use this technique to make sure you can respond to very rare events quickly. For example, you may want to respond to news signals from Bloomberg. Actionable news is rare, so if you want to keep your news parsing/analysis code warmed up and in the cache, it needs to constantly be reacting to warmup data.
I would never try to bolt those kinds of optimizations onto an existing system. It’d be too easy to miss something.
When Java gets value objects, this sort of work will begin to get a lot easier in Java as well, but there will be a lot of catching-up to do.
It's more likely that people will switch to OCaml or Rust than ever go back to unmanaged memory. First secure, then correct, then fast.
I spent a fair amount of time actually benchmarking and iterating on code to make java programs perform better in college, because thankfully they considered that a useful skill. What I see in practice in industry is architectural patterns that supposedly support maintainability at a great cost to performance, with the concrete parts being written by people who don't have any idea what the performance characteristics of the code they're writing are and why.
I think Java has a lot of merit, but it also has an "ecosystem" rife with bloated, dogmatic, hyper-abstracted code that was never really given a thought about optimality and requires purchasing big servers with lots of RAM.
As good at it is, and yes, it was and is good in the grand scheme; I'm disappointed that Java still has so much inertia, and that development and adoption of alternatives is so slow going. I feel as if there is a "it's good enough, why change anything" attitude that stems from lack of understanding that there are still big juicy low hanging fruit to be found and incorporated into our tools, to ultimately make software better AND easier to write/maintain. The line about "lots of smart people have been optimizing the JVM for a long time" seems like a go-to defense of what I see as stagnation.
Java has about 10M developers, who do everything from developing banking systems, GMail, Netflix, to people writing avionics, air-trafic and weapons control, and manufacturing control. Even 1% of Java developers is more than many language's entire ecosystems.
> I think Java has a lot of merit, but it also has an "ecosystem" rife with bloated, dogmatic, hyper-abstracted code that was never really given a thought about optimality and requires purchasing big servers with lots of RAM.
Again, it's really hard to generalize with such a diverse ecosystem. 1% of Java developers is still ~100K developers.
> that stems from lack of understanding that there are still big juicy low hanging fruit to be found and incorporated into our tools
I'm not sure there are, but if so -- go ahead and prove it.
> The line about "lots of smart people have been optimizing the JVM for a long time" seems like a go-to defense of what I see as stagnation.
Well, as one of the many people working on OpenJDK, I won't comment on that :)
I did say vtable dispatch was efficient, but I might not want to split my logic across classes. A chain of if-else is a linear scan and not efficient. Switch in its current form is too limited.
It is well known that Java has no value types (there is a JSR). Even for a primitive type, the collection libraries need to box all values in order to abstract the element type. Every HFT user on Wall Street has had to write their own collections library using custom code generation to work around this.
Strings in switch Statements:
https://docs.oracle.com/javase/7/docs/technotes/guides/langu...
Supporting a switch on static final references would be a good first step.
The JVM has complete knowledge of the class hierarchy. If a method has not been replaced in a subclass then no vtable entry has to be generated and the JVM just uses a static call.
Interestingly enough, that's a philosophy that's valuable and applicable in other skill sets as well. Woodworking/machining comes to mind and other trades.
Safety/security first, then precision, then speed once you got the other 2 working.
Use memory mapped files to "talk" between processes. Use flyweights for the messages. Avoid strings in the messages wherever possible. Use single-threaded processes wherever possible. Isolate cpus. Pin to isolated cpu. Turn off hyperthreading. Use some form of object pooling.
The last point is the only Java/GC language specific thing.
Having used Chronicle I can confirm that you can see single digit microsecond latency (or better), and reduce 99th percentile jitter with significant analysis.