- Record batching on client and broker side (works well with sequential IO)
- Batch compression (Json and other text formats could be compressed very well)
- Unflushed buffered writes
- Zero-copy approach for working with data
In addition to that the architecture of the Kafka also plays a major role so that the clients don't require much overhead. The code is written to not to put much pressure on GC and a number of other more specific optimizations was implemented on both client and server sides.
The most interesting part for me was how they achieve zero-copy when working with sockets. That whole section is worth reading and they could have made another blog post about only that.
> "For those unfamiliar with the term, ‘real-time’ does not mean ‘fast’, it means ‘predictable’."
So, if a system is predictable up to an upper bound of one year, is that still real-time? My understanding of real-time was pretty much as quickly as something happens with hard constraints on latency. I think "fast" is implied when we say real-time.
5 comments
[ 3.2 ms ] story [ 21.2 ms ] thread- Append-only log with sequential IO
- Record batching on client and broker side (works well with sequential IO)
- Batch compression (Json and other text formats could be compressed very well)
- Unflushed buffered writes
- Zero-copy approach for working with data
In addition to that the architecture of the Kafka also plays a major role so that the clients don't require much overhead. The code is written to not to put much pressure on GC and a number of other more specific optimizations was implemented on both client and server sides.
The most interesting part for me was how they achieve zero-copy when working with sockets. That whole section is worth reading and they could have made another blog post about only that.
Edit: formatting
So, if a system is predictable up to an upper bound of one year, is that still real-time? My understanding of real-time was pretty much as quickly as something happens with hard constraints on latency. I think "fast" is implied when we say real-time.