I'm wondering if compaction can be isolated as a separate processing task and it's own memory budget that separates it's processing from the memory pages utilized by request serving
Because compaction is about generating a new set of better optimized stables and then discarding the old
Cassandra is was full of all these tasks limped into the same problematic single jfm heap, even though the data could be partitioned into hundreds of jvms with smaller /easier to to gc heaps
Iirc scylla utilizes this fact with cpu pinning across cpus and other tricks that the older Cassandra could not.
yea that's why a lot of enterprises have moved to scylladb from cassandra since it is written in cpp and doesnt have the performance bottlenecks jvm brings.
Kernel 4.18 is _ancient_, weird to test on that. Also I wonder if the whole performance increase can't just be achieved by lowering the readahead size a bit. I see in their notes they have a decent SSD but set the blockdev readahead to 64. Depending on your workload it's much more performant to lower that to 16. As mentioned in TFA, fast storage is the most important factor here, and in my experience combining that with a smaller readahead pretty much fixes any read amplification issues you get with the larger readahead.
There are two performance problems presented here. One is the fact that the OS waits till a static memory bank water-level before triggering a cleanup.
The second is the one you pointed out; here again one thing to consider is cassandra's compression chunk size `chunk_length_in_kb`. readahead value less than chunk size makes cassandra slow in general. take a look at this https://thelastpickle.com/blog/2018/08/08/compression_perfor... for more info on it.
4 comments
[ 6.6 ms ] story [ 32.2 ms ] threadBecause compaction is about generating a new set of better optimized stables and then discarding the old
Cassandra is was full of all these tasks limped into the same problematic single jfm heap, even though the data could be partitioned into hundreds of jvms with smaller /easier to to gc heaps
Iirc scylla utilizes this fact with cpu pinning across cpus and other tricks that the older Cassandra could not.
There are two performance problems presented here. One is the fact that the OS waits till a static memory bank water-level before triggering a cleanup.
The second is the one you pointed out; here again one thing to consider is cassandra's compression chunk size `chunk_length_in_kb`. readahead value less than chunk size makes cassandra slow in general. take a look at this https://thelastpickle.com/blog/2018/08/08/compression_perfor... for more info on it.