I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one.
mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.
I get that there's a real issue there and it's interesting (to some) to address it, but it should have never surfaced this way in the first place.
Anyone running ripgrep on a an HPC cluster against a large cluster filesystem needs to stop and redesign their workflow. This generates high amounts of small I/O which is the Achilles heel of any large cluster filesystem. You are exporting your workload onto the metadata mechanisms of the filesystem rather keeping it within the higher bandwidth capable memory subsystem on your cluster. It doesn't take but a couple users running these types of jobs simultaneously to bring a high-bandwidth filesystem to its knees. Just stop it already.
Looking at the function the addr value should always equal end but also it looks like on x86 this value is discarded in pte_free_tlb so maybe this is not the problem or I’m looking at a different version.
Looks like the AI models either refused to work or produced obvious garbage. "GLM-5.2 is what finished the job for me, re-auditing K3's work and putting together an airtight case."
18 comments
[ 0.24 ms ] story [ 8.6 ms ] threadmallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.
I get that there's a real issue there and it's interesting (to some) to address it, but it should have never surfaced this way in the first place.
> I saw a fun bug report in ripgrep and a studious but pretty bad AI-generated analysis
Referring to https://github.com/dfoxfranke/ripgrep-3494-analysis which I indeed thought "that's an awful lot written to have been written by a human."
Looks like that thread is from...today!
https://github.com/torvalds/linux/blob/master/mm/memory.c#L1...
Assuming addr is a virtual address it probably makes sense that pte_free_tlb does not need it.
https://github.com/torvalds/linux/blob/master/arch/x86/mm/pg...
Looks like the AI models either refused to work or produced obvious garbage. "GLM-5.2 is what finished the job for me, re-auditing K3's work and putting together an airtight case."