TL;DR: Run-time dispatch in OpenBLAS chose code matching the CPU reported by the VM, thereby employing instructions that the real CPU doesn't have -> SIGILL.
I don't understand the "one day later my file system was broken" thing at the end though. Was this supposed to have any relation to the crash at all? It's way more mysterious than "SIGILL due to wrongly reported CPU type", but no elaboration is provided. What did the author even try to communicate there?
Since the author was dealing with a VM, my only guess is that for some reason the VM started receiving a different fake CPU? Maybe something like the virtualization software had changed? And that apparently caused file system mounting problems???
Author here. I don't know if the provider just changed the CPU type or if someone else ran into the same problem on that VM and complained to them.
The file system error is probably unrelated, I've seen a couple of those on other VMs from the same provider, but this time it probably caused it to fail to boot properly. More mysterious, but mostly out of my hands.
We can ask GDB to disassemble the current function in order to check the place where the code crashed and it will output a 4890-line monstruosity
That's because the "disassemble" command, contrary to what it says, is probably not what you want. On the other hand, "x/i $rip" will immediately show the exact instruction it's crashing on.
I don't understand what's so misterious about this. It's pretty obvious from the message that the problem is that program tried to run illegal instruction. Looking at opcode the instruction is part of AVX extensions.
I'm surprised we still encounter CPUs that don't have those these days. I'm pretty sure every CPU these days has AVX extensions (fact check this :) ).
By the way, title is somewhat click beauty. This is not a python crash. It's one of the shared libraries that were loaded by matplot lib that trigged the problem.
> I don't have the /proc/cpuinfo output any more (see below), but the model name was AMD Opteron 63xx class, or something similar. This is a virtual machine, so I'm assuming the hypervisor was reporting an older CPU model. [...] There actually was an Opteron 6300 series, which matches the reported model name and did support FMA4.
> [...] the CPU is a lie
The hypervisor is a lie.
I suspect the provider disabled the "fma4" instructions. If your program relies on model-specific CPU instructions, it can occasionally be a problem. I used to have a personal server at a big VPS provider. I compiled my program with "-march=native". One day, I found it suddenly started to crash. Upon further investigation, I noticed the "avx" flag had disappeared from my /proc/cpuinfo!
I filed a ticket demanding an explanation. The customer service replied and insisted that they do not just remove AVX, and recommended me to check my kernel. I upgraded my kernel, rebooted, and AVX mysteriously came back. I suspect that they actually did disable AVX to avoid CPU throttling (AVX is notorious for this, it can ironically degrade the overall server performance), and then restored it back after my complaint... Though I had no way to confirm.
If you use a small hosting provider, my suggestion is to never build programs with any CPU-specific instructions. Who knows what server they're moving to after a year?
Since the author no longer has the /proc/cpuinfo, I cannot say anything for sure. If the "fma4" flag was still there, it's a VM misconfiguration. If not, the software was only looking for the model number and not the flags, which is arguably a bug.
> Since the author no longer has the /proc/cpuinfo, I cannot say anything for sure. If the "fma4" flag was still there, it's a VM misconfiguration. If not, the software was only looking for the model number and not the flags, which is arguably a bug.
I touched on this in the article. I don't think Linux actually shows an fma4 CPU flag (it didn't on that VM), but a test program said it didn't support fma4. The OpenBLAS developers don't seem interested in fixing it [0], since it's unlikely to happen outside of misconfigured VMs.
My suspicion is that the server actually had an EPYC CPU, but the hypervisor was configured to say Opteron.
14 comments
[ 0.23 ms ] story [ 41.2 ms ] threadI don't understand the "one day later my file system was broken" thing at the end though. Was this supposed to have any relation to the crash at all? It's way more mysterious than "SIGILL due to wrongly reported CPU type", but no elaboration is provided. What did the author even try to communicate there?
guess that was like adding a random personal fact at the end. No relation with the article.
I agree it wasn’t clear at all.
The file system error is probably unrelated, I've seen a couple of those on other VMs from the same provider, but this time it probably caused it to fail to boot properly. More mysterious, but mostly out of my hands.
That's because the "disassemble" command, contrary to what it says, is probably not what you want. On the other hand, "x/i $rip" will immediately show the exact instruction it's crashing on.
I'm surprised we still encounter CPUs that don't have those these days. I'm pretty sure every CPU these days has AVX extensions (fact check this :) ).
By the way, title is somewhat click beauty. This is not a python crash. It's one of the shared libraries that were loaded by matplot lib that trigged the problem.
Op was obviously talking about x86 there, trying to run x86 on arm would fail long before reaching the avx instruction unless it was emulated.
Incidentally Apple’s rosetta (the x86_64-on-arm64 emulator) does not support AVX, if I remember correctly.
> [...] the CPU is a lie
The hypervisor is a lie.
I suspect the provider disabled the "fma4" instructions. If your program relies on model-specific CPU instructions, it can occasionally be a problem. I used to have a personal server at a big VPS provider. I compiled my program with "-march=native". One day, I found it suddenly started to crash. Upon further investigation, I noticed the "avx" flag had disappeared from my /proc/cpuinfo!
I filed a ticket demanding an explanation. The customer service replied and insisted that they do not just remove AVX, and recommended me to check my kernel. I upgraded my kernel, rebooted, and AVX mysteriously came back. I suspect that they actually did disable AVX to avoid CPU throttling (AVX is notorious for this, it can ironically degrade the overall server performance), and then restored it back after my complaint... Though I had no way to confirm.
If you use a small hosting provider, my suggestion is to never build programs with any CPU-specific instructions. Who knows what server they're moving to after a year?
Since the author no longer has the /proc/cpuinfo, I cannot say anything for sure. If the "fma4" flag was still there, it's a VM misconfiguration. If not, the software was only looking for the model number and not the flags, which is arguably a bug.
I touched on this in the article. I don't think Linux actually shows an fma4 CPU flag (it didn't on that VM), but a test program said it didn't support fma4. The OpenBLAS developers don't seem interested in fixing it [0], since it's unlikely to happen outside of misconfigured VMs.
My suspicion is that the server actually had an EPYC CPU, but the hypervisor was configured to say Opteron.
[0]: https://github.com/xianyi/OpenBLAS/issues/3638