Does this CPU time accounting have any noticeable performance impact?
I know in some embedded usecases, there are hundreds of thousands or millions of interrupts per second, and an interrupt routine might be only tens of clock cycles long. If that's the case, an extra bunch of instructions to time how long every single interrupt takes is a substantial overhead.
> Does this CPU time accounting have any noticeable performance impact?
Yes.
> Select this option to enable fine granularity task irq time accounting. This is done by reading a timestamp on each transitions between softirq and hardirq state, so there can be a small performance impact.
CONFIG_IRQ_TIME_ACCOUNTING:
Select this option to enable fine granularity task irq time
accounting. This is done by reading a timestamp on each
transitions between softirq and hardirq state, so there can be a
small *performance impact*.
If in doubt, say N here.
So I didn't enable this option in my kernels afraid of loosing performance on my laptop.
(Author here). It would be interesting to test the performance difference. I downloaded the source of one of my distro-provided kernels & recompiled it with different config setting (to compare perf with just one thing changed), but got into some trouble with my drivers and ran out of time.
I would say that on your laptop, you probably wouldn't even notice a difference (as the "rdtsc" timestamp reading operations are pretty efficient & done entirely within a CPU). But with anything that can interact with external hardware million(s) of times per second (heavy SSD I/O in servers and moving large amounts of data over network), may see a minor (but measurable) hit.
In my experience, when large amounts of CPU gets used by interrupt routines (either hardware or software interrupts), that more likely indicates some bug or configuration issue in a device driver or even some faulty hardware issue.
edit: so, running "dmesg" and checking hardware health metrics (mceloc, EDAC) is a reasonable thing to do when interrupt storms & related CPU usage suddenly show up.
The tool 'powertop' is very useful here, it will show what is causing any form of wakeup on your system, and offers a 'Tunables' tab with suggested settings to improve some aspects.
Powertop is great. You can even run it in non-interactive mode. That said, it only optimizes for lower power usage and lower interrupts. You can further reduce interrupts by disabling devices. If there are devices you don't need, you can blacklist them in /etc/modprobe.d/ provided you understand the potential dependencies other devices may have on the drivers. This can free up a tiny bit of memory as well.
Ahh, yes, that's actually a great point to be aware of, thanks @tanelpoder.
Looking at my machine: Funny to see 2h of interrupt processing over 12d uptime. Not that this is bad (0.80% on avg), but my first computer would have taken >6 weeks to process these interrupts - and not doing anything else. And that's only going by the clocks, a 80386 has a much worse IPC/pipeline than a 6th gen i7.
Irq accounting depends on architecture and code usage, e.g. like some kind of barrier (serializing) before the RDTSC.
on newer x86_64 the accounting code is around 100-200 cycles, so not a big deal, interrupt return and exit code is more than that (save context, load new context etc.).
The RDTSC and fence timings (Latency, Reciprocal throughput) [0]:
I guess the fence instructions are there for avoiding the CPU out-of-order execution getting the timestamp too early (or late) compared to the interrupt service routine's start & end?
Thank you, this is something I somewhat recently realized I don't know how to measure and troubleshoot properly - I've never been able to give a confident answer to if context switching/interrupts are a bottleneck or not, just "the number looks highish and I don't have anything else to blame".
17 comments
[ 2.7 ms ] story [ 51.6 ms ] threadI know in some embedded usecases, there are hundreds of thousands or millions of interrupts per second, and an interrupt routine might be only tens of clock cycles long. If that's the case, an extra bunch of instructions to time how long every single interrupt takes is a substantial overhead.
Yes.
> Select this option to enable fine granularity task irq time accounting. This is done by reading a timestamp on each transitions between softirq and hardirq state, so there can be a small performance impact.
[0] https://cateee.net/lkddb/web-lkddb/IRQ_TIME_ACCOUNTING.html
I would say that on your laptop, you probably wouldn't even notice a difference (as the "rdtsc" timestamp reading operations are pretty efficient & done entirely within a CPU). But with anything that can interact with external hardware million(s) of times per second (heavy SSD I/O in servers and moving large amounts of data over network), may see a minor (but measurable) hit.
In my experience, when large amounts of CPU gets used by interrupt routines (either hardware or software interrupts), that more likely indicates some bug or configuration issue in a device driver or even some faulty hardware issue.
edit: so, running "dmesg" and checking hardware health metrics (mceloc, EDAC) is a reasonable thing to do when interrupt storms & related CPU usage suddenly show up.
Looking at my machine: Funny to see 2h of interrupt processing over 12d uptime. Not that this is bad (0.80% on avg), but my first computer would have taken >6 weeks to process these interrupts - and not doing anything else. And that's only going by the clocks, a 80386 has a much worse IPC/pipeline than a 6th gen i7.
on newer x86_64 the accounting code is around 100-200 cycles, so not a big deal, interrupt return and exit code is more than that (save context, load new context etc.).
The RDTSC and fence timings (Latency, Reciprocal throughput) [0]:
[0] https://www.agner.org/optimize/instruction_tables.pdf