MenuMeters on Mac solves this problem for me: you get a customizable set of little system load graphs in the menubar which can be glanced at to notice anything off.
Heh, we need an external device, maybe something like a USB LED screen that shows CPU/disk/io usage outside of the regular desktop, much like we see on some fancy CPU coolers these days. At least for me I don't tend to keep things like usage utilities on screen and pinned to the top as they'll block the other things I'm doing on even multiple monitors.
If someone was clever about it, you could probably even feed data back to the external display via SSH from remote boxes.
Yup that's amazing... At some point in the past I considered building my PC inside an old amp but the amp only had two vumeters and nowadays CPUs have many cores. GP's YouTube link looks really great.
Long-term istat menus user here too, for the same reasons. I've found and removed so much crap through the years because I could notice it. I've fixed so many dumb issues because I could see it was doing absurd amounts of pointless, unexpected work.
And for istat menus specifically: it's super stable and they're extremely fast to support new OSes, unlike many fancy menubar apps. Easily worth the money for me.
Unfortunately, these performance monitors can make the computer use more power, by waking up every second to update the display, and therefore preventing the computer from entering deeper idle states. It's the same reason the default on-screen clocks no longer shows seconds by default, and blinking text cursors often blink for only five seconds or so before becoming static: waking up every second to update the screen can be costly.
Of course, a performance monitor which updates less often can avoid that issue (for instance, atop by default updates every ten seconds), but most people who run these graphical performance monitors want second-by-second updates.
Developers are being asked to use them. It doesn’t matter if developer machines use a bit more power to save magnitudes more over the lifetime of all client use of the resulting, more efficient, product.
Back in 2009, Matthew Garrett measured a blinking cursor as costing 2W of power consumption. Presumably it'd be less of a big deal now, but it does show that UI updates on a low frequency can cost watts.
I have used iStat Menus for a couple of hours and it made a sizable dent on my battery life, so I uninstalled it right away. macOS is a bit sensitive about being woken up constantly. Maybe it won’t be a problem on a desktop machine, but running these on Mac laptops is not free.
Note that iStat Menus have its own set of daemons it uses for keeping tabs on system usage. It doesn’t just hook to the kernel via interfaces and display stats.
Can we please stop obsessively monitoring (and $deity forbid, alerting on) low level metrics such as CPU and Disk I/O? Sure, they should be recorded as they are useful for troubleshooting and looking for bottlenecks. But relying on them for most software is trying to figure out if you should get a speeding ticket by looking at the tachometer.
CPU usage being high doesn't really tell you anything. You may suspect that there's an issue if you are familiar with the system, and you have only one system to worry about. It's definitely not OK if you have a 'modern' architecture and a fleet of machines running a distributed system. There's a whole lot of noise and you can't tell if traffic has increased, if someone pushed inefficient code, if there's a cooling problem and CPU is throttled, if a "batch" process started, or a million of reasons.
And even if it is high, and it is an anomaly(anomaly detection on those metrics can be useful at times), is it causing any issues for your customers? CPU usage can't tell you that.
Every piece of software has some work to do. What is that work? Figure that out and monitor that. Golden signals on the relevant metrics and you'll be in a far better position. Did the error rate increase? Are we getting higher response times? In that case it doesn't matter what the CPU metric says – it could even be lower than usual if there's a bottleneck somewhere else – it is a problem that needs to be addressed. You can then use the other metrics to confirm, or try to isolate the problem. But they should be supporting information, not your main diagnostics tool.
> In a project I worked on, our development builds started writing about 80 MB of log messages per second to disk.
Well, there you go. If you just look at disk I/O you'll see that it is high. But if you track metrics from your logging system, you'll be able to immediately see _why_
By all means monitor those metrics on your work laptop - very few people are going to bother running prometheus in their dev machine and setting up alerts for third party apps. But for the apps you are responsible for? Figure out, from day zero, what metrics you should collect for your app and monitor those instead.
For years, noisy hard drives with blinking lights served to highlight the difference between thrashy Microsoft operating systems and smoothly running, performant alternatives.
Most of laptops still have fans these days. They just spin less because the machine is much more powerful. Maybe reading 100MB/s is not as concerning as 30 years ago? My M1 macbook still gives me fan noise occasionally, like when I forget to exit an infinite loop. But I appreciate that most of the background tasks would not need my attentions anymore.
Back in the days I didn't enjoy the LED or the sound of disks. They disrupted my workflow and not always actionable. I mean, I don't really need to know that the machine is working hard on reading the optical discs I just inserted. I will take action if the reading slows down my foreground program.
Ha, back in 1995 I could HEAR in advance when my PC was going to crash. I could tell from the certain crunching / grinding sounds of the hard disk. I could hear it and think "Uh oh, here comes a crash..." And then blue screen of death.
Back when I had a PC which was somewhat capable of playing games I would play Left 4 Dead in cooperative mode. You play along with three other players, and navigate the levels attempting to move from one safe room to another, fighting a hoarde of infected. The game attempts to keep the players moving by adding "special" infected, which have different abilities and are good at forcing the team to leave the current area.
One of the special infected was the Tank, which is quite strong and can throw chunks of pavement. I always knew when he was coming because my PC was kind of weak and the fans would go nuts about 30 seconds before he made an entrance. The other players were surprised that I had a sixth sense when it came to knowing when it was happening.
Too much HDD noise was also a really good indication that your computer ran out of memory and stuff is being moved into swap. I remember some occasions where I would alt-tab from a game in Windows 98 and my computer would freeze for half a minute (while doing concerning HDD noises) until I regained control of the desktop.
On the Commodore 64, you knew your floppy was bad when the drive would reset itself, slamming the read/write head against the stop repeatedly. “tick-tick-tick-tick-BRRRAAAAAAP” ... eventually knocking the head out of alignment, requiring a hardware fix (not a difficult fix, but tedious). Copy protection was notorious for causing drive knocking, so people often used cracked versions of games they purchased just to prevent it (they tended to load much faster too).
The 1541 doesn't have a track 0 sensor, so they send it 40 step commands to move it back to 0. If it wasn't already on track 40, it will just bump against the stop repeatedly. I've never heard of it damaging the drive though.
My own 1541 never actually had problems, but it was folklore on BBS's, and I saw it happen with my brother's drive. Was easy enough to service though, just loosen a couple screws, nudge the motor a teensy bit, run a diagnostic disk, and repeat ... many many times. They might have just been prone to misalignment anyway, but the knocking couldn't possibly have helped.
For working with remote machines that I need to ssh into I've found mobaXTerm[1] to be a very useful terminal emulator. It has an optional remote monitoring feature that shows the usual stats as a small bar under the active terminal window.
When I used to use macOS on Intel laptops I would be hit, every so often, by gpg’s smartcard pinentry getting stuck in a 100% loop. The only thing that ever notified me of it was the fan turning on!
Yup! And if it is beefy enough, the fans will kick in from time to time. (Strangely, it gives me great joy to see 30 cores under 100% load and hear the fans hum in accord.)
Most consumer SATA drives actually report their activity on pin 11 of power connector, which can be easily hooked to individual LEDs (as seen on drive bays in server racks). SATA standard even has schematic diagrams for proper making of those blinkenlights. Unfortunately, most (probably all) factory-made wired connector plugs are made with a single piece of metal forming pins 10, 11 and 12 cramped onto the ground wire, and soldering or other contact isolation work is required. Soldering male and female connectors with full number of pins together to add a breakout is probably the easiest way.
Totally agree with this article, there are no immediate consequences to bad CPU bound code. I am seeing the consequences of this at my current position as a Senior Engineer at a startup where a lot of things were written naively, pushed the burden to cloud costs to keep pushing out features.
Something that really taught me to look for things like the "HDD Light" or Fan speed was starting my career in embedded systems. 16 bit MCUs really let you know if you are trying to much on them. They also let you know if you toggle the wrong pin by going up in flames.
The disconnect between your fingers and what actually runs the code is becoming greater and greater for newer developers. It will be interesting to see how computing power keeps up with bad code (it had been doing a good job so far).
Programmers don't (can't?) dog food[1] their code because their development machines are Intel Xeon or AMD Threadripper monstrosities with abominable GPUs to match.
Most apparent are the web programmers, most of them assume everyone has 10gbit fiber connections with 16 core CPUs and 128GB of RAM to feed to Chrome. And then they wonder why their shit runs like shit in the real world.
> their development machines are Intel Xeon or AMD Threadripper monstrosities with abominable GPUs to match.
Not all developers. In my entire career, I've never had a development system that was better than the average consumer machine. And, honestly, I wouldn't want one that was, for the exact reasons you state.
Are you certain about that? I’d wager that the average (median?) consumer machine these days is a a smartphone, and not an especially high-end one at that.
Yes, I'm sure. I'm not counting smartphones in this at all, though, because I don't develop apps for smartphones anymore (it's not a market that interests me). By "average machine", I meant a budget laptop or tower.
I've never been terribly bothered by compile times in general. 90% of my work is with compiled languages, and (in most languages) incremental compilation does a great job of keeping compile times low. Even for large projects on slow machines, I rarely see compile times exceed even a minute. More usually, compiling just takes a few seconds.
Sure, you do occasionally need to do a full recompile, and if the project is very large then this can take a nontrivial amount of time, but that doesn't really happen often enough for me to be bothered about.
If the project is of really significant size, or if the compiler/language/code structure is such that incremental compilation can't give you significant gains (or isn't even possible), you may need more grunt of course.
> Most apparent are the web programmers, most of them assume everyone has 10gbit fiber connections with 16 core CPUs and 128GB of RAM to feed to Chrome
Facebook marketplace. I don't know how you can make a grid of images max out a 5800x, but they've managed it. A markedly inferior product to every one of the classified ad competitors they've squashed out of existence.
That is one advantage of being a firmware engineer. In my case, this code runs on exactly one machine (usually) so whatever performance I get is very indicative of what the customer experience is like. My dev machines are usually just unfused versions of whatever hardware I'm working on.
Programmers developing webshits should really be forced to add 200ms to all their roundtrips. It’s worrying how many people assume everyone is 20ms from their “edge”, and openly advocate on HN for “new” technology that requires a roundtrip on every single state change, unwinding twenty years of client side advancements.
As someome who doesn't live in the US I appreciate the nod. Unless the company has edges in the southern hemisphere, every single widget has a round trip of 2-300ms. Something like Jira with it's cascading react components was a jumbling absurd mess until all the roundtrips completed, which could take 10+ seconds. A liftime from a UX perspective.
Maybe its because I work on embedded firmware, where detailed logging is occasionally at the expense of device function, but I find the absence of blinking LEDs really unsettling.
I've repeatedly removed and replaced the batteries on my thermostat's remote temperature sensor, because there's no visual feedback that its otherwise working, and my house was cold.
The absence of "working" lights when I know I'm doing a processor-intensive operation on a computer is similarly unsettling. Its not directly actionable feedback, certainly, but it can be a proxy for a more directly actionable, but more onerous to implement or use, feedback mechanism.
At my previous job all circuit board first revision review meetings I would never shut up about how I wanted as many LEDs on GPIOs as I could get so I could stick various Linux LED triggers on them. CPU, disk access, heartbeat, network, panic, etc. SUPER helpful for early debugging of “what the heck is going on now?” situations that are common on early prototypes.
There are plenty of ease to use libraries and tools to instrument and profile your code today.
If people don't use them, if they don't care about the performance, it is on them, not in the lack of noisy spinning disks and machines behaving as if they were hair driers.
My first PC - much like the one pictured in the story, had a fan that was spinning 100% of the time, because most PCs then were too primitive to have a temperature sensor that controlled fan speed. And you couldn't hear the hard drive because the fan was so noisy. So much for these performance indicators!
The only thing audible above the loud PC was when the hard drive died, and spent all its time making nasty mechanical clicking sounds. But since the computer had locked up by that point, it wasn't much of a help!
134 comments
[ 3.3 ms ] story [ 160 ms ] threadIf someone was clever about it, you could probably even feed data back to the external display via SSH from remote boxes.
It's underneath other windows, but usually visible off on the side.
This might be a dark hole for me, I'll end up with an office dressed up like an Soviet reactor control room.
And for istat menus specifically: it's super stable and they're extremely fast to support new OSes, unlike many fancy menubar apps. Easily worth the money for me.
Of course, a performance monitor which updates less often can avoid that issue (for instance, atop by default updates every ten seconds), but most people who run these graphical performance monitors want second-by-second updates.
https://lwn.net/Articles/317923/
Note that iStat Menus have its own set of daemons it uses for keeping tabs on system usage. It doesn’t just hook to the kernel via interfaces and display stats.
CPU usage being high doesn't really tell you anything. You may suspect that there's an issue if you are familiar with the system, and you have only one system to worry about. It's definitely not OK if you have a 'modern' architecture and a fleet of machines running a distributed system. There's a whole lot of noise and you can't tell if traffic has increased, if someone pushed inefficient code, if there's a cooling problem and CPU is throttled, if a "batch" process started, or a million of reasons.
And even if it is high, and it is an anomaly(anomaly detection on those metrics can be useful at times), is it causing any issues for your customers? CPU usage can't tell you that.
Every piece of software has some work to do. What is that work? Figure that out and monitor that. Golden signals on the relevant metrics and you'll be in a far better position. Did the error rate increase? Are we getting higher response times? In that case it doesn't matter what the CPU metric says – it could even be lower than usual if there's a bottleneck somewhere else – it is a problem that needs to be addressed. You can then use the other metrics to confirm, or try to isolate the problem. But they should be supporting information, not your main diagnostics tool.
> In a project I worked on, our development builds started writing about 80 MB of log messages per second to disk.
Well, there you go. If you just look at disk I/O you'll see that it is high. But if you track metrics from your logging system, you'll be able to immediately see _why_
Datadog (no affiliation here) has a good blog that set me on the right track: https://www.datadoghq.com/blog/monitoring-101-collecting-dat...
By all means monitor those metrics on your work laptop - very few people are going to bother running prometheus in their dev machine and setting up alerts for third party apps. But for the apps you are responsible for? Figure out, from day zero, what metrics you should collect for your app and monitor those instead.
https://i.imgur.com/oMPG7IO.png
https://github.com/maxiwell/conky-seamod
https://extensions.gnome.org/extension/5219/tophat/
For years, noisy hard drives with blinking lights served to highlight the difference between thrashy Microsoft operating systems and smoothly running, performant alternatives.
Back in the days I didn't enjoy the LED or the sound of disks. They disrupted my workflow and not always actionable. I mean, I don't really need to know that the machine is working hard on reading the optical discs I just inserted. I will take action if the reading slows down my foreground program.
One of the special infected was the Tank, which is quite strong and can throw chunks of pavement. I always knew when he was coming because my PC was kind of weak and the fans would go nuts about 30 seconds before he made an entrance. The other players were surprised that I had a sixth sense when it came to knowing when it was happening.
It's a windows only application though.
[1] https://mobaxterm.mobatek.net/
Since the first time I got a second HDD, I was always disappointed that there wasn’t a dedicated LED per HDD. NAS cases actually have that.
Something that really taught me to look for things like the "HDD Light" or Fan speed was starting my career in embedded systems. 16 bit MCUs really let you know if you are trying to much on them. They also let you know if you toggle the wrong pin by going up in flames.
The disconnect between your fingers and what actually runs the code is becoming greater and greater for newer developers. It will be interesting to see how computing power keeps up with bad code (it had been doing a good job so far).
Most apparent are the web programmers, most of them assume everyone has 10gbit fiber connections with 16 core CPUs and 128GB of RAM to feed to Chrome. And then they wonder why their shit runs like shit in the real world.
[1]: https://en.wikipedia.org/wiki/Eating_your_own_dog_food
Not all developers. In my entire career, I've never had a development system that was better than the average consumer machine. And, honestly, I wouldn't want one that was, for the exact reasons you state.
Sure, you do occasionally need to do a full recompile, and if the project is very large then this can take a nontrivial amount of time, but that doesn't really happen often enough for me to be bothered about.
If the project is of really significant size, or if the compiler/language/code structure is such that incremental compilation can't give you significant gains (or isn't even possible), you may need more grunt of course.
Facebook marketplace. I don't know how you can make a grid of images max out a 5800x, but they've managed it. A markedly inferior product to every one of the classified ad competitors they've squashed out of existence.
https://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski's_Law
https://en.wikipedia.org/wiki/Jevons_paradox
https://en.wikipedia.org/wiki/Parkinson%27s_law
https://en.wikipedia.org/wiki/Induced_demand
I wonder if this phenomenon can be generalized under one law, because clearly it pops up anywhere humans are concerned.
[1] https://github.com/exelban/stats
Not an edit:
Found it!
https://youtu.be/IZKttBr2Y8g?si=ij3eFlmtrpIsYrkf
I've repeatedly removed and replaced the batteries on my thermostat's remote temperature sensor, because there's no visual feedback that its otherwise working, and my house was cold.
The absence of "working" lights when I know I'm doing a processor-intensive operation on a computer is similarly unsettling. Its not directly actionable feedback, certainly, but it can be a proxy for a more directly actionable, but more onerous to implement or use, feedback mechanism.
If people don't use them, if they don't care about the performance, it is on them, not in the lack of noisy spinning disks and machines behaving as if they were hair driers.
My first PC - much like the one pictured in the story, had a fan that was spinning 100% of the time, because most PCs then were too primitive to have a temperature sensor that controlled fan speed. And you couldn't hear the hard drive because the fan was so noisy. So much for these performance indicators!
The only thing audible above the loud PC was when the hard drive died, and spent all its time making nasty mechanical clicking sounds. But since the computer had locked up by that point, it wasn't much of a help!