I love articles like this, I feel like it dives into each sub-topic the same way I would and goes into just enough detail that I can pick it up from there if I want to learn more
For top if you use the > character it will sort by memory usage. I use that sometimes to figure out why my host is becoming laggy. Also you'll see swapd is taking up CPU.
2 Settings I change on every htop which makes a HUGE difference.
1. I disable user threads. Those mostly just clutter up the htop view while providing no useful information.
2. I enable the process tree view. Very frequently, where a process comes from is much more important than other information. It also lets you see and track things like a compiler process which is eating through a bunch of files.
IMO, both these things should be the default behavior of htop.
Screws have been around for about 3 millennia at this point. They have patently failed to obviate the use of nails. So by this analogy we can expect the 'Only GPUs do the work.' believers to be still promising this, any day now, about three thousand years hence. (-:
> Nowadays most of my processing happens on the GPU, so htop/top better evolve or become mostly irrelevant
If you’re a 3D rendering designer, an ML engineer or a crypto bro, then sure.
Here are the common workloads (for the average SWE on HN) that use CPU/RAM:
- compilation/builds
- language servers and IDEs
- test suites
- local containers
- local databases
- node tooling
- browsers
- data processing
- compression and encryption
- searching/indexing
Well, unless one is using FreeBSD or OpenBSD, where the btop code is still using 32-bit integers to calculate 64-bit sizes, and falling prey to unsigned integer wraparound. htop's code calculates using size_t, which ends up as a 64-bit integer on 64-bit architectures.
It's bad the other way, too. btop on a 50×232 terminal over an actual 57600 BPS serial line is unusable, even in 16-colour mode. It does not have even the most basic of full-screen TUI redraw optimizations, and outputs the entire screen again and again.
Long time user of btop. Only thing it is missing is a ports column next to the others. Also I think the cpu/gpu graph graph is way oversized and would like more space occupied by the open file table in general.
It’s something I put in a very small terminal, hitting the 1,2,3,4,5 key until I get the view I need, just right, and there it sits telling me all during my busy compiles and things, just how max’ing my system is.
It’s very informative, well designed for a TUI, and shows me everything I need to see without requiring futzing around with the menu bar, or dealing with registration nagging, or notification popups, or any of the other mundane things the non-terminal performance monitors seem to think is necessary to stain my eyeballs…
The way to think about it is that btop is a prettier, more modern version of htop, uses fewer resources, but is in general a better designed-UI. It provides a better view of system resources than htop, imho .. its timeline-centric views especially.
I have both, and while btop can look nice, has some templates to cycle through, it lacks things I'm using in htop: s for stracing, l for listing open files, and x for which of them are locked.
It also has some templates to cycle through, and can be made to look nice, too.
That thing about the resource usage? A few MB more residential? Oh noez! (Maybe depends on compilation options) Only thing coming to mind are too fast updates, compared to btop defaults (Whatever the distro put in there).
Timelines? Don't really care about them in there. Have other means to get them. Seems gimmicky to me in that scope.
Only thing missing in htop is n for netstat/sockstat in htop. That would be nice in addition to s,l,x.
Btop is still installed only to pose sometimes, for some people. Because of the Ooooh Cyber! effect.
Timelines are by no means a gimmick. Out of the box, btop has good rendering of key metrics .. htop does too, but btop is easier to read in a console (imho). Nothing cyber about it. “Ooh cyber” is a lame straw man but hey, you had to make it.
Btop is pretty and has good defaults for many things.
A performance monitor is not a good interface to strace, nor is it a good replacement for lsof, nn, arp, etc.
No, not really. To me it felt exactly the same (immature) way. That was a reason why I kept using htop instead. I feel the same about eg k9s, but there's nothing better to migrate off it.
The way I'm using it, somewhat customized from the defaults, showing some more, but mainly still showing the process-tree as the largest field...
... I navigate by either arrow-keys or mouse, mark the process, and apply another view of that process with one keypress, be it the default output of strace for that process, or the default of lsof applied to it.
Why should I have to type more than that, when that's enough in most cases?
The same would apply to netstat/ss for that process if htop would support that.
IMO that's a rather comfortable way around all the pids, p(s)kills and whatnot else.
Call me lazy, but after decades of *nix I really don't care anymore ;->
When I read stuff like this, I come to the realization that even after daily driving Linux for 20+ years I still barely utilize its full potential. Great article.
Fun thing to discuss is what you can't see. Namely that it won't tell you anything about kernel space usage.
ie. I have some computers showing 100gb of 128gb used in free -mh but top shows nothing. It's due to a memory leak in a driver and the computers require reboots every couple of days atm due to this :(.
I appreciate the note on virtual memory not being reliable. This is what Windows task manager reports by default and it's terrible. Resident size is the most reliable metric. Anything else can be wrongfully inflated by things like harmless memory mapped files that won't actually hurt anything. eg. memory map 2GB of logfiles, it'll only be paged in if reading that portion of the logfile so isn't really using memory but users look at the processes and claim "OMG why does this app use so much memory". It doesn't. It uses very little. You're reading the memory usage wrong. Chrome actually had this problem for a while and they moved away from using memory mapped files. Not because memory mapped files are a bad thing but because users will read the memory usage and go crazy over what they see even though it's not really using that much actual physical memory.
If you use memory-mapped files, cached pages count towards the resident set size of your process. If you use ordinary file I/O, they don't. That behavior has amusing consequences in HPC clusters that monitor the memory usage of each job and kill them if they use more memory than they requested.
Just to clarify, Windows Task Manager uses Private Working Set by default for process memory usage which does NOT include shared pages with other processes such as libraries or memory mapped files (hence the name “private”). It only shows the memory that maps to privately allocated physical memory per process. It’s probably closer to Resident Set on Unix.
You probably meant the memory usage in performance tab but I wanted to clarify in case people mistake it for all memory usage fields.
One issue with that relative to RSS is permissions. Historically, all procs could see the RSS used by procs of all other users (at least if they could see the PIDs at all). So, RSS requires no special permissions, but the Linux kernel team decided PSS should not be as promiscuous for whatever reasons (I didn't do a deep dive). So, I'm always having to do (the equivalent of) `sudo pu`.
Resident set size is not the amount of memory that the process wants, it’s how much the OS is willing to give it. So once memory pressure kicks in it stops being representative. I’ve seen this misunderstanding lead to bad decisions a few times. I even went so far to remove this value form charts because a team member was going left when he should have gone right.
For the ones that don't know "nmon", have a look at it as well! (press "h" to see the list of available monitors - press it again to make it go away, press "q" to quit)
A different usage paradigm from *top that I have come to like better is to do differential ps-like reports and system-wide (like vmstat) reports which leaves everything in your terminal scrollback buffer as in: https://github.com/c-blake/procs { written in the uncommonly efficient, expressive Nim programming language }.
/*
* kernel/sched/loadavg.c
*
* This file contains the magic bits required to compute the global loadavg
* figure. Its a silly number but people think its important. We go through
* great pains to make it work on big machines and tickless kernels.
*/
64 comments
[ 3.8 ms ] story [ 53.6 ms ] threadI use htop often but pretty much only use it to find pid or cpu-culprits, and never really understood the rest.
https://github.com/fenrus75/powertop
1. I disable user threads. Those mostly just clutter up the htop view while providing no useful information.
2. I enable the process tree view. Very frequently, where a process comes from is much more important than other information. It also lets you see and track things like a compiler process which is eating through a bunch of files.
IMO, both these things should be the default behavior of htop.
What good does it do to stick your head in the sand?
CPUs are great for orchestrating work, GPUs are great for actually doing the work.
Right, and wouldn't it be really nice if we could check on our orchestrators to make sure their not bottlenecking ops?
"How come we can fully load the GPUs?" "Idk boss, amelius said htop et al were irrelevant so we can't really investigate"
If you have a systems tool that gives an overview of resources used, then better monitor them both.
Imho failing to do so is not future-proof. Your opinion might differ.
If you’re a 3D rendering designer, an ML engineer or a crypto bro, then sure.
Here are the common workloads (for the average SWE on HN) that use CPU/RAM:
Ok sure, top/htop is totally irrelevant now /s(Happened to me)
As others mention it - it seems to shows the Watts used as well :) (and network, and GPU, and disks,....)
[0]: https://github.com/aristocratos/btop
* https://news.ycombinator.com/item?id=48778757 (https://crocidb.com/post/freebsd-ate-my-ram/)
* https://github.com/aristocratos/btop/pull/1728
1.) No zram/zswap statistics. (Though htop only supports zram also.)
2.) No ZFS statistics breakdown.
3.) Doesn't support Arc GPU yet.
4.) I can't disable the disk fill bars, which makes the I/O rate graphs extremely squished unless the console window is very large.
htop scales all the way down to 40x8 on my TRS-80.
You'll be glad you did.
It’s very informative, well designed for a TUI, and shows me everything I need to see without requiring futzing around with the menu bar, or dealing with registration nagging, or notification popups, or any of the other mundane things the non-terminal performance monitors seem to think is necessary to stain my eyeballs…
It also has some templates to cycle through, and can be made to look nice, too. That thing about the resource usage? A few MB more residential? Oh noez! (Maybe depends on compilation options) Only thing coming to mind are too fast updates, compared to btop defaults (Whatever the distro put in there).
Timelines? Don't really care about them in there. Have other means to get them. Seems gimmicky to me in that scope.
Only thing missing in htop is n for netstat/sockstat in htop. That would be nice in addition to s,l,x.
Btop is still installed only to pose sometimes, for some people. Because of the Ooooh Cyber! effect.
But tbh that works with htop too.
Btop is pretty and has good defaults for many things.
A performance monitor is not a good interface to strace, nor is it a good replacement for lsof, nn, arp, etc.
No, not really. To me it felt exactly the same (immature) way. That was a reason why I kept using htop instead. I feel the same about eg k9s, but there's nothing better to migrate off it.
The way I'm using it, somewhat customized from the defaults, showing some more, but mainly still showing the process-tree as the largest field...
... I navigate by either arrow-keys or mouse, mark the process, and apply another view of that process with one keypress, be it the default output of strace for that process, or the default of lsof applied to it.
Why should I have to type more than that, when that's enough in most cases?
The same would apply to netstat/ss for that process if htop would support that.
IMO that's a rather comfortable way around all the pids, p(s)kills and whatnot else.
Call me lazy, but after decades of *nix I really don't care anymore ;->
ie. I have some computers showing 100gb of 128gb used in free -mh but top shows nothing. It's due to a memory leak in a driver and the computers require reboots every couple of days atm due to this :(.
I appreciate the note on virtual memory not being reliable. This is what Windows task manager reports by default and it's terrible. Resident size is the most reliable metric. Anything else can be wrongfully inflated by things like harmless memory mapped files that won't actually hurt anything. eg. memory map 2GB of logfiles, it'll only be paged in if reading that portion of the logfile so isn't really using memory but users look at the processes and claim "OMG why does this app use so much memory". It doesn't. It uses very little. You're reading the memory usage wrong. Chrome actually had this problem for a while and they moved away from using memory mapped files. Not because memory mapped files are a bad thing but because users will read the memory usage and go crazy over what they see even though it's not really using that much actual physical memory.
Just to clarify, Windows Task Manager uses Private Working Set by default for process memory usage which does NOT include shared pages with other processes such as libraries or memory mapped files (hence the name “private”). It only shows the memory that maps to privately allocated physical memory per process. It’s probably closer to Resident Set on Unix.
You probably meant the memory usage in performance tab but I wanted to clarify in case people mistake it for all memory usage fields.
Actually, Proportional Set Size is more accurate than RSS. See: https://en.wikipedia.org/wiki/Proportional_set_size
One issue with that relative to RSS is permissions. Historically, all procs could see the RSS used by procs of all other users (at least if they could see the PIDs at all). So, RSS requires no special permissions, but the Linux kernel team decided PSS should not be as promiscuous for whatever reasons (I didn't do a deep dive). So, I'm always having to do (the equivalent of) `sudo pu`.
https://nmon.sourceforge.io/pmwiki.php
Especially disk throughput and I/O (keys "d" & "D") can be very useful.