A few spy tools for your operating system (other than strace) (jvns.ca)

Article mentions lsof but doesn't go too deeply into it. lsof has an absolute _plethora_ of options and has been pretty indispensable to me. i.e. lsof -i (much akin to netstat), -u <user> (all files open by a particular user), etc etc. Check the man page :)

That's indeed a useful list. lsof alone covers a lot of ground!

Mac OS X has a pretty nice set of DTrace scripts built in: http://dtrace.org/blogs/brendan/2011/10/10/top-10-dtrace-scr...

The ones I use the most:

* iosnoop: see all disk I/O. especially useful to find disk-chatty/poll-y apps.

* execsnoop: see new processes being spawned.

* opensnoop: see file opens. especially useful for failed file opens that break an app.

* dtruss: see all system calls. get access to the entire OS interaction history of an process (or app).

* errinfo: trace failing system calls. where there is smoke...

* iotop -- who is using disk

There is just a crazy, crazy list of things available, built-in:

  man -k dtrace

Edit: wow HN formatting sucks so bad. I wish I could make that list more readable, but apparently I can't.

    - Four spaces at the start of a line makes 'preformat',
      useful for code or lists. 
      - You'll want to manually wrap long lines, though.

can't believe no body talks about ss (socket statistics) - http://www.cyberciti.biz/tips/linux-investigate-sockets-netw.... It crazy fast compared to netstat

If you haven't seen Brendan Gregg's Linux performance tools page & slides, you should check it out. He explores a wide variety of great stuff, especially perf and ftrace (as mentioned in the article) as well as a few dozen others that he describes in lesser detail: http://www.brendangregg.com/linuxperf.html

pmap: dump the memory mappings for a process, such as shared libs.

    $ pmap -x $$
    13609:   -bash
    Address   Kbytes     RSS   Dirty Mode   Mapping
    00110000       0      20       0 r-x--  libnss_compat-2.15.so
    00117000       0       0       0 r----  libnss_compat-2.15.so
    00118000       0       0       0 rw---  libnss_compat-2.15.so
    00119000       0      20       0 r-x--  libnsl-2.15.so
    [ ... snip ... ]

One minor correction: atop doesn't need to run as root, it does disable a couple of functions if it doesn't.

pidstat! It's like top, except it prints a continuous log of everything that used CPU (or I/O with -d, etc.) since the last sample. It's the only too, I know that can show per-process I/O activity. Requires root access, unlike top.

vmstat [1] is also a very useful command. It shows various essential statistics in a very compact one line display and optionally repeats this every [interval]. I often start something running while vmstat is giving me stats in another window every few seconds.

[1] https://www.linode.com/docs/uptime/monitoring/use-vmstat-to-...

I will just mention Sysdig for the ones who don't know about it already (http://www.sysdig.org/). It's really the swiss knife of monitoring tools, you can have any kind of information you can think of. If you have not already installed it, give it a try !

procstat, systat, sockstat, gstat, dtrace, truss on FreeBSD

I know there's more I'm not thinking of...

Don't forget powertop

Nethogs [1] groups bandwidth usage by process. Very handy. [1] http://nethogs.sourceforge.net/

Julia mentions atop (which is good), I think htop is also worth noting here as an improvement on top.

For viewing active network connections along with traffic stats, give iftop a try.