HN Explorer

80 comments

[ 0.33 ms ] story [ 224 ms ] thread
jpadvo
This would be fantastic if all it had was the instructions to see your own top commands. What a great way to discover opportunities for automation and time saving!

I can't wait for the full results.

hackerpants
You can leave your email address in the form if you want to get an update when I post them.

I'll also post the results to HN, of course :)

angersock
In case anyone is interested in the relevant incantation:

  history | awk '{print $2}' | sort | uniq -c | sort -rn | head -n 100
OP, maybe add a curl command or something to post the resulting answer to your website form?
hackerpants
Excellent idea! I am investigating how to best do that.
crymer11
This should submit the commands but not any of the optional questions:

  hist=`history | awk '{print $2}' | sort | uniq -c | sort -rn | head -n 100`; curl --data-urlencode -i -X POST -H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' -d "draftResponse=%5B%5D%0D%0A&pageHistory=0&entry.194207258=&entry.1414618252=&entry.1080345712=$hist" https://docs.google.com/forms/d/1XNMoSdfYFe_WkPfU--M88oL00PDLIOAo1HxjhZvZYJ4/formResponse
wging
To anyone copying/pasting: the whole command extends far to the right of what's visible.
JOnAgain
You probably want

`history -100`

on OS X, anyway, the default only gives 15 or so commands in history

hackerpants
Huh, I didn't know that. What command gives the entire history on OS X, then?
RBerenguel
I ran this in Mac OS without a hitch, gave 100. I use zsh, maybe the default shell doesn't?
RKearney
history shows the full .bash_history, but by default HISTFILESIZE is 500 so it only stores 500 entries.
nwh
On 10.8, I'm getting the full 500 lines of history. Not sure that the behaviour has ever been different.
McUsr
I believe I have set HISTFILESIZE TO 2000 or something, I still need to compact it, and have commands like ls pruned by HISTIGNORE.

I use the awkscript below to compact it:

# histsort.awk --- compact a shell history file # Thanks to Byron Rakitzis for the general idea

     {
         if (data[$0]++ == 0)
             lines[++count] = $0
     }
     
     END {
         for (i = 1; i <= count; i++)
             print lines[i]
     }
benmanns
Be careful if you are in the habit of using environment variables to specify API keys or database passwords. One of my top commands is `FACEBOOK_SECRET=...`.
hackerpants
Yes! I don't want to know anybody's secrets. I won't be publishing the raw data for this reason, in case anything like this gets through by accident.
eksith
Is it a good idea to keep passwords in environment variables?

Isn't it safer to create a credentials file and give it the appropriate chmod?

calpaterson
From a deployment point of view, environment variables are a pretty good choice.

http://www.12factor.net/config

vacri
vacri@devbox:~$ ps aux | grep elasticsearch

112 6725 0.1 36.7 1965924 1411164 ? SLl May03 37:11 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms1g -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-0.90.0.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/ -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.ElasticSearch

calpaterson
I'm not sure what you're trying to say, but environment variables and arguments are different things. Environment variables avoid exactly that problem
sdqali
I did history | grep -v "=" | .... to ensure that commands where I specified environment variables are ignored.
jbesurs
This skips all the commands I pipe through, which excludes over half of the commands I use!
ctrl_freak
Try:

  history | sed 's/|/\n1 /' | awk '{print $2}' | sort | uniq -c | sort -rn | head -n 100
Of course this doesn't differentiate between quoted pipes and ones actually being used.
claudius
The ‘awk’ there effectively still drops everything after the second limiter, i.e. you won’t get a different result.

Edit:

    history | sed -e 's/^ *[0-9]*  //' -e 's/| */\n/'  | awk '{print $1}' | sort | uniq -c | sort -rn | head -n 100
works for me.
hackerpants
Excellent point. I'm definitely going to put that caveat at the beginning of any analysis I post. (and thanks to ctrl_freak for posting an alternative!)
decklin
It's ironic that the standard one-liner for this uses a pipeline but only counts the first word of the first command of the pipeline. Given itself as input, "sort" ought to be the most used command, but "history" is the only one counted. (I bring this up not for the sake of standard HN nitpicking but to point out that you probably do want to cover at least `sort` in your workshop, and it will be underrepresented in these results, along with `grep`, `wc`, etc.)

Unfortunately, shell grammar is complex enough that a correct one-liner is probably infeasible. For example, one of my top "commands" if you count by words is an environment variable setting prepended to an actual command.

EDIT: you can get good enough results by just splitting on "|", as others have suggested here -- any parts of regular expressions, etc that aren't really commands will probably be infrequent enough to get lost in the noise, and treating || as containing an empty command won't hurt. If you're going to catch ||, though, might as well get && too... and now you're going down the rabbit hole :-)

hackerpants
Yup, this oversight is officially embarrassing.
ColinWright
Quick hack - you can include the commands you pipe to with this:

    history                  \
        | sed "s/^[0-9 ]*//" \
        | sed "s/ *| */\n/g  \
        | awk '{print $1}'   \
        | sort               \
        | uniq -c            \
        | sort -rn           \
        | head -n 100        \
        > commands.txt
I haven't tried to account for pipe symbols inside strings - it didn't seem work it.

In case there are commands you want then to exclude (which I do) then you might want to "head -200", remove the commands you don't want to provide, and then trim to 100.

Added in edit - having done this a good half of my top 100 are actually scripts, so this is pretty pointless for me unless I rummage through them to find the common commands.

Added in edit again:

OK, here's a version that only includes actual system commands, and hence filters out all my personal scripts and commands:

    history                      \
        | sed "s/^[0-9 ]*//"     \
        | sed "s/ *| */\n/g"     \
        | awk '{print $1}'       \
        | xargs which            \
        | sed "s.^/usr.."        \
        | grep ^.bin             \
        | sed "s/^.*\///"        \
        | sort                   \
        | uniq -c                \
        | sort -rn               \
        > commands.txt
hackerpants
Thanks very much! As of 2:14pm, it's using your new version for greater pipe accuracy =)
__david__
I think you want s///g on your second sed... Also that doesn't work in Mac OS X for some reason (their sed doesn't appear to interpret \n in the replacement text). I replaced it with perl to make that part work:

    perl -pe 's/ *\| */\n/g'
I still haven't gotten the whole thing to work yet because my history contains the above history pipeline and so it's splitting the "|" that inside the sed command onto multiple lines which is causing "xargs which" to balk because quotes are not matching or something:

    xargs: unterminated quote
Shells are amazing until spaces or quotes are involved! :-)
ColinWright
Yes, I've inserted the "g" in the appropriate sed commands. Thanks - good catch.

Some systems seem to require \r instead of \n - I know vim's behavior differs from sed's in this, so that might be an issue.

With xargs balking, you can throw the error stream at that point, for convert it to a loop over the alleged commands:

    for c in $( long thing before the xargs )
    do
        which $c
    done \
    | long thing after the xargs.
Specifically:

    for f in $( \
        history                      \
            | sed "s/^[0-9 ]*//"     \
            | sed "s/ *| */\n/g"     \
            | awk '{print $1}'       \
        )
    do
        which $f 2> /dev/null
    done                         \
        | sed "s.^/usr.."        \
        | grep ^.bin             \
        | sed "s/^.*\///"        \
        | sort                   \
        | uniq -c                \
        | sort -rn               \
        > commands.txt
And yes spaces are a pig, and can lead to all sorts of ambiguities that don't have reasonable ways of resolving them, especially in filenames.
__david__
"\r" doesn't work either. It's just not interpreting those kind of escapes. I get lines like this:

    historyrsed "s/^[0-9 ]*//"rsed 's/ *r*/\r/g'rless
The for loop is a good idea, though I prefer the "while read" idiom since it fits in with the pipeline better:

    ...
    | awk '{print $1}'                      \
    | while read line; do which $line; done \
    ...
That finally works for me. Here's the whole thing:

    history                                     \
        | sed "s/^[0-9 ]*//"                    \
        | perl -pe 's/ *\| */\n/g'              \
        | awk '{print $1}'                      \
        | while read line; do which $line; done \
        | sed "s.^/usr.."                       \
        | grep ^.bin                            \
        | sed "s/^.*\///"                       \
        | sort                                  \
        | uniq -c                               \
        | sort -rn
lobotryas
From what I recall to represent s newline in sed on osx you need to add an actual carriage return.

Take the poster's script above and copy it to a text file. Now replace \n with a new line, save and it should run.

NyxWulf
This works for me in zsh on OSX

history \ | sed "s/^[0-9 ]//" \ | perl -pe 's/ \| */\n/g' \ | gawk '{counts[$1] += 1} END { for (x in counts) { print counts[x],x}}' OFS="\t" \ | column -t \ | sort -k 1,1nr \ | head -100

Ok not sure how to format this on HN https://gist.github.com/nyxwulf/5608955#file-gistfile1-sh

wasd
I really want to see the output of this survey. Any chance we can sign up to get an email notification when its published?
vsbuffalo
I work in bioinformatics and git is my most frequently used command. I wish this was more often the case in science.
hackerpants
do you know about the organization Software Carpentry? http://software-carpentry.org/

They work on teaching scientists to be better at software, and are always looking for people to do workshops

madsushi
My top 5: vim, mysql, grep, php, nginx

I suppose you could guess that my Linux box is an LNMP server pretty easily.

hk__2
This probably won’t be useful, since I use a lot of aliases. Here are my more common commands:

    g (git)
    l (ls)
    c (cd)
    v (vim)
    make
    fg
    rm
    m (mv)
    s (sudo)
INTPenis
I run wildly different commands on different hosts so I actually took three hosts I'm often logged into and combined their most run commands in history into one 100 line file.

Fun project, good luck!

e3pi
Super-facile gpm plopping grabs across six nox tty consoles onto emacs and as apps arguments is the most pleasant, productive, and relaxed user interface I know.
dmckeon
For those using the bash shell, the hash builtin:

    $ hash | sort -nr
may be informative, producing counts and command paths, although only for the current shell instance.
frakkingcylons
Heh, my top five are javac, java, git, cd, and mrt.
eddieroger
I'm either really bad at using git, or love the crap out of it. Not sure which.
waltz 

    149 sudo
    123 cd
     79 ls
     57 nano
     47 get (alias for sudo apt-get install)
     40 acpi
     36 clear
     18 sshlg (custom command to access my ssh server)
lelandbatey 

    738 ll (alias for a more reformed `ls` output)
    549 python
    457 cd
    397 git
    173 exit
    157 vim
    150 less
    142 ssh
    122 ./synchVeiled.sh ( script I'd been using a lot for a project )
    121 sudo
    109 scp
    109 fab
    101 gca ( alias for "git commit -am" )
     97 curl
     88 bash
     74 grep
     68 startx
     56 rm
     56 lguf ( alias to show all files in a git repo that are not being tracked )
     55 history
     54 whois
     49 htop
     48 source
     40 echo
     39 foreman
coyotebush
My history file is de-duplicated (oh-my-zsh enables hist_ignore_dups), so this sort of approach is never accurate.

Git comes out on top, but presumably because most of my invocations are unique ("git commit -m ..."), while things like "ls" are farther down the list.

Relevant: https://github.com/paulmars/huffshell

D9u
I regularly wipe my .history. Sorry.
pavanky
The list from my laptop is very different from my development machine of choice. Not sure if adding it twice would skew the results.
codygman
Not very difficult to tell how I use my computer :D

    139 vim
    137 coverage
    135 ls
    105 cd
     98 git
     98 fg
     52 clear
     25 sudo
     19 ./manage.py
     17 grep
     16 mysql
     14 source
     14 find
     13 python
     11 rm
     10 ssh
      8 mv
      8 go
      7 ps
      6 rsync
      6 pip
      5 xargs
      5 mkdir
      5 cp
      4 xclip
      4 sed
      4 pianobar
      4 man
      3 vi
      3 touch
      3 pkill
      3 deactivate
      3 alsamixer
      2 time
      2 tar
      2 startx
      2 ssh-add
      2 sprunge
      2 kill
      2 java
      2 ./build.sh (script to build a Go program)
      2 ./bin/api_server
      1 xbacklight
      1 which
      1 s
      1 rmdir
      1 redshift
      1 lesss
      1 less
      1 jobs
      1 history
      1 hg
      1 export
      1 dmesg
      1 date
      1 cat
      1 ./all.bash
      1 acpi
gesman
'egrep' is the first on my list.

I guess the future is all about regular expressions :)

m0nastic
It might just be my OCD showing, but 'clear' is my number one by an order of magnitude.

next 20:

  177 ls
  167 cd
  122 brew
  120 vim
  119 cabal
  117 cindy (an alias for ssh'ing to my local dev box)
  115 sudo
  114 tmux
  111 man
  110 less
  110 openssl
  109 rm
  108 whois
  107 ghci
  107 lein
  106 ssh
   86 strings
   75 smbclient
   75 racket
   65 nc
dpritchett
I use ctrl+L a lot in my term. It clears.
albemuth
command + k on a mac, I don't even want to know how frequently I use that.
nailer
Control L works on Mac too.