HN Explorer

How expensive is SF rent? Use Unix commands to find out

4 points by jph ↗ HN
How expensive is SF rent? Here's a coding challenge that uses Unix command line tools to find out.

Hacker friends suggested I post the result and code here because I'm writing open source statistics software and this is a practical example of results and code.

The results for a one bedroom in SF: mean $3348, median $3475, interquartile range $1514, and standard deviation $1130.

How to use Unix commands to find out: use `curl` to fetch Craigslist apartment listings, then `sed`, `grep`, and `tr` to filter and parse the data, then `num` to calculate the statistics.

    curl -s http://sfbay.craigslist.org/search/sfc/apa\?min_price\=1\&bedrooms\=1 |
    sed 's/\(<p class="row"\)/\n\1/g' |
    grep ' 1br ' |
    tr ">" "\n" |
    sed -n 's/^\$\([0-9]\+\).*/\1/p' |
    num mean median interquartile-range standard-deviation OFMT="%.0d"
The software I'm personally writing is the `num` command to calculate the statistics. The software is free at http://www.numcommand.com

11 comments

[ 3.8 ms ] story [ 40.4 ms ] thread
DrScump
What happens if the price overflows the limit of a 32-bit integer?
jph
If the price overflows 32-bit, then everyone in SF moves. :)

For this script, the fetch returns at most 100 rows, so won't overflow.

More broadly, the Num command runs using awk, and number handling depends on your system's awk implementation and C libraries. Current awk implementations use an internal representation of all numbers, including integers, that uses double-precision floating-point numbers. I know GNU awk supports arbitrary precision arithmetic, and also can use --bignum, GNU MPFR and GNU MP (GMP); these will be great additions to the Num command for the next major version upgrade.

[deleted]
(comment deleted)
gesman
@echo "very expensive"
romanovcode
sed: RE error: illegal byte sequence gawk: cmd. line:1691: fatal: division by zero attempted
jph
Can you contact me about this? I'll diagnose it. My direct email is joel@joelparkerhenderson.com. I really appreciate it.
atsaloli
I had a perl script to do mean, median and std dev; it'd be great to have this functionality available through the OS vendor as a package.
jph
Yes. I'm writing the Num command specifically to become an OS vendor package. (For example I'm writing Num using Awk, which is available on every current POSIX system.)