HN Explorer

24 comments

[ 3.0 ms ] story [ 61.7 ms ] thread
dragandj
The problem with these kind of extremely short texts is that they jump into explaining technical concepts without even mentioning basic things such as random variable for example.

Then, readers learn that statistics is about applying a few summing and averaging procedures, and trust the numbers they get as a result.

RA_Fisher
I totally agree. It's much better to start with probability and then move to statistics. It gives both a foundation and the motivation for statistics.
analog31
Indeed, and I think Python could be great for learning about probability too. It comes with an endless supply of random numbers. ;-)
melling
Anyone have any recommendations on where to start? In particular, I’ve been looking for a good book on probability.
joshvm
Jaynes is the usual 'standard' probability text, and it's well written and not too dry. It's mathsy though.
dragandj
William Bolstad - Introduction to Bayesian Statistics

Or E.T. Jaynes - Probability Theory, The Logic of Science (but this one is much more detailed and has lots of philosophy)

Or Ian Hacking - An Introduction to Probability and Inductive Logic (this is the light approach written for philosophy students)

mubaris
I'll be posting a separate blog post about probability soon. Anyway, thanks for the suggestion.
gshubert17
Nice example. But two typos,

Missing a close paren to open(); and splitlines() is misspelled. Code should be:

with open('salary.txt') as f: X = f.read().splitlines()

nafizh
Calculating mean, median, and average in Numpy. How is this in the first page of hacker news?
branchless
It's taking space that could be taken by yet another article about confidence/burnout/fitness/sleeping/nutrition!
[deleted]
(comment deleted)
bckygldstn
The formula you give for variance is the sample variance, with N - 1 as the divisor. It gives an unbiased estimate of the variance of a population, assuming X is a sample of that much larger population.

However, by default numpy gives you population variance, with N as the divisor. This assumes X is the entire population, which is different from R and Matlab.

As a result, the answer in your code is different from the formula just above it.

  np.var(X)
gives 207, while

  np.sum((np.array(X) - np.mean(X))**2) / (len(X) - 1)
gives 236. If you want the N-1 corrected sample variance from numpy you'd use

  np.var(X, ddof=1)
cristoperb
It's a little bit confusing that the article defines variance as "the averaged squared deviation of each data from your dataset" but then the formula shown is the unbiased estimation of the variance (with n-1 in the denominator) without explanation.
lettuce
For such basic functionality, is there any reason not to just use Python 3's `statistics` module?

  from statistics import mean, median, stdev, variance
jzwinck
Two reasons. First is that NumPy works in Python 2 as well as 3.

Second is that everyone uses NumPy for this. It is proven, it is fast, and using it for simple things is a great way to get started on a path toward using it for more complex things. For example I use NumPy for tabular text processing sometimes, as it is much faster than the built-in stuff if you have a lot of data.