HN Explorer

Ask HN: What are your favorite statistics and probability textbooks?

452 points by newguynewguy ↗ HN
I took stats in undergrad, but it was a very rudimentary "push x sequence of buttons on your calculator in y situation" ordeal and left me with less applicable knowledge than I'd like.

Since sometime after graduation, I've taken up serious study of higher-level maths as a hobby. I think it will be most useful in my career if I also have a strong grasp on probability and statistics.

[edit] The suggestions so far have been great, but it occured to me to add that I'm working through Spivak and Apostol right now with Dedekind's essays on the side. Hopefully that gives an idea of the tone/rigor I'm after. Answers to problems is also ideal.

86 comments

[ 2.8 ms ] story [ 134 ms ] thread
jamessantiago
I'd recommend almost anything by Andy Field. He has a clear cut approach with a bit of comic wit to make the reading more enjoyable for hobby reading. His book on SPSS was fantastic, but may be a bit too specialized for what you're looking for. Check out discovering statistics:

https://www.discoveringstatistics.com/

_eigenfoo
If you're looking for just mathematical statistics, I liked Hogg McKean and Craig's "Introduction to Mathematical Statistics" (4th edition was much better than the ones after it, imo).

But if you're looking to learn prob/stats for applications to ML, most ML textbooks have a chapter or two reviewing the relevant stuff. I liked the first two chapters of Bishop's PRML for that.

eigenvalue
Feller Vol. 1 is considered the classic text for discrete probability: https://epubs.siam.org/doi/10.1137/1011021

Volume 2 is easily found as a pdf on google, but it's much harder.

soVeryTired
Feller is damn hard work. I'd put feller's probability on the same level as Rudin's real analysis.
EpiMath
Agree Feller is excellent. I don't think it has much in the way of prerequisites and comes at probability from a solid "fundamental understanding" POV.

Volume 2 is more difficult but also very good.

Don't forget that you always read mathematics ( or probability theory ) with a pencil in hand. It's a participatory activity.

graycat
Both Feller volumes have some good stuff. E.g., volume 2 has the renewal theorem -- roughly, arrivals from many independent sources look like a Poisson process, i.e., times between arrivals are independent identically distributed random variables with exponential distribution where the parameter in the distribution is the arrival rate. E.g., expect that between 1 PM and 2 PM, arrivals at a popular Web site will be Poisson, and might use that for capacity planning or anomaly detection.

But as my main probability prof summarized, an unguided tour of Feller is not promising. I believe he was correct. Look elsewhere for the first intuition, common applications, or the high quality math, e.g., based on sigma algebras and the Radon-Nikodym theorem, the limit theorems -- laws (weak and strong) of large numbers, central limit theorem (the strong Lindeberg-Feller version, irony here), the ergodic theorem, martingales, Skorohod representation, Kolomogorov extension theorem, Lebesgue decomposition, etc.

kgwgk
Lindley’s Understanding Uncertainty

Jaynes’s Probability Theory: The Logic of Science

MacKay’s Information Theory, Inference, and Learning Algorithms (http://www.inference.org.uk/itila/book.html)

ReadEvalPost
Jaynes is a delight to read if you like an opinionated style over the more usual dry prose of other textbooks.
mhneu
If you're working through Spivek and Apostol, get Feller v1 and v2. Same level of rigor and emphasis on intuition. (If you're having fun after that, pick up Gallager's book on stochastic processes. Similar approaches, intuition, and focus on discrete probability to skip the metric theory complications of continuous dists.)

Another very good book is Bertsekas and Tsitsiklas, Introduction to Probability. This book will also give you some intuition.

I've heard Grimmitt recommended but I think B&T is better.

You could also start with Papoulis (a stochastic processes classic book, but it does the intro probability too.)

abstrakraft
In my experience, those who grew up with Papoulis, 2nd edition loved it. I grew up with Papoulis, 3rd edition, and find the layout and typesetting to be among the worst of any book I've ever read, bad enough to significantly affect usability and ability to quickly find things.
mhneu
Yeah, I personally don't love Papoulis. But it has been regarded as the standard stochastic processes text.
selimthegrim
What are people’s thoughts on Loeve? Too old, too much to bite off?
srean
It's an excellent book to get ones fundamentals right. Might not work for a large part of hacker news audience. Since CS deals primarily with discrete spaces they don't need as much rigour in analysis or measures.
ArtWomb
I think you will enjoy Efron & Hastie's Computer Age Statistical Inference: Algorithms, Evidence and Data Science.

https://web.stanford.edu/~hastie/CASI/

MIT 6.008 Introduction to Inference also goes over many of the classic problems such as Cookie Jar, Monte Hall, Fair Dice, One-Armed Bandits, etc. with tons of labs.

I find this stuff easier to grok by just writing code and running a simulation. To that end, I'd setup an Anacondas environment and start experimenting with PyMC3. As your intuition suggests, there's no downside to mastery of probabilistic programming. Best of luck!

https://docs.pymc.io/

kgwgk
I think Efron & Hastie may be a bit too advanced and terse for the OP. They cover many things in not so many pages and "our intention was to maintain a technical level of discussion appropriate to Masters’-level statisticians or first-year PhD students."

But given that they distribute the PDF for free it's worth checking out. Hastie, Tibshirani & Friedman's The Elements of Statistical Learning and the watered-down and more practical Introduction to Statistical Learning are also nice. All of them can be downloaded from https://web.stanford.edu/~hastie/pub.htm

achompas
Elements of Statistical Learning is the other text I came in here to recommend.

One of my most valuable activities in grad school was printing and studying each chapter of EoSL.

It's a comprehensive text on the fundamentals of statistics and machine learning, a solid foundation for the cutting-edge techniques relying on deep learning and reinforcement learning.

jackgavigan
Against the Gods by Peter L Bernstein, Chaos by James Gleick, and The (Mis)Behaviour of Markets by Benoit B Mandelbrot and Richard L Hudson.
kgwgk
In this less mathematical, more "pop science" vein I remember I enjoyed The Flaw of Averages by Sam Savage (who happens to be the son of Jimmie Savage) but I read it many years ago and I'm not sure I would still recommend it.
johnmyleswhite
All of Statistics by Larry Wasserman is the book I recommend for anyone who knows calculus and linear algebra and wants to learn statistics.
ellisv
Came here to post this and “Advanced Data Analysis from an Elementary Point of View“ by Cosmos Shalizi
madcaptenor
I just wish this would come out in paper book. It’s excellent but I have difficulty reading math books online.
mkl
Supposedly in process. From the book's website: "The book is under contract to Cambridge University Press; it should be turned over to the press [...] before the end of 2015." But it sounds like that was after a few delays already, so maybe there have been more.
_qhtn
You can print ebooks.
antognini
Amazing book. It has an ambitious title, but it really lives up to it. I just wish I had read it years ago.
shoyer
Yes, I loved this book! It’s compact enough that you can really get a flavor of things very quickly.
fusiongyro
Kolmogorov's Foundations of the Theory of Probability is pretty good, IMO. I haven't made it very far, but he has great examples, plenty of rigor, and the book is like $10.
halhen
I'm not well rounded enough to draw a clear path from where you are. For me Gelmans Data Analysis Using Regression and Multilevel/Hierarchical Models [0] drove home many, many points. More recently, I have a sense/hope that Pearl's The Book of Why [1] might take this to yet another level.

[0] http://www.stat.columbia.edu/~gelman/arm/ [1] https://www.amazon.com/Book-Why-Science-Cause-Effect/dp/0465...

anonymouslee
I've been enjoying the struggle through Statistical Inference by George Casella as a foundational text. This was the introductory text for stats graduate students at Arizona State University a few years back.

The Mathematical Methods of Statistics by Harald Cramér is also excellent.

achompas
+1 to this, I'm a big fan of Casella. I personally think it's most useful as a reference, moreso than a textbook.
frabbit
Is that the same as Casella & Berger? Nice Sherlock Holmes quotes at the beginning of each chapter.
mlevental
casella and burger is the standard grad math stats book and good-ish departments i.e. it proves things using calculus but it's for statisticians (covers things like efficiency and biasedness and etc).
vinchuco
FWIW here is a list of texts used in a data science conferene as references in presenter's slides (Cassella Berger was mentioned here by others, Hastie Tibshirani is also good). https://hastebin.com/raw/upiqumenas
stakhanov
"Theory of Probability" by Bruno de Finetti is the best text on probability ever written.
kgwgk
"A book destined ultimately to be recognized as one of the great books of the world" according to the foreword by Lindley included in the English translation. "Probability is a description of your (the reader of these words) uncertainty about the world. So this book is about uncertainty, about a feature of life that is so essential to life that we cannot imagine life without it."
[deleted]
(comment deleted)
geokon
A great starting book that helps build a good intution is "An Introduction to Error Analysis" by John Taylor (of Classical Mechanics fame). Basically if you are taking any kind of measurements and need to compose them intelligently this is where to start. From sig-figs to chi squared. A must read for any starting researcher

My experience otherwise with stats books has been horrendous.. (havent found a good "medium level" book to move on to yet)

dyukqu
Anyone has anything to say about The Lady Tasting Tea[0]? I'd like to hear/read your thoughts about this book. (I came across it on the comment section of John D Cook's blog post[1] years ago but haven't had a chance to read it yet).

[0] https://en.wikipedia.org/wiki/The_Lady_Tasting_Tea

[1] https://www.johndcook.com/blog/2013/01/12/elementary-statist...

rmbeard
Nice pop science/history of stats book. entertaining read. You need to have quite a deep understanding of statistics to really understand it though.
mindcrime
I enjoyed it. I read it when I knew almost nothing about statistics, and found that it motivated me to learn more. And it's frankly an interesting / entertaining read in its own right. It's not a textbook, so don't expect to come away knowing a lot of stats after reading it (unless you already do!), but I'd say it's worth reading.
adamnemecek
Shafer & Vovk: probability and finance. It bridges this gap for me that I didn’t quite understand.
cfusting
Probability and stochastics by Erhan Cinlar is a modern, measure theoretic approach to probability.
georgewsinger
All of Statistics by Larry Wasserman (for its rigor).
dwohnitmok
As a first text, I really like Introduction to Probability by Blitzstein and Hwang. It's not quite as mathematically sophisticated as some of the other treatments (e.g. you're not going to see the sigma-algebra treatment of events over a sample space), but it has a very heavy focus on building good statistical and probabilistic intuition which I think is invaluable for studying statistics in and of itself rather than statistics as a subset of measure theory. The book imparts a good knack for being able to quickly glance at a statistical problem and see "oh it's one of those kinds of problems," or "hmmm looks like there's a good symmetry argument [or other general approach] I can use here," which continues to reap rewards in more advanced and specialized statistical studies.

There are accompanying lectures, problems, and accompanying solutions on its website https://projects.iq.harvard.edu/stat110/home

Spooky23
Not a textbook, but I found “Statistics in a Nutshell” to be a good refresher for my long dormant stats knowledge.
soniman
This isn't what the OP is looking for but Introduction to Probability with Texas Hold ’em Examples is a fun intro to probability.