50 comments

[ 568 ms ] story [ 2535 ms ] thread
What is kind of weird is seeing Information Theory and Kolmogorov Complexity being described as "kind of new".

I guess in some sense it is, but for myself in 2012 I recall first hearing about the work of both Claude Shannon and Andrey Kolmogorov as a university student in 1982 in a first year maths / CS bridging lecture called computational mathematics or some such.

Pretty much everything to do with signals, compression, error correction, etc. happened post the invention of the telegraph - you had Gray Codes (patented Frank Gray ~ 1945ish) popping up in recreational mathematics in the 1880s and being applied to the telegraph by Émile Baudot, from that time forward we see the unfolding of much of what is CS / computational mathematics today.

I guess everything last century is "kind of new" but it is starting to feel as though it's all been around for a while now.

Think of it this way: The mathematical capstone of most undergrad non-math-major courseloads is Calculus II, which focuses on mathematics largely developed by Newton and Leibniz, as modified by Riemann.

That takes you all the way to the mid 19th Century for what little of Riemann's work the course focuses on, with the bulk (Newton's and Leibniz's contributions) coming from almost two centuries earlier.

It's pretty common these days to include a dynamical-systems course in the math core for science majors, which is largely 1960s-70s developments.

Beyond that I think information theory often tends to be treated as more an area of engineering than math, along with digital signal processing, to take another common example. So it's usually taught by engineering or CS departments if by anyone. At schools with a strong mathematical bent to their engineering/CS departments there are usually good courses on both, though.

This is a rather simplistic account. Newton and Leibniz neither started nor finished the development of calculus, they just happened to make the connection between differentiation and integration which opened up the way for the spectacular development of the subject in later years and Leibniz invented some of the notation we still use, but there was lots of progress before and lots of progress since then, especially regarding logical foundations of the subject and a big part of it is now incorporated in the undergraduate curriculum. Linear-algebra based multi-variate calculus is pretty common in universities now too and it wasn't well developed before the 20th century.
I actually needed to read this again... and perfect timing, too.

Hmm... a feed sprinkled with systematic reposts of things that you enjoy, that motivate you, or that you don't want to forget. Has anyone tried that?

If upvotes were public (or at least available to you when logged in), it could just trawl HN's history at some set interval (one year past?) and feed you anything you upvoted that day.

edit: or rather, a few intervals to allow for reposts even after that year period

A bit off-topic, but isn't HN checking whether a URL has already been submitted before?

I ask because sometimes I tried to submit an article that was already there and HN just pointed me to the original discussion, so I don't understand how can duplicates happen.

There is, but after a while, it expires, allowing the article to get resubmitted.
I think that in this article Steve discounts the effect of actually doing the math for yourself. It's comparatively easy to read about something and think that you've understood it. It's another thing entirely to be able to do that math on paper and get the 'correct' answer without needing help.
There's actually a place slightly beyond "able to do it on paper" at which you grok the maths so well that you can apply it in non-obvious situations without any conscious effort. I'm not sure how to get there in general, but I've noticed that I'm more likely to be able to do this for subjects that I learned in proof-heavy classes.
I've always envied people who could do that. A friend of mine told his mathematics professor a riddle (http://en.wikipedia.org/wiki/Random_permutation_statistics#O...) that had taken my friend hours to logic out. The professor began talking it out and, within about 15 seconds, had produced the simple group theoretic solution that Wikipedia gives.
I have a Masters in math and the applying things in non-obvious situations is one of the things that I really gained from it. Several times I've been in situations where someone is struggling with a problem, and where I can quickly spot a simple mathematical approach to solve it. Many times the mathematics itself is quite simple advanced high school or early college stuff, stuff the person struggling with problem has also studied.

The difference is that to them linear algebra (for example) is something abstract you use to solve linear algebra problems, while to someone with a more deep and solid understanding of math, linear algebra is a simple and general tool that can be applied in all manners of situations.

One of the school math topics that I have made us of a lot is function transformation and translation.

Being able to see things like "what I really need here is this part of a sine curve, stretched along the X axis, compressed along the Y axis, and moved up into the positive", and then write a simple one-liner to do slideshow animation, has proven useful again and again.

To be honest, I didn't really understand the point of it during class, and actually failed that part of the exam. A few years later when I needed it, it just popped out of my brain attic and made sense.

> I understand what an equation means if I have a way of figuring out the characteristics of its solution without actually solving it.

-- Paul Dirac

"In mathematics you don't understand things. You just get used to them" - John von Neumann
One big pet peeve from math class: circular functions are taught in "Trigonometry" while imaginary numbers and vectors are lumped together with basic "Calculus" meaning Euler's formula seems like terrifying voodoo for years after it is first described!
Yes, but if we introduced Euler's work that early trigonometry would be reduced to the study of a single equation!
But in fact there's not much more in trigonometry. I always wondered how American schools can spend whole semester on single, simple subject.
Yeah trigonometry does become pretty trivial if you're prepared to use complex numbers and some calculus. Here in the UK we were introduced to this approach later on at A-level, but still had to tackle an incredible number of "solve for the missing length/angle/area" triangle problems first.

(One thing the endless triangle problems do provide is practice at converting geometric problems into algebraic ones. But yes, too much of them I'd say.)

I guess teachers consider complex numbers an abstraction too far in order to teach applications to geometric problems. They don't want to field endless 'woah that blows my mind' type questions about i at that stage. The downside then being that you have to memorise (or read off a cheat sheet) a lot more identities in order to manipulate things algebraically. And to convince yourself of the truth of those identities you need to rely on geometric proofs, which come more naturally to some than others. Probably makes the subject more intimidating than it should be.

I learned maths by attending:

http://www.cs.ox.ac.uk/softeng/subjects/SEM.html

Which is open to anybody willing to pay ~$2k for a week and get their ass to Oxford - you don't need to be enrolled as a student.

If you don't have that kind of money to spend (or can't get to Oxford), the course is essentially the first few chapters of: http://www.usingz.com/ which is available for free.

That looks excellent, but a very narrowly defined course around the concepts used by Z. The CS course I did had 3 years of maths and then I did postgraduate work with Control Engineers and I realised that, compared to them, I hardly knew anything (good fun learning as much as I could though).

However long you spend learning maths, probably more than any other subject, you are only going to end up knowing an awful lot about less and less (the usual curse/joy of academic specialization).

Given the post was 5~6 years back, I would be interested on Steve Yegge's reflection on the post. I would be interested to ask question like 1. How much did he progress mathematically ? 2. How it helped him ?

Would be interesting if Steve Yegge can do an AMA on reddit.

Yes, yes, yes. Discrete math is useful every single day as a programmer, unless you see absolutely no need to ever think about what's happening under the hood. Sets, combinatorics, matrices, and on and on. Everything I learned in my "discrete computational structures" course is directly applicable in my day-to-day programming life.
Could you recommend a book or an online course? Or anything really? As more of a self-taught programmer, I'd love to be more acquainted with the mathematics that go along with computer science, but I'm really not sure where to start. I've been waiting for khan academy's offering on the subject but it's been years and it seems like they'll never get around to it.
> For programmers, the most useful branch of discrete math is probability theory. ... What's probability theory, you ask? Why, it's counting. ...

That paragraph made my left eye twitch a little. Not a big mistake, perhaps, but what he calls probability theory is actually combinatorics. Probability theory is decidedly non-discrete (integration of density functions?) and involves only two integers: 0 and 1. (A joke).

Many people are introduced to combinatorics in the context of probability theory. It (combinatorics) can be used to compute various probabilities, but still. You wouldn't call trigonometry calculus just because a course in calculus might involve the derivative of sine and cosine, so don't confuse probability theory and combinatorics.

How is probability theory decidedly non-discrete? As far as I know as long as the number of possible outcomes is discrete (and in CS applications this is the common case) the integration of density functions turns into simple summations and there is a vast body of results and research limited to just this area of probability theory.
Sigh. Maybe "decidedly" was not the correct word. Still, "if you only consider the discrete parts of probability theory, then probability theory is discrete math" is not useful.
Combinatorics deals with discrete structures, the kind encountered in programming, while probability concerns itself with determining the likelihood of a given event. The former is clearly more relevant to general programming.

Probability does use some tools from combinatorics and vice versa, but the two subjects have vastly different goals.

Yes, I understand combinatorics is different from probability theory and yes, he is somewhat mistaken in not distinguishing combinatorics from probability theory, but on the other hand the article is clearly meant to be informal, it isn't a dissertation on the division of (highly overlapping) sciences. Above all else, I find the statement from the critique above that "probability is decidedly non-discrete" wrong much more clearly then any statements from the original posting.

Similarly I find it highly questionable whether combinatorics is "clearly more relevant to general programming". In fact, what is "general programming"? Probability theory pervades all of Computer Science - from probabilistic algorithms (QuickSort), through cryptography, optimization algorithms (genetic algorithms, simulated annealing), networking (information theory, queuing theory), machine learning, the list goes on and on. For business programming, statistics (based on probablity theory) is crucial. Some of those applications include combinatorics, but I find it harder to find such a long list of applications of combinatorics being used without probability theory - it is a more specialized field.

I guess it might have been better to say there's a Venn diagram of "probability" and "discrete structures" and there's plenty of non-overlap in both directions.

But, it's important to appreciate that, even for probabilities on finite or countable sets, you get pulled into continuous math. I don't think we disagree here, but maybe some examples would be worthwhile, just for definiteness.

Expectations (means, variances) of discrete variables will be real-valued. There are continuous processes that are intimately connected to what you thought were purely discrete outcomes (e.g., the relationship between Poisson counts and exponential waiting times). There will be limiting processes of discrete structures that bring in continuous probabilities (the Binomial -> normal limit, and all its generalizations) and provide considerable insight. The Stirling formula, which is key to insight about factorials, comes from calculus.

Finally, generating functions are one of the main tools for solving the discrete summations you mention, and these are continuous, and pull you straight into complex analysis. ("Who changed the subject here? I was just trying to add some binomial coefficients and now we're talking about derivatives of analytic functions at zero?")

Or, as my edition of Concrete Mathematics says (sec 5.4): "We come now to the most important idea in this whole book, the notion of a generating function."

lots of physicists commenting on the article, getting annoyed that the author doesn't dig calculus.

As a physicist (turned programmer turned physicist again) I'd just like to say that I quite like the idea of a liberal arts mathematics course - I would be much better at it if I had an overview of the field, and thus knew where to look for my solutions (like I do when debugging a program under X environment or framework).

It's a bit short sighted to claim that trigonometry and calculus are 'only necessary for scientists'.

For example, if you write code for some user interface, you'll have a very hard time to get smooth animations if you don't understand derivatives.

I've never found any really good resource to learn Math from.

The problem I get is that whenever I see a big equation I tend to see a few greek symbols that I have never seen before and I have literally no idea whatsoever what they mean.

At least when looking at some unfamiliar code etc I can usually tell from function and variable names approximately what something is supposed to be.

Wikipedia is hopeless for learning because the math articles seem to be more interested in being complete and accurate than being accessible. When I hit a math page on something on wikipedia it tends to link off to a whole bunch of other pages on increasingly abstract parts of Math leaving me more confused than when I started.

I'd never have time to go through all of these topics and learn them properly, if I was going to do that I'd have become a mathematician.

Khan Academy is good, but seems to focus more on the mechanical skill of doing math rather than explaining concepts.

This was true for me, and partly out of frustration of this I picked up a major in math in addition to comp sci so I could get through more math heavy computer science books.

It depends what you are reading, but a good discrete math textbook provides a lot of groundwork for mathematics commonly used in computer science, but expects little in terms of prerequisites, so notation is usually explained.

At some level, learning the fundamentals is just necessary, but for computer science discrete mathematics and linear algebra (the basics of which are easy and also don't have much in terms of prerequisites) will get you far.

On the other hand higher math is ruthlessly cumulative. If you find that you need to understand Lagrange multipliers, for instance, you'll see they relate to the gradient, which has its own symbol, which is defined in terms of partial derivatives, which have their own symbol, which relates to limits of vectors, which have their own symbols and notations, etc.

I wouldn't say that it is just higher math. As a high school student, here is my path of e^ix.

Physics 1 Advanced, Simple Harmonic Motion is being taught. When told to find a case where the second derivative of a function is a negative constant times the function, I think e^ix. The answer is sin(x), but Euler's equation now makes sense. The most beautiful equation now makes sense.

A month later, glancing in a book of useful mathematical equations, I see sin(x) = (e^ix - e^-ix)/2i. Hyperbolic trig suddenly makes sense, which is useful over a year after finishing it. The weird equation for the normal curve from statistics starts to make more sense, because e^(-xx)=(e^(ix))^ix, which helps explain the 1/sqrt(2pi) weirdness. I start to understand why the most beautiful equation has its name.

I think that most math is cumulative, but only some parts are hard enough to make you notice it.

Wikipedia is a fantastic resource, but it is first and foremost an Encyclopedia, a reference. It has many times helped me refresh my memory or helped explain something I read first in another context. But it is not a good place to learn a brand new subject.

There are many good resources to learn math, most in the form of books (some of which have digital versions). Personally, I find it best to find a subtopic that interests me and dive in. Though I will say that I would recommend "Chapter Zero" to most people wanting to get back into math after a long break.

Yes, Wikipedia has it's advantages as an encyclopedia.

I simply mentioned it here since Steve advocates just diving in and following links on wikipedia as a way to learn a subject.

This might be a great way to find out what stuff is out there to learn or for somebody more familiar with Math to get a synopsis of a topic but as a learning resource for a newcomer to any particular field it seems likely to reduce you to a jibbering wreck very quickly.

Shameless plug, but I try to explain math concepts on my blog. Here's an example for e, which eluded me for a while:

http://betterexplained.com/articles/an-intuitive-guide-to-ex...

My goal is to explain ideas as I wish they were shared with me: informally, with the primary focus on intuition (there are plenty of places to practice the mechanics). Hopefully it can come in useful for you.

I just wanted to say that your explanation for e is excellently done. I found it incredibly helpful - it was exactly the kind of intuitive, step-by-step explanation I was looking for. I hope that you're planning on releasing more content.
Thanks, really glad it was helpful. I'm planning on going back and covering all the math I thought I'd learned in school :).
The best resources for learning math are textbooks. Learning math from pages on the internet is difficult. Mathematicians don't have quite the strong online presence that programmers do so the online literature suffers.

Textbooks are fantastic, though. They are written to be self contained and will start at the beginning (usually with an introduction to fundamental subjects such as set theory). If you're serious about learning math, they are the way to go.

Amen.

Mathematical notation is optimized for symbol manipulation using a pen and paper, not communication of information. Even to us programmers, it's a slew of one-letter variable names that very rarely seem to have the same meaning.

Furthermore, those single-letters are not from a character set that has been stuffed into our heads since we were 3 years old, so there's an amount of translation going on, similar to the process of learning a foreign language (hear spanish -> translation to english -> process thought -> generate response in english -> translate to spanish -> speak spanish vs. hear spanish -> process throughout -> speak spanish).

Bret Victor's project seems to be attempting this a little bit: http://worrydream.com/KillMath/

I really like "Concrete Mathematics". It has nice, interesting problems, a very readable style, and introduces all notation used. It also has well written solutions to all of the problems.

That said, I read the book after I had a math degree, so while it seems very accessible (more so than most undergrad discrete math books), perhaps someone who read it as an undergrad can comment.

I got my BS in engineering (electrical), managed to scrape my way through all the hard math classes, but never really had a firm grasp on any of it. A bit later on in the real world, doing some signal processing work, some of it started to snap together. But then I read the Structure and Interpretation of Computer Programs. Which I found to be more math heavy than any other prob books I'd read previously. But the way everything was describe, and laid out in the context of programming, all my past math just came streaming back, and suddenly made sense. It was like that "a-ha" moment, times 1000.

I don't know what the point of my ramblings are. I guess math finally made sense after learning programming first, much like yegge proposes in his post. But I had the basics down first, and I was at least familiar with the concepts.

I recently decided to start learning maths again. I stopped it after my penultimate year of secondary school because I found it such a joyless subject. I remember friends from the 'advanced maths' class lording it over me saying they were doing 'matrices' (no idea what they were) and 'second order differentiation' (differentiation was hard enough, so that must be way too tough, right?).

I figure i've really only missed out on a couple years of math education compared to the people I know who I consider 'good' at maths. And to my advantage I did do some statistics at university. With all the improved learning materials available to me, plus an alliance with programming, and my improved bullshit-detector for bad teaching and studying practices, it should be a breeze to catch up.

One problem I still have though is that maths just gets so incredibly boring... at least classes do. I used iTunes U and Khan Academy to study calculus and linear algebra. I had to start skipping past some of the really mechanical parts, because as the article said, as a programmer you just think 'put that in a function and never worry about it again.'

Breadth not depth is definitely what I'm after, although I do worry that it's the equivalent of being a musician who knows lots of diverse harmonic theories but still hasn't mastered some scales that would let him/her jam with other musicians.