Ask HN: Best beginner friendly linear algebra book?
Hello all, the title really says it all. Hoping to find a linear algebra book that is friendly for visual learners.
EDIT: thank you all for the great responses!
EDIT: thank you all for the great responses!
128 comments
[ 4.7 ms ] story [ 191 ms ] threadPerhaps a game development book is even more visual? I haven't read it (yet), but this book is getting recommendations: https://gamemath.com/book/
I don't think I've been able to find any particularly good visual LinAlg books - most of what you're trying to achieve is actually quite abstract and I found the classic books a little confusing.
As an addendum - if you live stateside, classes at community colleges may be quite inexpensive and fairly approachable.
The author is also on HN:
https://news.ycombinator.com/user?id=ivan_ah
For everyone who might be interested, here is an extended preview of the book here: https://minireference.com/static/excerpts/noBSLA_v2_preview.... and you can also download the standalone LA concept map here: https://minireference.com/static/conceptmaps/linear_algebra_...
If you prefer video explanations, I've got some of those up on youtube that given an overview of the topics in the book, see https://www.youtube.com/watch?v=2G3PmEZI6n8&list=PLGmu4KtWiH...
I highly recommend getting a print copy because it will help you concentrate on the material and read deeply without distractions. I also have a free-PDF-if-with-purchase-of-print-copy policy, so if you send get the print version and send me an email I'll hook you up with the PDF for free (PDF good for reference and for searching).
Happy to take questions about anything related to the book. Post here or send email. Contacts in profile.
His course at MIT is legendary, completely available online https://ocw.mit.edu/courses/18-06-linear-algebra-spring-2010...
And there’s so much good linear algebra stuff on YouTube from 3brown1blue.
If you can do one thing now, watch this Veritasium video to disprove the myth that you’re a visual learner: https://youtu.be/rhgwIhB58PA.
The point is that there's little evidence behind different people having different learning styles, and that in general everyone is every "style".
This implies that vision, in addition to many other sensory modalities, is useful. As you point out, the utility of of 3b1b is in line with this point.
A lot of the "examples" were "This is complicated and abstract, so we'll just say it is and go to textbook form".
Try the Kahn videos, then watch the 3B1B videos, which are very visual, but somewhat advanced. Or, watch both of them several times in parallel.
Elsewhere, I've discovered that this poster is going into image processing, which is likely "signals and systems" linear algebra.
In signals and systems, your vectors can have infinite dimension, and these infinite-dimension vectors Fourier-transform into other infinite dimension vectors under a new basis.
Any field with more than "3 dimension" vectors / matricies is very difficult to visualize geometrically. Trying to do so is counter-productive to the understanding of the field. This geometric interpretation is really useful in graphics programming / 3d animation however.
---------
Or perhaps a more concrete example... your "visualize the matrix in X dimensions" advice just doesn't cut it if you're dealing with an 8x8 matrix JPEG DCT coefficient matrix (https://en.wikipedia.org/wiki/JPEG#Discrete_cosine_transform), unless you can imagine 8-diemsional space in your brain.
On the other hand, imagining the 8x8 matrix as 64 linearly-independent "Basis" to your 64-dimension discrete signal is... easier. (Well... for a definition of easier at least). And the transform from time domain into Fourier domain is a transformation in basis that contains the same information.
That one and this one are both quite interesting to me - my focus isn’t signals and systems. Thanks!
I was just thinking that the (3D) vector approach would be a good start along the path to mathematical maturity in linear transformations.
https://youtu.be/fNk_zzaMoSs
Higher math tends to be abstract; you can't visualize higher-dimensional linear algebra concepts directly. The standard resources (Strang, Axler, etc) are worth the effort.
Applied learning is the best way to learn linear algebra.
This is highly important. Linear algebra is applicable to so many fields, but learning linear algebra for say... Graphics Programmers, is a completely different feel from learning linear algebra for an Electrical Engineer Signals-and-systems engineer.
Graphics programmers largely need to learn "how to use" matricies. Emphasis on associative properties. Emphasis on non-communitive operations.
In contrast, Electrical Engineers / Signals-and-systems want to learn linear-algebra as a stepping stone to differential calculus. In this case, you're going to be focusing more on eigen-values, spring-mass systems / resonant frequencies, applicability to calculus and other tidbits (how linear algebra relates to the Fourier Transform).
----------
The graphics programmer (probably) doesn't need to learn eigenvalues. So any textbook written as "linear algebra for graphics programmers" can safely skip over that.
The electrical engineer however needs all of this other stuff as "part" of the linear algebra class.
I'm sure other fields (statistics, error-correction codes/galois fields, abstract algebra, etc. etc.) have "their own ways" of teaching linear algebra that is most applicable to them.
Yes, "linear algebra" is broadly applicable. But instead of trying to "learn all of it", you should instead focus on the "bits of linear algebra that is most applicable to the problems you face". That shrinks down the field, increases the "pragmatism" of your studies.
Later, when you're more familiar with "some bits" of linear algebra, you can then take the next step of generalizing off of your "seed knowledge".
--------
I personally never was able to learn linear algebra from a linear algebra book.
Instead, I relearned linear algebra 4 or 5 times as the "basis" of other maths I've learned. I learned it for differential calculus. I relearned linear algebra for signals. I relearned linear algebra for Galois fields/CRC-codes/Reed Solomon. I relearned linear algebra for graphics.
Yes, it seems inefficient, but I think my "focus" isn't strong enough to just study it in the abstract. I needed to see the "applicable" practice to encourage myself to learn. Besides, each time you "relearn" linear algebra, its a lot faster than the last time.
Image filtering _is_ signal processing, two-dimensional signal processing to be precise.
Traditionally, a college would take you through linear algebra -> differential equations -> signals and systems, to approach this subject.
I found it easier to go through the reverse: start at signals-and-systems (to see what you have to learn), then work your way back down to linear algebra, and then work your way back up to signals and systems.
---------------
From a "signals and systems" point of view, your image filtering functions are 99% going to just be a "kernel" applied to an image.
https://en.wikipedia.org/wiki/Kernel_(image_processing)
IMO, its easier to start with a 1-dimensional version, where you perform kernels upon sound and/or RADAR signals rather than 2-dimensional images.
https://en.wikipedia.org/wiki/Convolution#Visual_explanation
You can see that the 1-dimensional version of the convolution applied between (data x kernel) is extremely simple and "obvious" to think about, given this GIF: https://upload.wikimedia.org/wikipedia/commons/6/6a/Convolut...
Where blue-box is the original signal, and red-box is the convolution-kernel, and the black-line is the output of blue convolve with red.
From there, you generalize the 1-dimensional convolution, into a 2-dimensional convolution. To do so, you need to study linear algebra and matricies. But now that you're "focused" upon the convolution idea, as well as the idea of a "kernel", everything should be "more obvious" to you as you go through your studies.
You can see that a "Matrix", in your specific field of study, represents a kernel to a discrete system. The image you want to manipulate is a 2-dimensional signal. A "matrix" is many different things to many different mathematicians / engineers. "Focusing" upon your particular application is key to learning as quickly as possible. (You can generalize later after you've mastered your particular field).
Still, the study of signals / systems is a very generalized and large field. Mechanical engineers study this, because it turns out that an "impulse" that is "convoluted" with a "kernel" is descriptive of how a speed-bump affects your car's suspension system (!!!!). (EDIT: A youtube video demonstrating the same math for earthquakes vs buildings: https://www.youtube.com/watch?v=f1U4SAgy60c)
So studying signals-and-systems is still a very abstract goal of yours. It sounds like you need to focus upon the image-processing portions of signals-and-systems.
---------
IMO, you'll find that there's probably very little linear algebra you actually need to learn for your particular path.
Signals processing, though, is one of the places where I actually think a decent understanding of some of the higher-level concepts in linear algebra is really helpful. Linearity itself comes to mind, and maybe it just reflects my physicist's education, but it's hard for me to imagine having a working understanding of Fourier transforms without getting the idea of changing bases. I feel like you're about halfway through a first linear algebra course before you'd get there.
EDIT: that said, a good signals processing book probably covers a lot of this in sufficient depth if you can figure it out. The other catch-all comment I'd make is that linear algebra from a math class can look somewhat different from practical linear algebra on a computer. (That it's often a bad idea to directly invert a matrix for many computing applications is non-obvious from math class.) A book like Trefethen and Bau is great on that latter subject but is not a good starting point for OP.
> Instead, I relearned linear algebra 4 or 5 times as the "basis" of other maths I've learned. I learned it for differential calculus. I relearned linear algebra for signals. I relearned linear algebra for Galois fields/CRC-codes/Reed Solomon. I relearned linear algebra for graphics.
If I were way better at websites and at advanced mathematics than I actually am, I'd make a site for learning math in a top down manner where you start with some result or application that interests you and then are taught just enough more elementary math to support that result or application.
The site would have a list of results and applications, and for each tell what math is necessary to understand it. You pick a result or application that interests you, either because it is interesting to you itself or because you see that it depends on some more elementary math that you wish to learn.
Once you pick, the site would show you a proof of the result or development of the application, at a level that one would find in a journal aimed at professionals in the relevant field. This of course will most likely be largely incomprehensible at this point.
You can select any part of the proof or development and ask the site for more information. There are two kinds of additional information you can ask for.
One is to ask for smaller steps. You use this when there is some step A -> B where you are comfortable with A and B but just don't see how it jumps from A to B. You understand what A means, what B means, just not why A -> B. The site fills in the intermediate steps.
The other is to ask what something means. This is for when the proof uses something you have not yet studies. For example if the proof uses integration and you have not yet studied it calculus that would be a great place to use a "what does this mean?" request. The site would then give you a short explanation of integration.
A key feature of the site would be that this is all recursive. If you use a "what does this mean?" request on an integral and get the short explanation of integration, you could use "smaller steps" requests and "what does this mean?" requests in that explanation.
Using "what does this mean?" requests recursively should let you go all the way down to things that can be explained with only high school algebra and precalculus.
Note that if you've never studied anything past high school algebra and precalculus and then use the site to learn something like say an analytic proof of the prime number theorem you will learn much elementary calculus but not all. You will learn just what is needed for the prime number theorem.
But there would be other interesting theorems and applications that use different parts of elementary calculus, so doing those would fill in more of your elementary calculus.
The site should have a planner that lets you pick areas of undergraduate or masters level math that you would like to learn and then shows you lists of interesting theorems and applications it has that will cover those areas.
I think this would be an interesting and effective way to learn. At all points everything you are learning goes directly toward supporting the top level proof you have chosen to learn, and you have an idea of why it is useful because you are there because you've already encountered something where you need it.
I think that for many people this will provide better motivation. In the conventional approach, where you do say a whole class in calculus or abstract algebra, then do a more advanced class that uses those results, and so on, a lot of time you are learning stuff with no idea of why it is useful.
For more and deeper, see the other recommendations here.
I don't remember how I found this guy but watching him feels more like learning from a friend who's extremely knowledgeable about linear algebra rather than sitting in a university course.
Like others here I recommend 3B1B, which may be what you are looking for visually, but whatever you end up with it is absolutely crucial that you do exercises. Do many of them. It is the only way to get better.
It's definitely not the norm compared to many of the other listings in this thread but it definitely gave me a better understanding of many algebraic properties and helped build an intuition around spaces, vectors, products, etc.
It doesn't have a ton of graphics, to which you might snub your nose at it (you mentioned visual learning), but the graphics it does have are incredibly useful for building a geometric understanding of what linear algebra concepts map to. The subsection on quaternions and pseudoscalars is one of the best descriptions of such in my experience.
Well, here's a comment from elsewhere in the thread:
> If you can do one thing now, watch this Veritasium video to disprove the myth that you’re a visual learner: https://youtu.be/rhgwIhB58PA.
( https://news.ycombinator.com/item?id=31707314 )
I haven't watched the video, but, like your parent comment, I was already aware that "learning styles" was a research area supported almost exclusively by fraud. If you want more links, you can find them pretty easily through https://en.wikipedia.org/wiki/Learning_styles#Criticism .
Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological science in the public interest, 9(3), 105-119. — https://ve42.co/Pashler2008
Willingham, D. T., Hughes, E. M., & Dobolyi, D. G. (2015). The scientific status of learning styles theories. Teaching of Psychology, 42(3), 266-271. — https://ve42.co/Willingham
Massa, L. J., & Mayer, R. E. (2006). Testing the ATI hypothesis: Should multimedia instruction accommodate verbalizer-visualizer cognitive style?. Learning and Individual Differences, 16(4), 321-335. — https://ve42.co/Massa2006
Riener, C., & Willingham, D. (2010). The myth of learning styles. Change: The magazine of higher learning, 42(5), 32-35.— https://ve42.co/Riener2010
Husmann, P. R., & O'Loughlin, V. D. (2019). Another nail in the coffin for learning styles? Disparities among undergraduate anatomy students’ study strategies, class performance, and reported VARK learning styles. Anatomical sciences education, 12(1), 6-19. — https://ve42.co/Husmann2019
Snider, V. E., & Roehl, R. (2007). Teachers’ beliefs about pedagogy and related issues. Psychology in the Schools, 44, 873–886. doi:10.1002/pits.20272 — https://ve42.co/Snider2007
Fleming, N., & Baume, D. (2006). Learning Styles Again: VARKing up the right tree!. Educational developments, 7(4), 4. — https://ve42.co/Fleming2006
Rogowsky, B. A., Calhoun, B. M., & Tallal, P. (2015). Matching learning style to instructional method: Effects on comprehension. Journal of educational psychology, 107(1), 64. — https://ve42.co/Rogowskyetal
Coffield, Frank; Moseley, David; Hall, Elaine; Ecclestone, Kathryn (2004). — https://ve42.co/Coffield2004
Furey, W. (2020). THE STUBBORN MYTH OF LEARNING STYLES. Education Next, 20(3), 8-13. — https://ve42.co/Furey2020
Dunn, R., Beaudry, J. S., & Klavas, A. (2002). Survey of research on learning styles. California Journal of Science Education II (2). — https://ve42.co/Dunn2002
[0] The Biggest Myth In Education https://www.youtube.com/watch?v=rhgwIhB58PA
https://sciencebasedmedicine.org/brain-based-learning-myth-v...
For example, if some particular method is computationally efficient relative to others, the text makes a note of it, and has lots of computational examples. Most of the examples could be set up fairly straightforwardly with something like a Python notebook and Numpy for matrices. It also covers things like computational errors wrt floating-point operations when doing vector and matrix calculations, efficient algorithms for approximating eigenvalues of a matrix, etc.
And!, the full text is available on archive.org with a free account:
https://archive.org/details/linearalgebramod0000pool
However, there's no magic bullet that will let you learn linear algebra in a couple of hours. At some point you have to sit down and work to figure it out. The field has university departments researching it, so there's a lot more to it than just multiplying m×p by p×n matrices.
I don't know what you mean exactly by beginner, but assuming you have some level of mathematical maturity, UT Austin has an edx course you can audit for free, "Linear Algebra Foundations to Frontiers", and FastAI also have a pretty good free video series/course on it too.
I think it's great for an intermediate student, or someone who's also watching Strang's lectures.
For the graphical part, start with, say, (3,7). Regard that as the coordinates in the standard X-Y coordinate system of a point on the plane. So, the X coordinate is the 3 and the Y coordinate is the 7. You could get out some graph paper and plot the thing. So, more generally, given two numbers x and y, (x,y) is the coordinates of a point in the plane. We call (x,y) a vector and imagine that it is an arrow from the origin to an arrow head at point (x,y). Then we can imagine that we can slide the vector around on the plane, keeping its length and direction the same.
What we did for the plane and X-Y we could do for space with X-Y-Z. So, there a vector would have three coordinates. Ah, call them components.
Now in linear algebra, for a positive integer n, we could have a vector with n components. For geometric intuition, what we saw in X-Y or X-Y-Z is usually enough.
We can let R denote the set of real numbers. Then R^n denotes the set of all of the vectors with n components that are real numbers. Our R^n is the leading example of a vector space. Sometimes it is good to permit the components to be complex numbers, but that is a little advanced. And the components could be elements of some goofy finite field from abstract algebra, but that also is a bit advanced.
Here we just stay with the real numbers, elements of R.
Okay, suppose someone tells us
ax + by = s
cx + dy = t
where the a, b, c, d, s, t are real numbers.
We want to know what the x and y are. Right, we can think of the vector (x,y). Without too much work we can show that, depending on the coefficients a, ..., t, the set of all (x,y), that fits the two equations has none, one, or infinitely many (points, vectors) solutions.
So, that example is two equations in two unknowns, x and y. Well, for positive integers m and n, we could have m equations in n unknowns. Still, there are none, one, or infinitely many solutions.
C. F. Gauss gave us Gauss elimination that lets us know if none, one, or infinitely many, find the one, or generate as many as we wish of the infinite.
We can multiply one of the equations by a number and add the resulting equation to one of the other equations. We just did an elementary row operation, and you can convince yourself that the set of all solutions remains the same. So, Gauss elimination is to pick elementary row operations that make the pattern of coefficients have a lot of zeros so that we can by inspection read off the none, one, or infinitely many. Gauss elimination is not difficult or tricky and programs easily in C, Fortran, ....
Quite generally in math, if we have a function f, some numbers a and b, and some things, of high generality, e.g., our vectors, and it is true that for any a, b and things x and y
f(ax + by) = af(x) + bf(y)
then we say that function f is linear. Now you know why our subject is called linear algebra.
A case of a linear function is Schroedinger's equation in quantum mechanics, and linear algebra can be a good first step into some of the math of quantum mechanics.
Let's see why those equations were linear: Let
f(x,y) = ax + by
Then
f[ c(x,y) + d(u,v)]
= f[ (cx, cy) + (du,dv) ]
= f(cx + du, cy + dv)
= a(cx + du) + b(cy + dv)
= c(ax) + d(au) + c(by) + d(bv)
= c(ax + by) + d(au + bv)
= cf(x,y) + df(u,v)
Done!
This linearity is mostly what makes linear algebra get its mathematical theorems and its utility in applications.
We commonly regard the plane with coordinates X-Y as 2 dimensional and space with coordinates X-Y-Z as 3 dimensional. If we study dimension carefully, then the 2 and 3 are correct. Similarly R^n is n dimensional.
We can write
ax + by = s
cx + dy = t
as x(a,c) + y(b,d) = (s,t)
So, (a,c), (b,d), and (s,t) are vectors, and x and y are coefficients that let us write vector (s,t) as a linear combination of the two vectors (a,c) and (b,d).
Apparentl...
But I included enough that you could extend to some relatively deep parts!
But among what I omitted was determinants. So, I couldn't mention your points!
To me the one easy approach to determinants is to say that they give a volume, and the volume is zero if and only if there is linear dependence.
https://news.ycombinator.com/item?id=24892907