Think of those graphs as being something like the rubber sheet analogy that's often used to explain general relativity - it's not a great analogy in all respects, but humans can't really visualize 4 dimensions and it captures the gist of the idea well enough. Machine learning models have a similar problem: In reality the data points in k-means clustering exist in some very high-dimensinal space that's impossible to draw. So for the sake of visual aids it's common to draw 2D plots where the only thing they're meant to suggest is that points that appear close together in the visual aid would also be fairly close to each other in the actual problem space.
They don't mean anything in particular. The actual analysis is being done in a high-dimensional space, in which each post is represented by a high-dimensional vector of the form [0,0,1,0,...., 0,1,0]. The length of the vector is the total number of distinct words used across all blog posts (maybe something like 30,000), and each entry is either 0 or 1 depending on whether the corresponding word occurs in this post. All the distances and cluster centers are actually being computed in this 30000-dimensional space; the two-dimensional visualization is just for intuition.
If you're wondering how the author came up with the two-dimensional representation, the article doesn't say, but it's likely he used something like Principal Component Analysis (http://en.wikipedia.org/wiki/Principal_component_analysis). This is a standard technique for dimensionality reduction, meaning that it finds the "best" two-dimensional representation of the original 30,000-dimensional points, where "best" in this case means something like "preserves distances", so that points that were nearby in the original space are still relatively close in the low-dimensional representation.
The graphs are just illustrations. The actual graphs would be high-dimensional (one dimension or axis for each word in the total word set), and each axis would have only two ticks on it: 0 and 1. A post is mapped to the binary vector in this space whose coordinates are determined by the presence or absence of words in the post.
The point of the graphs are as an example more than anything — in any real world solution it's not trivial to plot your documents (as representing documents with thousands of words in 1d, 2d or 3d is not trivial).
But if _you could_ represent your documents in 2d, you could plot them like I do here. And, the X-axis would represent 1 feature of your documents, and the Y-axis another.
Tl;dr don't focus on the axes — focus on the idea of being able to compare the relative distances between documents.
If we have x, y, z, i, j, and n then we'd have six dimensions.
If each word represents a dimension, any blog title with that word gets a 1 on that word dimension, or a 0 if does not.
You can then calculate the distance between these points considering all of the dimensions, although clearly it's going to be a little limited on binary inputs, since the square of the differences is always going to be 1 or 0. Like for 4 dimensions, sqrt( ( 0 - 1 )^2 + ( 0 - 0 )^2 + ( 0 - 0 )^2 + ( 1 - 0 )^2 ), which just always gets simplified to something like sqrt( 1 + 0 + 0 + 1 ), which is a little boring.
The binary values cause your data points to all be stacked directly on top of each other, which leads me to believe that using binary inputs is a less than ideal application for k-means. Just look at it in the 2d case, where you have either [0,1], [0,0], [1,0], or [1,1] for each data point. Not very hard to determine the clustering there... basically just doing an overly complex boolean expression.
Also: aerodynamics is not really hard, anyone can fold paper planes!
Or: programming 3D games is easy, just build new levels for an old game!
Or: I don't know what I am doing here, but look, this photoshop effect looks really cool on my holiday photos!
etc.
Seriously:
The writer would not be able to write anything about K-Means if not for people looking at it from a mathematical view point. This angle is of tremendous importance if you want to know how your algorithm behaves in corner cases.
This does not suffice, if you have an actual application (e.g. a recommendation or a hand tracking or an object recognition engine). These need to work as good as you can make it because every improvement of it will result in $$$.
Author is not proposing "1 weird machine learning trick mathematicians HATE!"
They are encouraging non-maths people to give the basics a try even though it seems intimidating. And it worked on me, like some others in this thread my maths is shaky beyond diff calculus and my eyes glaze over when I see notation - but now I'd like to give this a whirl. I have no illusions that I will suddenly be an expert.
I feel I'm in a somewhat unique position to talk about easy/hardness of machine learning; I've been working for several months on a project with a machine learning aspect with a well-cited, respected scientist in the field. But I effectively "can't do" machine learning myself. I'm a primarily 'self-trained' hacker; started programming by writing 'proggies' for AOL in middle school in like 1996.
My math starts getting pretty shaky around Calculus; vector calculus is beyond me.
I did about half the machine learning class from coursera, andrew ng's. Machine learning is conceptually much simpler than one would guess; both gradient descent and the shallow-neural network type; and in fact it is actually pretty simple to get basic things to work.
I agree with the author that the notation, etc, can be quite intimidating vs what is "really going on".
however, applied machine learning is still friggin' hard; at least to me; and I consider myself a pretty decent programmer. Naive solutions are just unusable in almost any real application, and the author's use of loops and maps are great for teaching machine learning; but everything needs to be transformed to higher level vector/matrix problems in order to be genuinely useful.
That isn't unattainable by any means; but the fact remains (imho) that without the strong base in vector calculus and idiosyncratic techniques for transforming these problems into more efficient means of computations; usable machine learning is far from "easy".
I was going to say something around those lines but you save me the effort.
Although in my opinion using ML and have some decent results is not that hard. Use it and being close to the state in the art (i.e improving your recognition rate from 65% to 80%) is a completely different matter and it is hard and it requires a fully understanding of the details of the algorithm in use.
Still it doesn't meant that you can't do interesting things with an overall understanding of machine learning, but making a good cat detector is hard [0] [1]
For a 65% to 80% jump, I'd say it depends on the subfield. With NLP (which is basically applied machine learning), it's not uncommon to see algorithms with >80% accuracy.
Mathematical symbols are overloaded by context, this could be done but you'd still need to tell it whether it was a formula from "machine learning" or "topology".
Then you'd still fail to account for symbols invented/repurposed in a single paper or by a single author.
Basically, the tool itself is a machine learning problem.
Even the simple case of taking a PDF, finding the first use of any given symbol in that paper, and turning the PDF into an interactive document where hovering on any use of the symbol shows a tooltip showing the first use - now that would be both useful and feasible.
But... the Venn diagram of people able to do such a thing, people motivated to do such a thing (i.e. people reading academic papers who aren't math whizzes), and people with the time to do such a task... seems not to have a sufficient intersection. Most of us in this thread fulfill categories 1 and 2, but not 3... come to think of it, I have work to do... ducks.
A better approach might be translating them into more uniform s-expressions. Mathematica does this. You can also check out a similar approach in Structure and Interpretation of Classical Mechanics where tensor expressions are reduced to scheme expressions.
This sounds to me like a pointless exercise. There is a reason for using mathematical notation for non trivial formulas, which is that is more compact and succint, to allow it to convey information efficiently and unambiguously. Think of a formula with a few levels of parentheses; you're not going te be able to express that clearly in a paragraph of text. It's not so much the symbols and notation itself which is hard to grasp, but the mental model of the problem space; once you have that, the formula will usually make sense because you can relate it to this mental model.
>There is a reason for using mathematical notation for non trivial formulas, which is that is more compact and succint, to allow it to convey information efficiently and unambiguously.
Maybe, but not always. Remember that Richard Feynman took great issue with how integration was taught in most math classes and devised his own method (inspired from the Calculus for the Practical Man texts).
You can always try to find an even better notation, but the only point I was making is that for certain cases anything is better than a wall of akward text.
No. Feynman never took issue with integration notation or how integration is defined or taught. The story you're referring to is how he learned of a technique for computing integrals that was not covered in schools. The technique is called "differentiation under the integral", and is arguably even more involved.
I agree to some extent. However, I do think that there is a level of "ramping up" that requires a good bit of teaching and understanding.
It sort of is the members only cigar club of academics. While I can read, understand, and subsequently implement most things in computer science land, when it comes to mathematical notation, there is a lot left to understand.
I think a LOT of people would benefit from a five or six video course simply showing a translation of complex notations to a working algorithm in a popular language, so the commonly used symbols start having meaning.
Not in the context of a length limited conference paper. An implementation adds details for memory allocation and use of data structures &c. This is a distraction in the exposition of an algorithm. It's useful if authors make a well commented implementation available, but the actual paper should contain an abstract definition of the algorithm.
Many mathematical concepts cannot be translated into programming, since computation is by definition discrete. For example, even a simple concept like irrational numbers cannot be completely captured by code.
Mathematical equations don't have any meaning taken by themselves; they just express a certain relationship between quantities without stating anything specific about what those quantities are.
Any translation of an equation to English would be a certain (usually lossy) interpretation of the equation for some particular quantities. But the same equation can have multiple interpretations in wildly different contexts.
Striking example is quantum mechanics. Physicists in the first half of the 20th century figured out a mathematical model (e.g. the Shroedinger equation) that worked perfectly well for predicting the behaviour of fundamental particles. Interpreting what the equation actually means (let's say in English) proved to be hard though and spawned multiple, sometimes conflicting or paradoxical interpretations. In fact, to this very day physicists are split on their favorite interpretation of QM. On the positive side, the mathematics of QM just works no matter how the result or the calculation are being interpreted.
Yes. If the symbols are in MathML there are accessibility tools which will read them out to you - effectively translating them into plain spoken english.
What you're suggesting is converting a universal, concise, formal language into the potentially ambiguous and/or redundant "plain English". What's the point of that? One has to learn some math before being able to read math.
That sounds good but what I think you're really asking for is an explanation of each of the symbols in their given context and an introduction to the mathematical "jargon" and idioms of the equation's branch of mathematics.
Such things do exist. Unfortunately, they are called "textbooks".
I agree that the notation is tough. Especially for folks who need to learn to do this stuff as a one-off and not as a career. That's why I wrote a book that teaches the algorithms with nearly no math notation while trying to be very rigorous:
http://www.amazon.com/Data-Smart-Science-Transform-Informati...
This response, and the replies to it actually come as a bit of a surprise to me. I am a graduate student in artificial intelligence who has a comparatively weak background in mathematics who is still very capable of applying, implementing, and improving upon machine learning techniques. I have never taken a calculus or linear algebra class. (Don't ask how I managed that, an effect of the odd route I took to get where I am). The point is you shouldn't be telling yourself you can't do it because you don't know the math, you may surprise yourself.
That being said there are plenty of barriers to a software engineer picking up some papers on machine learning and getting started. These are the result of some serious issues that I believe exist with academia at present and specifically the "publication game". I will not however get into this as this comment will quickly turn into the longest paper I've ever written. It is a problem with the system, and as the reader it is not your fault.
Your issue with applied machine learning may be largely imagined. Take for example the k-means clustering algorithm in the article. This is quite useful in a number of places be it data analysis or intelligent user interfaces, its straightforward to implement, and many usable implementations exist online. The same case is true for other models such as linear regression, naive bayes, even things like support vector machines are pretty easy to use these days.
In any case, as selfish as this is, this sentiment you're expressing provides some comfort that I'll be in demand when I enter the job market. :P Though if you really want to implement and apply some simple yet useful algorithms he recommends Programming Collective Intelligence at the end of the article and I can definitely second that recommendation.
I think you're glossing over the fact that for a non-trivial problem, you have the pick the right machine learning algorithm from a large set of possibilies, and typically you have to set a number of parameters to make it work. On top of that you have to follow proper methodology to get meaningful results; e.g., to determine whether the accuracy you're getting generalizes to other data as well. I think all of this contributes to making applied machine learning a non-trivial skill. And indeed, AI is just the right background for that, but the point is that you need such a background.
The combinatorial number of possibilities quickly grows beyond what can be tested automatically, so you have to start from good intuitions. That's actually what machine learning is _not_ great at; it finds patterns in data (maybe even previously unknown ones), but arguably doesn't come up with creative insights.
Thats true, though as my optimization friends put it, sometimes a really close answer that is easy to come to is often better than a precise answer which is difficult to come to. Selecting the right algorithm is a bit of an art though some simple rules can make it pretty easy. Say clustering for groups, linear regression for estimating continuous values, naive Bayes / logistic regression for classification. Without parameter tuning you can go from having a bunch of data to having a model that does something quite easily.
When were talking about applied machine learning it is generally in the context of developing a product, a feature. The most important part is going from data to feature, not necessarily squeezing the last few percentage points of performance. In this context getting a good answer easily is more important than getting the best answer.
Just wanted to say thanks for the recommendation! I actually started writing the book because I felt that a lot of the existing textbooks were unnecessarily obtuse. The details are complicated but you can still get some interesting results and have some fun with a high-level understanding.
(I'll warn you that a lot about the book is very dated now)
First off thanks for writing the book. It was the first book recommended to me for ML while I was doing my undergrad and it got me using ML as opposed to studying ML. The book opened my eyes to the fact that I really could do these things, and it definitely played a role in influencing me to pursue my MSc in AI. So again, thank you. :)
Secondly I'll agree that the book is most definitely dated at this point though it provides a survey of quite a broad selection of algorithms. It certainly isn't going to put the reader on the cutting edge or anywhere near it but it will get them started which appears to be the hardest part for some of those participating in this comment thread.
Good point. However, the author does recommend using open source ML libraries, so applying the techniques may not require some of the implementation knowledge you suggest. It's likely more relevant to understand the semantic constraints and pitfalls (eg, am I over-fitting right now?) since the work of a small number of talented implementers can be easily reused.
I feel the exact opposite. ML is a huge part of my job (or was), and I topped out at Discrete Math 2 or so, finishing up a few courses of the usual Calculus but never anything as esoteric as Real Analysis or the like. (I think I finished up some Linear Algebra.)
Anyway, my math is not that great and I'm way out of practice when it comes to theoretical work. Not only that, I suck at algorithm design and I'm at best a moderately decent developer from a theory standpoint.
But applied machine learning came natural, because I have a strong grasp of frequentist and Bayesian statistics as well as solving the problems backwards. If you try to apply ML or any other "exotic" solution by starting with "oh man here is this awesome technique," you will definitely be frustrated; I know that feeling all too well.
ML is just another tool in the shed. With practice, you start to understand where simple linear regression techniques are useful and when more heavy-duty methods are applicable. Keep solving real world problems rather than trying to look for stuff to hit with your new hammer (ML in this case) and I bet you'll come around a lot quicker.
My take on this is somewhat different - my background is pure mathematics in my undergraduate degree, then applied machine learning at Facebook (ads optimization) and GS (algo trading), and currently at graduate school in ML.
My take is that 'applying' machine learning is much easier than one would think, but 'understanding' machine learning is _much_ harder than one would think.
I imagine most people start with an application - that's why they're interested in the first place. Applying an ML algorithm to a given dataset is trivial - only a few lines in R/Python with scikits-learn, etc. There's even a handy cheat-sheet [1] to tell you which algorithm to use. Your regression model spits out a few parameters, some estimates of quality of fit, maybe some pretty graphs, and you can call it a day.
The next stage is implementation - up until now, the black box has been spitting out parameters, and you want to understand what these numbers actually are and how they are determined. This means getting your hands dirty with the nitty-gritty of ML algorithms - understanding the algorithms statistical basis, understanding the computational performance of the algorithms (as you alluded to), and reading textbooks/journal articles. Many of these algorithms have a simple 'kernel' of an idea that can be obscured by a lot of mathematical sophistication - For SVM, the idea of finding a plane that separates your data as much as possible, logistic regression is just finding the line of best fit in logit space, k-means is just of repeatedly assigning points to clusters then updating clusters, etc.
It's not _that_ difficult to get working implementations of these algorithms (although debugging an incorrect implementation can be frustrating). You can implement simple versions of gradient boosting machines - a theoretically sophisticated technique that often works very well in practice - in around 100 lines of Python [2] or Go [3]. You can get implement Hopfield networks (a basic neural network for unsupervised learning) in Julia [4] and Haskell [5] without much trouble either.
I think where a lot of the value from spending 3-5+ years in graduate school comes in when you an off-the-shelf technique isn't cutting it - maybe you're overfitting, you want to play with hyperparameters, your datasets are scaling faster than the off-the-shelf implementations can deal with them, etc. Solving these kind of problems is where a lot of the mathematical/statistical sophistication can come into play - Bayesian hyperparameter optimization of existing ML techniques improved on state-of-the-art in image processing [6], understanding _why_ the LASSO works (and understanding when it doesn't) can mean having to understand the distribution of eigenvalues of random Gaussian matrices [7]. That's saying nothing of figuring out how to take an existing algorithm and scaling it out in parallel across 1000+ machines [8], which can mean using some deep convex optimization theory [9] to justify your approximations and prove that the approximated model you learn is well-behaved, etc.
I suspect that some of the gulf between ML in practice and theory arises from the differing goals in the two parties. If you just want an parameter estimation, then a lot of the rigour will seem useless, and if you want to understand a procedure deeply, you can't do away with rigour. Take the k-means paper used in the article. In the screenshot, the paper's author is presenting results of the flavour
> Under weak conditions on the data, as the number of iterations increases, regardless of the random choices made, the k-means clustering tends towards a good clustering of the data.
Depending on your viewpoint, this result is either obvious, non-obvious, or irrelevant. To prove this needs (at a glance) theorems from topology, measure theory, martingale convergence, etc.
I suppose some people find that fascinating, and want to understand _why_ this is true, and what 'good&...
totally agree with you. i'm usually on the applications side of things, and you benefit enormously if you can understand the math behind an algorithm (it's also pretty much required for implementing most papers). that being said, actually writing a new paper is a completely different beast. in fact, many people who can write machine learning papers aren't going to be that good at actually implementing a useful solution without lots of engineering resources backing them up. there are hax, then there are math skills. overlap is rare.
It's easy until you have to start adjusting parameters, understand the results meaningfully, and tune the algorithms for actual "Bit Data". Try doing most statistical analysis with dense matrices and watch your app go out of memory in two seconds.
It's great that we can stand on the shoulders of giants, but having a certain understanding of what these algorithms are doing is critical for choosing them and the parameters in question.
Also, K-means is relatively easy to understand untuitively. Try doing that with Latent Dirichlet Allocation, Pachinko Allocation, etc. Even Principal Component Analysis and Linear Least Squares have some nontrivial properties that need to be understood.
Upvoted this for an interesting read, but I agree with the sentiments in the comments that (1) ML is in general hard (2) some parts of ML are not that hard, but are likely the minority (3) we are standing on the shoulders of giants, who did the hard work.
Of the two methods described - search vs. clustering - the first one - simpler and not involving ML - is better for this use case. The only reason it seems to give worst results is because it's only used with the titles and not the full body (unlike the clustering approach). So I guess machine learning is easier to mis-use than it looks...
I disagree with this article, although I did find it interesting. Replace k-means with a supervised learning algorithm like an SVM, and use some more complicated features other than binary and this article would be a lot different.
Also - maybe "article recommendation" is "easy" in this context, but other areas such as computer vision, sentiment analysis are not. Some other questions I might ask
How do you know how well this algorithm is performing?
How are you going to compare this model to other models? Which metrics will you use? What statistical tests would you use and why?
What assumptions are you making here ? How do you know you can make them and why?
There are a lot of things that this article fails to address.
Disclaimer: I realize more complex models + features don't always lead to better performance, but you need to know how to verify that to be sure.
SVM prediction is pretty straight-forward, even if you're using a kernel/dual formulation:
"Here are some training points from classes A and B, each with an importance weight. Compute the weighted average of your new point's similarity to all the training exemplars. If the total comes out positive then it's from class A, otherwise it's from class B."
If you tried to explain training from a quadratic programming perspective it would get messy, but if you use stochastic gradient descent then even the training algorithm (for primal linear SVM) is pretty light on math.
Of course, this is only possible after a decade or two of researchers have played with an digested the ideas for you. I think until Léon Bottou's SGD SVM paper, even cursory explanations of SVMs seemed to get derailed into convex duality, quadratic programming solvers, and reproducing kernel hilbert spaces.
As far as I know, k-means (as the author described it) has two parameters: initial number of clusters K and max iterations (although I could be wrong), SVMS have:
I think there's a big difference between explaining the essence of an algorithm and understanding all the details and decisions that go into a particular implementation.
If explaining SVM, I'd stick with the linear primal formulation, which only really requires choosing the slack parameter C. If I needed to perform or explain non-linear prediction, I'd switch to the RBF kernel, which gives you one additional parameter (the RBF variance).
fair enough - btw i just realized i bookmarked your blog (http://blog.explainmydata.com) a few weeks back - i was really enjoying those posts. nice work
tldr: the ML algorithms look hard reading the papers, while the code looks simpler and shorter, also you can get pretty decent results in a few lines of R/Python/Ruby so ML is not that complex.
I disagree in so many ways:
1. complex algorithms are usually very short in practice (e.g. dijkstra's shortest path or edit distance are the firsts that come to mind)
2. ML is not just applying ML algorithms: you have to evaluate your results, experiment with features, visualize data, think about what you can exploit and discover patterns that can improve your models.
3. If you know the properties of the algorithms you are using then you can have some insights that might help you on improving your results drastically. It's very easy to apply the right algorithms with the wrong normalizations and still get decent results in some tests.
Most machine learning concepts are very simple. I agree with the author that mathematical formulae can be unnecessarily confusing in many cases. A lot of the concepts can be expressed very clearly in code or plain english.
For example, a matrix factorization could be explained with two arrays, a and b, that represent objects in the prediction:
for each example
for each weight w
prediction += a[w] x b[w]
err = (prediction - actual_value)
for each weight w
a[w] += err x small_nuumber
b[w] += err x small_number
It's that simple. Multiply the weights of a by the weights of b, calculate error and adjust weights, repeat.
K-Nearest Neighbor/KMeans are based on an even simpler operation:
dist = 0
for each weight w: dist += (a[w] - b[w])**2
Then make predictions/build clusters based on the smallest aggregate distance.
There are more advanced concepts. There are some serious mathematics involved in some predictors. But the most basic elements of statistical prediction are dead simple for a trained programmer to understand. Given enough data, 80% solutions can easily be achieved with simple tools.
We should be spreading the word about the simplicity of fundamental prediction algorithms, not telling people that it's hard and a lot of math background is required. Machine learning is very powerful and can improve all of our lives, but only if there is enough data available. Since information tends to be unevenly distributed we need to get the tools into the hands of as many people as possible. It would be much better to focus on the concepts that everyone can understand instead of keeping statistics secrets behind the ivy clad walls of academia.
Adding to that, a lot of methods that fall under the "machine learning" label are concepts that are pretty well understood in statistical point estimation, testing of hypotheses, and numerical analysis. I'm not 100% that it really ought to be treated as a new discipline.
Spot on. I agree completely. I work at this stuff for a living, and it never ceases to amaze how the academic literature and math papers are discussing fundamentally simple ideas using styles and techniques programming evolved past in the last few decades. Comments, structure, not-greek-characters (but they make me look smart!), abstractions. When was the last time you saw a stat/math paper in machine learning with comments/structure? And guess what, its as undecipherable as code without comments/structure.
On the other hand, I'm also learning that what I/others find easy, a great deal of people/programmers find hard. The number of programmers/hackers who can actually implement such techniques on new real-world problems if they don't have someone holding their hand I'm discovering is a very small minority. So maybe its harder than it looks after all, and we just think its easy because we've spent so much time with it? After all, programming isn't a walk in the park for most people, and machine learning isn't a walk in the park for most programmers.
> When was the last time you saw a stat/math paper in machine learning with comments/structure? And guess what, its as undecipherable as code without comments/structure.
What do you mean? Academic papers are usually written in plain english with mathematical formulas when it's necessary. What kind of comments would you like to see?
Well, to quote the article, you can see things like "p is absolutely continuous with respect to the Lebesque measure on En" and hundreds of variable names and sub/superscripts, none of which are intuitively named. It's really hard to argue that anyone who is not in academia would understand this without passing through it multiple times.
That being said, I think mathematicians should be perfectly permitted to do this, seeing as most people who read their papers are themselves, mathematicians. Thus spelling out every dumb detail would probably just be a waste of their time for the one case that the brave soul who is a programmer tries to decipher it.
Like another comment above alluded to, mathematicians tend to have parallel gripes about code. While technically we're speaking the same language, the dialect is often quite different.
A lot of the difficulty in mathematics is trying to come up with the correct, precise definitions for "fundamentally simple" ideas (the French are historically well known for this). There is literally no way to get the precision afforded by mathematics without requiring substantial jargon, it would be like trying to write an entire operating system without functions. Mere intuition doesn't cut it here, we want logically airtight proofs, and for this you need exact definitions and terminology.
For example, everyone intuitively knows what holes are.
Well, how do you rigorously define a hole? It took centuries
to come up with the correct definition, which is given in terms of homotopy groups, and the definition of homotopy groups will look incomprehensible to the non-mathematician.
Seemingly "easy" statements like "bounded three dimensional objects without holes are spheres" (i.e. the Poincare conjecture) turn out to be very, very hard (worth a million dollars and a Fields medal!)
Papers are not just blocks of equations. There is english giving context. There is also the issue of length limits for a paper. It's assumed that the reader has a background in the area and so the authors can be a bit more concise.
> structure
Most papers in math follow a common structure of definition-theorem-proof. You can think of lemmas as subroutines that are 'called' in a proof.
>not-greek-characters (but they make me look smart!)
Unless you're advocating spelling out things (i.e. calling something "decay constant" instead of using an alpha), I don't think there is a fundamental difference between using a Greek or English alphabet. Greek is nice because it gives you more characters to use beyond the English alphabet (which is already used extensively in math). If you _are_ advocating replacing single symbols with words, I think that would be pretty hard to parse, and rather cumbersome in longer formulas.
>abstractions
Math is _all about_ abstraction. I don't know what you mean by this.
I like the simple explanation of K-Means, and I like the contrast with the dense set-theoretic language -- a prime example of "mathematosis" as W.V.O. Quine put it.
I admit that was an unfair swipe at formalism, but I do think that formalism is far more effective in the light of a clear, intuitive exposition that conveys the concept straight into the mind. Formalism is good for battening things down tightly, after the fact.
On Kaggle
"The top 21 performers all have an M.S. or higher: 9 have Ph.D.s and several have multiple degrees (including one member who has two Ph.D.s)."
While this may be true that doesn't by any means imply that someone with "only" a B.Sc. in Computer Science can't understand and use machine learning. In fact the only thing this implies is that those who have focused the most in the field are performing the best in competitions. Fairly standard wouldn't you say? The result shouldn't discourage anyone.
I've done pretty well at Kaggle with just a Bachelors in American History: http://www.kaggle.com/users/19518/vik-paruchuri, although I haven't been using it as much lately.
A lot of competition winners have lacked advanced degrees. What I like most about Kaggle is that the only thing that matters is what you can show. I learned a lot, and I highly recommend it to anyone starting out in machine learning. I expanded on some of this stuff in a Quora answer if you are interested: http://www.quora.com/Kaggle/Does-everyone-have-the-ability-t....
A lot of concepts are easier when you know how they work.
CPUs were magical for me before I took a computer architecture course. So was AI and machine learning. Once you see the "trick" so to speak you lose some of the initial awe.
It's actually incredibly hard, especially if you want to achieve better results than with a current 'gold standard' technique/algorithm, applied on your particular problem.
While the article doesn't have this title (why would you even choose one with such a high bias?), I presume the submitter decided upon this title after being encouraged by this affirmation of the article's author: 'This data indicates that the skills necessary to be a data “wizard” can be learned in disciplines other than computer sciences and mathematics.'.
This is a half-baked conclusion. I'd reason most Kaggle participants are first of all, machine learning fans, either professionals or 'amateurs' with no formal qualifications, having studied it as a hobby. I doubt people with a degree in cognitive sciences or otherwise in the 'other' categories as mentioned in the article learned enough just through their university studies to readily be able to jump into machine learning.
Is k-means really what people are doing in serious production machine-learning settings? In a previous job, we did k-means clustering to identify groups of similar hosts on networks; we didn't call it "machine learning", but rather just "statistical clustering". I had always assumed the anomaly models we worked with were far simpler than what machine learning systems do; they seemed unworthy even of the term "mathematical models".
K-means is simple, easy to implement and it often works quite well. It's unlikely to be state-of-the-art for any particular problem, but if you just need to do some straightforward clustering as part of a larger data-processing pipeline, K-means is a reasonable default choice, at least until you figure out that the clustering step is your bottleneck in terms of overall performance.
For example, this paper http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.p... demonstrates that using K-means to learn feature representations for images (i.e., use k-means to cluster small patches of pixels from natural images, then represent images in the basis of cluster centers; repeat this several times to form a "deep" network of feature representations) gives image classification results that are competitive with much more complex / intimidating deep neural network models.
Whatever problem you are solving, if you don't try k-means before spending a huge amount of programmer and computer time on more complex algorithms, you are doing it wrong.
Just as most programming is CRUD, most machine learning in practice is k-means or linear models.
Is k-means really what people are doing in serious production machine-learning settings?
Yes, that is one option. Ask me over Christmas and I'll tell you a story about a production system using k-means. (Sorry guys, not a story for public consumption.)
More broadly, a lot of machine learning / AI / etc is simpler than people expect under the hood. It's almost a joke in the AI field: as soon as you produce a system that actually works, people say "Oh that's not AI, that's just math."
Most of the classifiers I've seen used in industry aren't even k-means. They're fixed heuristics (often with plausibly effective decision boundaries found via k-means).
Consider the Netflix Prize, where the most effective algorithm couldn't actually be used in production. Commercial applications tend to favor "good enough" classification with low computational complexity.
> It's almost a joke in the AI field: as soon as you produce a system that actually works, people say "Oh that's not AI, that's just math."
I think that's the standard "insert joke here" any time someone has a "What is AI?" slide.
Wait till you have to hand craft your algorithms because the off the shelf ones are too slow ;). In the end you can stand on the shoulders of giants all day, but until you actually sit down and write an SVM or even something more cutting edge like stacked deep autoencoders yourself, machine learning isn't "easy".
In the end, libs are there for simpler use cases or educational purposes. Realistically, that's more than good enough for 90% of people.
That being said, it's not impossible to learn. Oversimplifying the statistics, tuning, and work that goes in to these algorithms you're using though? Not a good idea.
I'm cynical about how machine learning of this type might be used in practice and this is an illustration of why: the stated goal is a "you might also like" section.
There is no reason to believe the results are any better than a random method in respect of the goal (and it's reasonable to believe they may be worse) - we would have to measure this separately by clickthrough rate or user satisfaction survey, perhaps.
I believe you would get far better results by always posting the three most popular articles. If you want to personalise, post personally-unread articles. A lot less technical work, a lot less on-the-fly calculation, a lot more effective. The machine learning tools do not fit the goal.
The most effective real example of a "you might also like" section is the Mail Online's Sidebar of Shame. As best as I can tell, they display their popular articles in a fixed order.
Machine Learning seems to make it easy to answer the wrong question.
This is really only an argument that the example in the article is not realistic (which it doesn't have to be, it might be expository). There are in fact countless applications of machine learning in actual daily use, such as detecting credit card fraud, where simpler manual methods would perform measurably worse in terms of money lost.
Sure, there are realistic applications of Machine Learning with great results.
But the article has failed in its headline goal ("ML is easier than it looks") if it chooses an example that is mathematically more complicated and still less effective than a naive alternative.
The first difficult task is to identify an effective method.
The question "do I need hard math for ML" often comes up in #machinelearning at irc.freenode.net
My point here is: You don't need hard math (most of the times) because most machine learning methods are already coded in half a dozen different languages. So its similar to fft. You do not need to understand why fft works, just when and how to apply it.
The typical machine learning workflow is: Data mining -> feature extraction -> applying a ML method.
I often joke that I'm using Weka as a hammer, to check, if I managed to shape the problem into a nail. Now the critical part is feature extraction. Once this is done right, most methods show more or less good results. Just pick the one that fits best in results, time and memory constrains. You might need to recode the method from Java to C to speedup, or to embed it. But this requires nearly no math skills, just code reading, writing and testing skills.
As a graduate in artificial intelligence and machine learning I can tell you that machine learning IS hard.
Sure, the basic concepts are easy to understand. Sure, you can hack together a program that performs quite well on some tasks. But there are so much (interesting) problems that are not at all easy to solve or understand.
Like structural engineering it is easy to understand the concepts, and it is even easy to build a pillow fort in the living room, but it is not easy to build an actual bridge that is light, strong, etc.
I recently read a book called "Data Smart" [1], where the author does k-means and prediction algorithms literally in Excel. This was quite eye opening as the view to ML is not so enigmatic to enter. However, the translation of your data into a format/model to run ML is another challenge.
Matrix multiplication, orthonormal basis, triangular matrix, gradient descent, integrals, Lebesgue mesure, convex, and the mathematical notation in the paper are not harder than the code shown here. It is better to have solid prof of what you are doing is sound and will converge before jumping into the code.
They should try using tf-idf to create the initial representation of the keywords per post...also, I find there are many cases where applying machine learning/statistics correctly is harder than it looks, this single case not withstanding.
Well I don't think any of the problems you mentioned are difficult to solve, even by a non-seasoned programmer.
For your first question, just use Euclidean distance. (In other words, sqrt((x1 - y1)^2 + (x2 - y2)^2 + ... + (xn - yn)^2). IMO, anyone who can program can do that.
Also for your second question - probably - but he's using a very simplistic model. To fix that, you simply fix the heuristic. For example, you could instead score groups by how frequently certain important words occur, like "team" and "support".
Well in this case, you can also do the normal, in other words adding the differences. The poster asked about finding the differences between points in N-dimensional space in general (he didn't specify for the author's example). But the idea is that your means wouldn't be bucketed into just a couple of values, but have a larger distribution. In any case, it doesn't matter much for this problem, but if you scored it by word count rather than a simple 1 or 0, it would matter.
It's not that it's hard to code, it's that a aimplistic approach will lead to questions that might not be intuitive.
If you use Euclidean distance, then you essentially count 1 for each word difference. That means long posts will have longer distances from shorter posts with the same topic.
If you fix that by treating "important" words differently, then you have to figure out importance, and then normalize by number of terms, etc.
Before you know it you're building a naive Bayes classifier. Again, the challenge isn't in the coding, it's in understanding the limitations of the more simplistic design.
The post does do a good job of showing how easy it is to implement knn.
The post doesn't really go into centroid selection or evaluation, or the fact that clustering on text is going to be painful once you move to a larger dataset.
172 comments
[ 1.8 ms ] story [ 215 ms ] threadThink of those graphs as being something like the rubber sheet analogy that's often used to explain general relativity - it's not a great analogy in all respects, but humans can't really visualize 4 dimensions and it captures the gist of the idea well enough. Machine learning models have a similar problem: In reality the data points in k-means clustering exist in some very high-dimensinal space that's impossible to draw. So for the sake of visual aids it's common to draw 2D plots where the only thing they're meant to suggest is that points that appear close together in the visual aid would also be fairly close to each other in the actual problem space.
If you're wondering how the author came up with the two-dimensional representation, the article doesn't say, but it's likely he used something like Principal Component Analysis (http://en.wikipedia.org/wiki/Principal_component_analysis). This is a standard technique for dimensionality reduction, meaning that it finds the "best" two-dimensional representation of the original 30,000-dimensional points, where "best" in this case means something like "preserves distances", so that points that were nearby in the original space are still relatively close in the low-dimensional representation.
The point of the graphs are as an example more than anything — in any real world solution it's not trivial to plot your documents (as representing documents with thousands of words in 1d, 2d or 3d is not trivial).
But if _you could_ represent your documents in 2d, you could plot them like I do here. And, the X-axis would represent 1 feature of your documents, and the Y-axis another.
Tl;dr don't focus on the axes — focus on the idea of being able to compare the relative distances between documents.
If we have x, y, z, i, j, and n then we'd have six dimensions.
If each word represents a dimension, any blog title with that word gets a 1 on that word dimension, or a 0 if does not.
You can then calculate the distance between these points considering all of the dimensions, although clearly it's going to be a little limited on binary inputs, since the square of the differences is always going to be 1 or 0. Like for 4 dimensions, sqrt( ( 0 - 1 )^2 + ( 0 - 0 )^2 + ( 0 - 0 )^2 + ( 1 - 0 )^2 ), which just always gets simplified to something like sqrt( 1 + 0 + 0 + 1 ), which is a little boring.
The binary values cause your data points to all be stacked directly on top of each other, which leads me to believe that using binary inputs is a less than ideal application for k-means. Just look at it in the 2d case, where you have either [0,1], [0,0], [1,0], or [1,1] for each data point. Not very hard to determine the clustering there... basically just doing an overly complex boolean expression.
etc.
Seriously: The writer would not be able to write anything about K-Means if not for people looking at it from a mathematical view point. This angle is of tremendous importance if you want to know how your algorithm behaves in corner cases.
This does not suffice, if you have an actual application (e.g. a recommendation or a hand tracking or an object recognition engine). These need to work as good as you can make it because every improvement of it will result in $$$.
They are encouraging non-maths people to give the basics a try even though it seems intimidating. And it worked on me, like some others in this thread my maths is shaky beyond diff calculus and my eyes glaze over when I see notation - but now I'd like to give this a whirl. I have no illusions that I will suddenly be an expert.
My math starts getting pretty shaky around Calculus; vector calculus is beyond me.
I did about half the machine learning class from coursera, andrew ng's. Machine learning is conceptually much simpler than one would guess; both gradient descent and the shallow-neural network type; and in fact it is actually pretty simple to get basic things to work.
I agree with the author that the notation, etc, can be quite intimidating vs what is "really going on".
however, applied machine learning is still friggin' hard; at least to me; and I consider myself a pretty decent programmer. Naive solutions are just unusable in almost any real application, and the author's use of loops and maps are great for teaching machine learning; but everything needs to be transformed to higher level vector/matrix problems in order to be genuinely useful.
That isn't unattainable by any means; but the fact remains (imho) that without the strong base in vector calculus and idiosyncratic techniques for transforming these problems into more efficient means of computations; usable machine learning is far from "easy".
Although in my opinion using ML and have some decent results is not that hard. Use it and being close to the state in the art (i.e improving your recognition rate from 65% to 80%) is a completely different matter and it is hard and it requires a fully understanding of the details of the algorithm in use.
Still it doesn't meant that you can't do interesting things with an overall understanding of machine learning, but making a good cat detector is hard [0] [1]
[0] http://research.google.com/archive/unsupervised_icml2012.htm...
[1] http://techtalks.tv/talks/machine-learning-and-ai-via-brain-...
Then you'd still fail to account for symbols invented/repurposed in a single paper or by a single author.
Basically, the tool itself is a machine learning problem.
But... the Venn diagram of people able to do such a thing, people motivated to do such a thing (i.e. people reading academic papers who aren't math whizzes), and people with the time to do such a task... seems not to have a sufficient intersection. Most of us in this thread fulfill categories 1 and 2, but not 3... come to think of it, I have work to do... ducks.
Maybe, but not always. Remember that Richard Feynman took great issue with how integration was taught in most math classes and devised his own method (inspired from the Calculus for the Practical Man texts).
It sort of is the members only cigar club of academics. While I can read, understand, and subsequently implement most things in computer science land, when it comes to mathematical notation, there is a lot left to understand.
I think a LOT of people would benefit from a five or six video course simply showing a translation of complex notations to a working algorithm in a popular language, so the commonly used symbols start having meaning.
Mathematical equations don't have any meaning taken by themselves; they just express a certain relationship between quantities without stating anything specific about what those quantities are.
Any translation of an equation to English would be a certain (usually lossy) interpretation of the equation for some particular quantities. But the same equation can have multiple interpretations in wildly different contexts.
Striking example is quantum mechanics. Physicists in the first half of the 20th century figured out a mathematical model (e.g. the Shroedinger equation) that worked perfectly well for predicting the behaviour of fundamental particles. Interpreting what the equation actually means (let's say in English) proved to be hard though and spawned multiple, sometimes conflicting or paradoxical interpretations. In fact, to this very day physicists are split on their favorite interpretation of QM. On the positive side, the mathematics of QM just works no matter how the result or the calculation are being interpreted.
eg http://www.dessci.com/en/products/mathplayer/tech/accessibil...
(yes, I know that's for IE. There are other systems though).
Such things do exist. Unfortunately, they are called "textbooks".
That being said there are plenty of barriers to a software engineer picking up some papers on machine learning and getting started. These are the result of some serious issues that I believe exist with academia at present and specifically the "publication game". I will not however get into this as this comment will quickly turn into the longest paper I've ever written. It is a problem with the system, and as the reader it is not your fault.
Your issue with applied machine learning may be largely imagined. Take for example the k-means clustering algorithm in the article. This is quite useful in a number of places be it data analysis or intelligent user interfaces, its straightforward to implement, and many usable implementations exist online. The same case is true for other models such as linear regression, naive bayes, even things like support vector machines are pretty easy to use these days.
In any case, as selfish as this is, this sentiment you're expressing provides some comfort that I'll be in demand when I enter the job market. :P Though if you really want to implement and apply some simple yet useful algorithms he recommends Programming Collective Intelligence at the end of the article and I can definitely second that recommendation.
When were talking about applied machine learning it is generally in the context of developing a product, a feature. The most important part is going from data to feature, not necessarily squeezing the last few percentage points of performance. In this context getting a good answer easily is more important than getting the best answer.
(I'll warn you that a lot about the book is very dated now)
First off thanks for writing the book. It was the first book recommended to me for ML while I was doing my undergrad and it got me using ML as opposed to studying ML. The book opened my eyes to the fact that I really could do these things, and it definitely played a role in influencing me to pursue my MSc in AI. So again, thank you. :)
Secondly I'll agree that the book is most definitely dated at this point though it provides a survey of quite a broad selection of algorithms. It certainly isn't going to put the reader on the cutting edge or anywhere near it but it will get them started which appears to be the hardest part for some of those participating in this comment thread.
Have you considered a second edition?
Anyway, my math is not that great and I'm way out of practice when it comes to theoretical work. Not only that, I suck at algorithm design and I'm at best a moderately decent developer from a theory standpoint.
But applied machine learning came natural, because I have a strong grasp of frequentist and Bayesian statistics as well as solving the problems backwards. If you try to apply ML or any other "exotic" solution by starting with "oh man here is this awesome technique," you will definitely be frustrated; I know that feeling all too well.
ML is just another tool in the shed. With practice, you start to understand where simple linear regression techniques are useful and when more heavy-duty methods are applicable. Keep solving real world problems rather than trying to look for stuff to hit with your new hammer (ML in this case) and I bet you'll come around a lot quicker.
My take is that 'applying' machine learning is much easier than one would think, but 'understanding' machine learning is _much_ harder than one would think.
I imagine most people start with an application - that's why they're interested in the first place. Applying an ML algorithm to a given dataset is trivial - only a few lines in R/Python with scikits-learn, etc. There's even a handy cheat-sheet [1] to tell you which algorithm to use. Your regression model spits out a few parameters, some estimates of quality of fit, maybe some pretty graphs, and you can call it a day.
The next stage is implementation - up until now, the black box has been spitting out parameters, and you want to understand what these numbers actually are and how they are determined. This means getting your hands dirty with the nitty-gritty of ML algorithms - understanding the algorithms statistical basis, understanding the computational performance of the algorithms (as you alluded to), and reading textbooks/journal articles. Many of these algorithms have a simple 'kernel' of an idea that can be obscured by a lot of mathematical sophistication - For SVM, the idea of finding a plane that separates your data as much as possible, logistic regression is just finding the line of best fit in logit space, k-means is just of repeatedly assigning points to clusters then updating clusters, etc.
It's not _that_ difficult to get working implementations of these algorithms (although debugging an incorrect implementation can be frustrating). You can implement simple versions of gradient boosting machines - a theoretically sophisticated technique that often works very well in practice - in around 100 lines of Python [2] or Go [3]. You can get implement Hopfield networks (a basic neural network for unsupervised learning) in Julia [4] and Haskell [5] without much trouble either.
I think where a lot of the value from spending 3-5+ years in graduate school comes in when you an off-the-shelf technique isn't cutting it - maybe you're overfitting, you want to play with hyperparameters, your datasets are scaling faster than the off-the-shelf implementations can deal with them, etc. Solving these kind of problems is where a lot of the mathematical/statistical sophistication can come into play - Bayesian hyperparameter optimization of existing ML techniques improved on state-of-the-art in image processing [6], understanding _why_ the LASSO works (and understanding when it doesn't) can mean having to understand the distribution of eigenvalues of random Gaussian matrices [7]. That's saying nothing of figuring out how to take an existing algorithm and scaling it out in parallel across 1000+ machines [8], which can mean using some deep convex optimization theory [9] to justify your approximations and prove that the approximated model you learn is well-behaved, etc.
I suspect that some of the gulf between ML in practice and theory arises from the differing goals in the two parties. If you just want an parameter estimation, then a lot of the rigour will seem useless, and if you want to understand a procedure deeply, you can't do away with rigour. Take the k-means paper used in the article. In the screenshot, the paper's author is presenting results of the flavour
> Under weak conditions on the data, as the number of iterations increases, regardless of the random choices made, the k-means clustering tends towards a good clustering of the data.
Depending on your viewpoint, this result is either obvious, non-obvious, or irrelevant. To prove this needs (at a glance) theorems from topology, measure theory, martingale convergence, etc.
I suppose some people find that fascinating, and want to understand _why_ this is true, and what 'good&...
It's great that we can stand on the shoulders of giants, but having a certain understanding of what these algorithms are doing is critical for choosing them and the parameters in question.
Also, K-means is relatively easy to understand untuitively. Try doing that with Latent Dirichlet Allocation, Pachinko Allocation, etc. Even Principal Component Analysis and Linear Least Squares have some nontrivial properties that need to be understood.
Also - maybe "article recommendation" is "easy" in this context, but other areas such as computer vision, sentiment analysis are not. Some other questions I might ask
How do you know how well this algorithm is performing?
How are you going to compare this model to other models? Which metrics will you use? What statistical tests would you use and why?
What assumptions are you making here ? How do you know you can make them and why?
There are a lot of things that this article fails to address.
Disclaimer: I realize more complex models + features don't always lead to better performance, but you need to know how to verify that to be sure.
"Here are some training points from classes A and B, each with an importance weight. Compute the weighted average of your new point's similarity to all the training exemplars. If the total comes out positive then it's from class A, otherwise it's from class B."
If you tried to explain training from a quadratic programming perspective it would get messy, but if you use stochastic gradient descent then even the training algorithm (for primal linear SVM) is pretty light on math.
Of course, this is only possible after a decade or two of researchers have played with an digested the ideas for you. I think until Léon Bottou's SGD SVM paper, even cursory explanations of SVMs seemed to get derailed into convex duality, quadratic programming solvers, and reproducing kernel hilbert spaces.
As far as I know, k-means (as the author described it) has two parameters: initial number of clusters K and max iterations (although I could be wrong), SVMS have:
UPDATE: here is a gist:
https://gist.github.com/josephmisiti/7572696
taken from here : http://www.csie.ntu.edu.tw/~cjlin/libsvm/
I do not think that picking up LibSVM, compiling it, and running it is straight forward for everyone ...
If explaining SVM, I'd stick with the linear primal formulation, which only really requires choosing the slack parameter C. If I needed to perform or explain non-linear prediction, I'd switch to the RBF kernel, which gives you one additional parameter (the RBF variance).
Implementations of K-means, by the way, can also potentially have lots of parameters. Check out: http://scikit-learn.org/stable/modules/generated/sklearn.clu...
I'm going to finish grad school soon and the job I'm going to will be pretty rich with data analysis, hopefully will lead to more blog posts.
I disagree in so many ways: 1. complex algorithms are usually very short in practice (e.g. dijkstra's shortest path or edit distance are the firsts that come to mind) 2. ML is not just applying ML algorithms: you have to evaluate your results, experiment with features, visualize data, think about what you can exploit and discover patterns that can improve your models. 3. If you know the properties of the algorithms you are using then you can have some insights that might help you on improving your results drastically. It's very easy to apply the right algorithms with the wrong normalizations and still get decent results in some tests.
For example, a matrix factorization could be explained with two arrays, a and b, that represent objects in the prediction:
It's that simple. Multiply the weights of a by the weights of b, calculate error and adjust weights, repeat.K-Nearest Neighbor/KMeans are based on an even simpler operation:
Then make predictions/build clusters based on the smallest aggregate distance.There are more advanced concepts. There are some serious mathematics involved in some predictors. But the most basic elements of statistical prediction are dead simple for a trained programmer to understand. Given enough data, 80% solutions can easily be achieved with simple tools.
We should be spreading the word about the simplicity of fundamental prediction algorithms, not telling people that it's hard and a lot of math background is required. Machine learning is very powerful and can improve all of our lives, but only if there is enough data available. Since information tends to be unevenly distributed we need to get the tools into the hands of as many people as possible. It would be much better to focus on the concepts that everyone can understand instead of keeping statistics secrets behind the ivy clad walls of academia.
Adding to that, a lot of methods that fall under the "machine learning" label are concepts that are pretty well understood in statistical point estimation, testing of hypotheses, and numerical analysis. I'm not 100% that it really ought to be treated as a new discipline.
On the other hand, I'm also learning that what I/others find easy, a great deal of people/programmers find hard. The number of programmers/hackers who can actually implement such techniques on new real-world problems if they don't have someone holding their hand I'm discovering is a very small minority. So maybe its harder than it looks after all, and we just think its easy because we've spent so much time with it? After all, programming isn't a walk in the park for most people, and machine learning isn't a walk in the park for most programmers.
What do you mean? Academic papers are usually written in plain english with mathematical formulas when it's necessary. What kind of comments would you like to see?
That being said, I think mathematicians should be perfectly permitted to do this, seeing as most people who read their papers are themselves, mathematicians. Thus spelling out every dumb detail would probably just be a waste of their time for the one case that the brave soul who is a programmer tries to decipher it.
For example, everyone intuitively knows what holes are. Well, how do you rigorously define a hole? It took centuries to come up with the correct definition, which is given in terms of homotopy groups, and the definition of homotopy groups will look incomprehensible to the non-mathematician. Seemingly "easy" statements like "bounded three dimensional objects without holes are spheres" (i.e. the Poincare conjecture) turn out to be very, very hard (worth a million dollars and a Fields medal!)
Papers are not just blocks of equations. There is english giving context. There is also the issue of length limits for a paper. It's assumed that the reader has a background in the area and so the authors can be a bit more concise.
> structure
Most papers in math follow a common structure of definition-theorem-proof. You can think of lemmas as subroutines that are 'called' in a proof.
>not-greek-characters (but they make me look smart!)
Unless you're advocating spelling out things (i.e. calling something "decay constant" instead of using an alpha), I don't think there is a fundamental difference between using a Greek or English alphabet. Greek is nice because it gives you more characters to use beyond the English alphabet (which is already used extensively in math). If you _are_ advocating replacing single symbols with words, I think that would be pretty hard to parse, and rather cumbersome in longer formulas.
>abstractions
Math is _all about_ abstraction. I don't know what you mean by this.
http://plotting-success.softwareadvice.com/who-are-the-kaggl...
To other readers, follow that link at your own risk. Vikp has a blog[1] which has hours and hours of excellent writing about machine learning.
[1] http://vikparuchuri.com/blog/
CPUs were magical for me before I took a computer architecture course. So was AI and machine learning. Once you see the "trick" so to speak you lose some of the initial awe.
While the article doesn't have this title (why would you even choose one with such a high bias?), I presume the submitter decided upon this title after being encouraged by this affirmation of the article's author: 'This data indicates that the skills necessary to be a data “wizard” can be learned in disciplines other than computer sciences and mathematics.'.
This is a half-baked conclusion. I'd reason most Kaggle participants are first of all, machine learning fans, either professionals or 'amateurs' with no formal qualifications, having studied it as a hobby. I doubt people with a degree in cognitive sciences or otherwise in the 'other' categories as mentioned in the article learned enough just through their university studies to readily be able to jump into machine learning.
See: http://en.wikipedia.org/wiki/AI_winter
The AI Winter was a long time ago now.
For example, this paper http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.p... demonstrates that using K-means to learn feature representations for images (i.e., use k-means to cluster small patches of pixels from natural images, then represent images in the basis of cluster centers; repeat this several times to form a "deep" network of feature representations) gives image classification results that are competitive with much more complex / intimidating deep neural network models.
Just as most programming is CRUD, most machine learning in practice is k-means or linear models.
Yes, that is one option. Ask me over Christmas and I'll tell you a story about a production system using k-means. (Sorry guys, not a story for public consumption.)
More broadly, a lot of machine learning / AI / etc is simpler than people expect under the hood. It's almost a joke in the AI field: as soon as you produce a system that actually works, people say "Oh that's not AI, that's just math."
Consider the Netflix Prize, where the most effective algorithm couldn't actually be used in production. Commercial applications tend to favor "good enough" classification with low computational complexity.
> It's almost a joke in the AI field: as soon as you produce a system that actually works, people say "Oh that's not AI, that's just math."
I think that's the standard "insert joke here" any time someone has a "What is AI?" slide.
In the end, libs are there for simpler use cases or educational purposes. Realistically, that's more than good enough for 90% of people.
That being said, it's not impossible to learn. Oversimplifying the statistics, tuning, and work that goes in to these algorithms you're using though? Not a good idea.
There is no reason to believe the results are any better than a random method in respect of the goal (and it's reasonable to believe they may be worse) - we would have to measure this separately by clickthrough rate or user satisfaction survey, perhaps.
I believe you would get far better results by always posting the three most popular articles. If you want to personalise, post personally-unread articles. A lot less technical work, a lot less on-the-fly calculation, a lot more effective. The machine learning tools do not fit the goal.
The most effective real example of a "you might also like" section is the Mail Online's Sidebar of Shame. As best as I can tell, they display their popular articles in a fixed order.
Machine Learning seems to make it easy to answer the wrong question.
But the article has failed in its headline goal ("ML is easier than it looks") if it chooses an example that is mathematically more complicated and still less effective than a naive alternative.
The first difficult task is to identify an effective method.
My point here is: You don't need hard math (most of the times) because most machine learning methods are already coded in half a dozen different languages. So its similar to fft. You do not need to understand why fft works, just when and how to apply it.
The typical machine learning workflow is: Data mining -> feature extraction -> applying a ML method.
I often joke that I'm using Weka as a hammer, to check, if I managed to shape the problem into a nail. Now the critical part is feature extraction. Once this is done right, most methods show more or less good results. Just pick the one that fits best in results, time and memory constrains. You might need to recode the method from Java to C to speedup, or to embed it. But this requires nearly no math skills, just code reading, writing and testing skills.
Sure, the basic concepts are easy to understand. Sure, you can hack together a program that performs quite well on some tasks. But there are so much (interesting) problems that are not at all easy to solve or understand.
Like structural engineering it is easy to understand the concepts, and it is even easy to build a pillow fort in the living room, but it is not easy to build an actual bridge that is light, strong, etc.
[1] http://www.amazon.com/Data-Smart-Science-Transform-Informati...
But it would help to highlight some of the fundamental challenges of a simplistic approach.
For example, how is the author computing the distance between points in n-dimensional space?
And does this mean that a one-paragraph post and a ten-paragraph post on the same topic probably wouldn't be clustered together?
For your first question, just use Euclidean distance. (In other words, sqrt((x1 - y1)^2 + (x2 - y2)^2 + ... + (xn - yn)^2). IMO, anyone who can program can do that.
Also for your second question - probably - but he's using a very simplistic model. To fix that, you simply fix the heuristic. For example, you could instead score groups by how frequently certain important words occur, like "team" and "support".
If you use Euclidean distance, then you essentially count 1 for each word difference. That means long posts will have longer distances from shorter posts with the same topic.
If you fix that by treating "important" words differently, then you have to figure out importance, and then normalize by number of terms, etc.
Before you know it you're building a naive Bayes classifier. Again, the challenge isn't in the coding, it's in understanding the limitations of the more simplistic design.
The post doesn't really go into centroid selection or evaluation, or the fact that clustering on text is going to be painful once you move to a larger dataset.