What a beautiful article. I have read Bayes many times and each time just when I thought I had it it slipped away. This article does a beautiful job of explaining Bayes in simple terms and its application in classification. Thank you!
I agree. My first machine learning class, my prof only allowed scikit learn for boilerplate stuff like train test splits and we had to code a bunch of different models from scratch. It was really educational. I’m still slightly traumatized from my EM assignment!!
Perhaps the author should have done both: a handwritten NB followed by the sklearn implementation.
It’s good to know how to create these algorithms from scratch (especially to demystify them) but at the same time you’re probably better off using sklearn and using your time instead to manage your data (unless there are significant customizations needed to fit the problem that sklearn can’t provide).
Bayes is very simple, easily understood by a 6 year old, if explained properly. There are scary looking integral equations used to express it, but ultimately if you can count you can implement this.
Let's build a bayesian spam engine:
Start with a known-good email (not-spam) and a solicitation (spam). For each word in the not-spam document, count the number of times each word appears. That's your not-spam "corpus". Repeat for the known-spam document. Now you have a list of words+frequency (dictionary of counts). There will be many words which appear in both lists. This is normal.
Now, when you have an unclassified document, for each word in the document check both lists (spam & not-spam). In which list does this word appear more frequently? if the count for that word is higher in the spam list, then this is a "spam" word, therefore this document is +1 spam. If a not-spam word, then this document is +1 not-spam. Repeat for all words. At the end, which count (spam or not-spam) has the higher total? There's your answer. Spam or not-spam. Done.
Simple counting and comparison.
Seriously, that's the algorithm. It's that simple.
If the system miscategorizes a document, move it to the appropriate group yourself and count all the words again. (retrain). The system just "learned" from the mistake. It got "smarter".
You can easily add more categories, other than "spam" and "not spam" -- "questions from customers", "family & friends", etc. It works the same. The group with the highest total "wins" the classification.
extra credit:
Now here's the interesting part. The corpus doesn't have to be words. It could be a switch set to "on" or "off." Joint angles on a robot. Ping time of a packet. Temperature from a thermistor.
And the classification doesn't need to be spam/not-spam, it could be "friend vs foe", upright/inverted, safe/not-safe, or anything else. The algorithm doesn't care, it's looking for probability of being in a classification group.
>Seriously, that's the algorithm. It's that simple.
It's funny because when I saw that line I thought I would be pedantic and double check to see if you got the laplacian smoothing correct (because in practice it's never "that simple" for any numeric algorithm) and then realized that you don't understand how to implement Naive Bayes' at all.
For starters you aren't using probability. If you want to put all of the words in each document together into a two dictionaries of counts, then for each word in the unclassified document you want to look at the product of the probability of those words appearing in the spam corpus vs the non-spam corpus. That probability is n_word/total_words the corpus.
This is where you need to do some smoothing because if a word does not appear in the one of the corpora then you will get a probability of 0 for that class. Smoothing just adds 1 to the numerator and N_classes to the denominator. It is the equivalent of assuming a weakly informative uniform prior.
Implementation of NB can be much more interesting than that:
- try different models for the likelihood (the author mentioned Gaussian, Bernoulli, and Multinomial which are part of scikit but didn't bother to implement them)
- understand how the "naive" assumption makes implementation easier but less expressive (author mentions it in words but pictures are critical here, e.g. the Gaussian case)
This would be way better in a format where I didn't have to click over and over. One page with the text going down, even.
Yours was a decent example. I've seen one for weather conditions on a golf course and people's inclination to play used to predict how busy you'd be based on the weather that was really good.
They're slides intended for a live presentation. You can use space bar or arrow keys to flip through as well. We've found this format to work well for our 2000+ member meetup group, but I might follow your suggestion to write it out long-form sometime. The more technical documentation is at: https://github.com/73rhodes/dclassify
This is basically the core of anything that uses 'Naive Bayes', that's really all there is to it. Note that Naive Bayes tends to 'clamp' to 0 or 1 after enough evidence has been processed, you're not going to end up with a value somewhere in the middle in the vast majority of cases. Also, when it doesn't work there won't be any hint that you are mis-classifying, because of the above mentioned property the posterior returned is not going to help much in terms of determining your confidence level. Note that if you evaluate a lot of evidence sequentially and the criteria are not 'independent' then you won't get good results. Independent criteria can vary independently from each other, so for instance if you base three of your criteria on someone's IP address you won't get much mileage out of the second and third and you're going to over-represent that factor in the weighing of the evidence.
Still, it is super easy to get up and running, will work with remarkably little data to train with (determine those false positive and true positive ratios) and runs very fast during classification, it also requires very little in terms of hardware (no GPU or anything like that).
Yes, you can improve on this, but it isn't always worth the effort or the resources, I've built more than one revenue generating tool with this.
Yep, sorry that could have been made clearer. I typically have all the tests in an array of structs that way I can add a pointer to a function to actually execute the test and add a name to it for debugging purposes.
I wrote a starter article quite a while ago about using Naive Bayes for analyzing clickstream data if anyone is interested: https://tinyurl.com/ulmam85
Recommend downloading the PDF for better rendering of math symbols.
25 comments
[ 3.5 ms ] story [ 81.8 ms ] threadHow can someone write that much text, and then all they end up doing is importing scikit-learn?
I can figure out how to do that myself in 2 minutes.
The real understanding of machine learning comes when you start implementing the models yourself.
It’s good to know how to create these algorithms from scratch (especially to demystify them) but at the same time you’re probably better off using sklearn and using your time instead to manage your data (unless there are significant customizations needed to fit the problem that sklearn can’t provide).
Let's build a bayesian spam engine:
Start with a known-good email (not-spam) and a solicitation (spam). For each word in the not-spam document, count the number of times each word appears. That's your not-spam "corpus". Repeat for the known-spam document. Now you have a list of words+frequency (dictionary of counts). There will be many words which appear in both lists. This is normal.
Now, when you have an unclassified document, for each word in the document check both lists (spam & not-spam). In which list does this word appear more frequently? if the count for that word is higher in the spam list, then this is a "spam" word, therefore this document is +1 spam. If a not-spam word, then this document is +1 not-spam. Repeat for all words. At the end, which count (spam or not-spam) has the higher total? There's your answer. Spam or not-spam. Done.
Simple counting and comparison.
Seriously, that's the algorithm. It's that simple.
If the system miscategorizes a document, move it to the appropriate group yourself and count all the words again. (retrain). The system just "learned" from the mistake. It got "smarter".
You can easily add more categories, other than "spam" and "not spam" -- "questions from customers", "family & friends", etc. It works the same. The group with the highest total "wins" the classification.
extra credit: Now here's the interesting part. The corpus doesn't have to be words. It could be a switch set to "on" or "off." Joint angles on a robot. Ping time of a packet. Temperature from a thermistor.
And the classification doesn't need to be spam/not-spam, it could be "friend vs foe", upright/inverted, safe/not-safe, or anything else. The algorithm doesn't care, it's looking for probability of being in a classification group.
There you go. Have fun!
It's funny because when I saw that line I thought I would be pedantic and double check to see if you got the laplacian smoothing correct (because in practice it's never "that simple" for any numeric algorithm) and then realized that you don't understand how to implement Naive Bayes' at all.
For starters you aren't using probability. If you want to put all of the words in each document together into a two dictionaries of counts, then for each word in the unclassified document you want to look at the product of the probability of those words appearing in the spam corpus vs the non-spam corpus. That probability is n_word/total_words the corpus.
This is where you need to do some smoothing because if a word does not appear in the one of the corpora then you will get a probability of 0 for that class. Smoothing just adds 1 to the numerator and N_classes to the denominator. It is the equivalent of assuming a weakly informative uniform prior.
Thanks. It's been 15 years since I implemented it. Guess I failed the interview.
In this case, wouldn't it actually be better to just drop the denominator, because it will be the same for both (spam & not spam).
I never met a measure space I didn't like!
- try different models for the likelihood (the author mentioned Gaussian, Bernoulli, and Multinomial which are part of scikit but didn't bother to implement them)
- understand how the "naive" assumption makes implementation easier but less expressive (author mentions it in words but pictures are critical here, e.g. the Gaussian case)
- using log probabilities for numerical stability
- reframing the model as a linear model
Yours was a decent example. I've seen one for weather conditions on a golf course and people's inclination to play used to predict how busy you'd be based on the weather that was really good.
Edit: Now it's like.. 50/50?
Still, it is super easy to get up and running, will work with remarkably little data to train with (determine those false positive and true positive ratios) and runs very fast during classification, it also requires very little in terms of hardware (no GPU or anything like that).
Yes, you can improve on this, but it isn't always worth the effort or the resources, I've built more than one revenue generating tool with this.
Edit: Ah I'm guessing false positive and and true positive rate?
This makes adding tests very easy.
Recommend downloading the PDF for better rendering of math symbols.