Ask HN: Why is still so hard to detect spam?
I'm using macOS Mail as a secondary client alongside Google Inbox (which works great), and it's amazing how bad the spam filter is.
I've been training it to detect spam by marking emails manually for about 2 weeks, and it's still having a hard time recognising most spam emails.
I know that spammers get clever trying to trick spam filters, but I feel like some emails are really a no-brainer they're spam: for instance, I'm getting hundreds of email in Chinese, and I don't even have a Chinese font installed (which is information I assume macOS could potentially have access to). What are the chances it's not spam?
3 comments
[ 3.1 ms ] story [ 16.4 ms ] threadPersonally, I have been relying exclusively on server-side spam filters (GMail in my case) for years now and it works well enough. Server-side spam filters on the other hand cannot know for sure you're not expecting emails in Chinese. I suppose that's what these spammers are capitalising on in that case.
Spam filtering on the client side is expensive, both in terms of the effort required to create the software and regarding CPU cycles on your local machine. Server-side spam filters like Google's use simple voting algorithms that scale very well. On the client side you need language models tuned specifically to your email usage in order to avoid false positives. These models and the training required are quite complex. Simple Bayesian filters simply aren't good enough anymore to beat today's sophisticated spam creation algorithms.
It's a bit like an inverse P vs. NP problem: Spam looking like natural language on the surface can easily be created but verifying that it's spam is very hard, on a local level at least.
Still, I think Mail could mark emails written in a language you don't speak as spam pretty easily.
My mail server does do some gray and red filtering, so it starts with a relative clean slate (avg 3 spams/day over 20 addresses which have been public for at least 10 years), so that might be part of it.