HN Explorer

17 comments

[ 4.5 ms ] story [ 45.0 ms ] thread
Morendil
Yes: http://bit.ly/lerK1S

(This is a time series graph of the count of my GMail spam folder since 2009, updated daily.)

ScottWhigham
??? You are getting more than 4,000 spam emails a day now and were above 30,000? Do you have your email listed publicly?

Also how do you automate the counting and storing it in the spreadsheet? I think that's pretty damn cool.

Morendil
It's per-month, not per-day - GMail only stores 30 days' worth of email in the Spam folder.

As to how - cron job on my server querying GMail via IMAP. Source below. Manually copy and paste the resulting text file to a GDoc spreadsheet; use the Time Series widget to generate the chart.

    require 'net/imap'
    require 'date'
    imap = Net::IMAP.new('imap.gmail.com',993,true)
    imap.login('yourlogin','yourpwd')
    msgs = imap.status("[Gmail]/Spam",["MESSAGES"])["MESSAGES"]
    puts Date.today.to_s + "\t" + msgs.to_s
aquarin
Does this graph show some seasonal pattern?
Morendil
Doesn't look like it, but perhaps a closer analysis would show some. One thing it shows for sure is a change, early on, in GMail's algorithm for deleting old mails left in the Spam folder - that's where it goes from jagged to smoother. And it confirms the article's "spam at its lowest" story.
ColinWright
That's almost bump-for-bump identical to my data, only I show data going back to 2002, and you're only showing back to 2009. The correspondence is remarkable:

http://www.solipsys.co.uk/new/SpamGraphs.html?HN

Morendil
That's one reason I set up this cron in the first place - I wanted to see if "my spam" was the same as "everyone else's spam".

I wish I'd had your foresight and set this up earlier, but now I'm glad I didn't let that prevent me from doing it anyway; it's always fun to play scientist. :)

kalleboo
I suspect spammers have moved from email to the web. I'm seeing more forum/comment form spam now than ever (especially on disqus sites, I've only seen 1-2 spam facebook comments yet). Too bad web spam is basically impossible to measure overall.
ch0wn
I've noticed an increase of comment spam on reddit, as well. The spammers there sometimes have quite clever algorithms using phrases of top-voted comments and embedding links in them.
Shenglong
Definitely. I ran a forum with quite a few users, and spent way too much time dealing with spam. :( I've found that the best way to avoid forum spam is indirect methods. Instead of a captcha, I set an invisible section where only newbies could see - and that's the only area they could post. After one post there, they were upgraded to regular member.
getsat
It's usually just people running Xrumer[1] to get more backlinks to their site. You can pay people $5 on fiverr.com to register 5,000+ forum accounts and drop a link back to your site.

Backlinks = higher SERP positioning = $$$

[1] http://en.wikipedia.org/wiki/XRumer

bruce511
From the article: "Worldwide spam is now down to one in every 1.37 emails. In the United States, spam accounts for 73.7% of all emails."

This is classic journalism, putting two related facts in the same sentence, but in such a way as to make comparison really really hard.

It's like saying that the average height of people world-wide is 1.8m, but in the US it's 6 foot. Huh?

So - for the record, 1 in 1.37 is 72.99%. So this sentence could, and should read;

"Worldwide spam is now down to 72.99% of all emails. In the United States, it accounts for 73.7% of all emails."

I'm guessing that .7% is well within their margin of error, meaning that the paragraph overall is somewhat redundant.

Then a bunch of graphs, with no Y-Axis scale showing vast up and down fluctuations further erode what credibility the author had left.

In short - a fluff piece devoid of the really interesting information.

chalst
A fluffy piece but not one devoid of interesting information, at least for me - I found the reminder of events over the last three years useful to putting the numbers in context.

The most egregious misuse of numbers lies in the claim [t]he geographic center of spammed accounts has also shifted from Russia to Saudi Arabia, which is justified by the fact that Saudi Arabia (population of around 25 million) has a spam rate of 82.2% against Russia's (pop. c. 140m) spam rate of 79.4%. It's been a while since I've seen such extreme analytical incompetence from a journalist.

agj
I've noticed this trend directly. Our incoming non-spam rate has stayed steady at an avg of about 10 msgs/min in the least few years, but our overall average rate has dropped from 250 msgs/min to about 60 msgs/min.

Another trend that I've noticed recently, and what seems is an example of a targeted attack, is an increase in brute force attacks on user accounts. This is likely anecdotal though, if something works once for spammers, they are pretty relentless at trying to attack it again.

I do find it interesting to watch these trends shift, and to have to respond to new methods. It's a great, big, penis-enlargement-pill-fueled, game of cat and mouse to me.

georgemcbay
As a non-admin user, I haven't really noticed. Not because I still get a lot of spam but because it hasn't been a problem for me for a few years now and it continues to be not-a-problem.

With that in mind it makes sense that the levels are dropping... spam filters have gotten good enough to make spamming not worth the trouble in terms of effort/return (especially given the increased risk of a successful anti-spam lawsuit against you as a spammer), so the levels drop, mostly invisibly to the users who weren't seeing the majority of the spam in the first place.

[deleted]
(comment deleted)