1.) Run your own SMTP infrastructure. Setup SPF/DKIM/DMARC. Realize your outbound emails still don't always reach their destination. Also you have to fight inbound SPAM.
2.) Use gmail or Google Apps. Things just work. Cede control to Google.
That doesn't really hold when their competitors have teams of engineers working on DDoS mitigations and successfully handle most of these attacks. Falling to a DoS is a technical failure just like other causes of outages.
Yes -- but the people you can contact may not have the ability to fix problems if they're complex. For example, we wanted to disable clicking on links in email for our users because of phishing -- support wasn't able to help with that.
I pay $5 / month for google apps for my personal domain. I've had to contact support twice, the first time I got a phone call < 10 minutes after my email, and the second was < 20 minutes via email.
In my experience, Google support is _excellent_ for their paid products.
Don't do #1 if you want to reliably accept or send mail. Fun for experimenting/learning, though.
Also, you can buy your own domain, and have Google merely act as the SMTP relay and temporary storage. They can also forward your mail to another server for you.
While there are pain points when it comes to hosting your own mail, it isn't nearly as difficult as you make it.
It gets more difficult if you are providing a service that has to have uptime guarantees or are providing mail to many users but if you take the time to learn and educate yourself on current standards, hosting your own mail for fun and for profit is a doable thing that more developers and admins should do.
We are at a time when we actually have relatively easy to use software to managing mail servers, so us it.
For what it is worth, I have not had problems with Google, Microsoft, Yahoo, or domains that use their services whether it comes to sending or receiving. Sometimes a server is stuck in an SPAM prevention queue or I might have to whitelist a particularly silly server, but that doesn't happen very often.
> 1.) Run your own SMTP infrastructure. Setup SPF/DKIM/DMARC. Realize your outbound emails still don't always reach their destination. Also you have to fight inbound SPAM.
And if someone wants to DDoS you, you're a lot more vulnerable than a major provider like Fastmail.
Personally, I use a hybrid solution: I use Gandi's SMTP servers for outbound and inbound mail, but I run my own IMAP server for unlimited storage under my control.
If the attacker has ever seen the headers of a message you sent through fastmails SMTP service they have your public IP (Received from header) and can Dos you directly anyway.
They do something similar with their webmail service, but the data is encrypted so it can't be read by a third party.
EDIT: Fastmail is fairly priced (for me) and i like the features they offer but i wish they wouldn't do this (or rather, i wish they would do the same for the SMTP service as they do for the webmail service)
I've never been subject to DOS in fifteen-ish years of running SMTP.
Personal users aren't of interest to DOS ransomers since a call to the ISP will drop their traffic at the border. Without SLAs costing me money, as would be the case with a big provider, I coulf outlast the DOS. Just inconvenient and annoying.
Sure, but I know nothing about Rackspace and refuse to trust them as a result. Plus I doubt they have anything that can compete with Fastmail's web interface, powered by the JMAP protocol they authored.
So here's what I know about Fastmail that I want to know about any provider I host with:
- I know Fastmail will refuse any requests from the US govt to access data because they're Australian and legally the request has to come from an Australian court
- I know Fastmail's servers use encrypted storage
- I know the specs of Fastmail's servers (they post them on their Help/FAQ)
- I know Fastmail's actually trying to make email better, obvious by their invention of the JMAP protocol
- I trust Fastmail to be able to recover from any serious issues because they actually have on staff developers of Cyrus, the open source IMAP software they use. This means their admins have actual recourse when Bad Things^TM happen, vs the usual when an admin runs out of options, eg, "let's just post on mailing lists and hope we can find an answer"
- I know the fine details of how their spam filtering works, because it's publicly documented. (and it's quite well integrated with some tricks I couldn't employ at my last ISP job as I didn't have developers to assist)
- I know their infrastructure is primarily hosted in NYI with the backup in Iceland.
- I know they are serious about security, as they've been proponents of full SSL/TLS vs STARTTLS which could be MITM and downgraded (yes, many MTAs will let you require STARTTLS, but there are always possibilities of client bugs that could be exploited when you let an attacker intercept plaintext and inject data before the upgrade to a validated TLS session)
- I know how their backups work, because it's documented and I also have the ability to undelete emails which almost no provider gives the end user.
- I know their support is responsive and competent, as they've actually fixed Webmail bugs and put them into production for me within 48 hours
- Fastmail does PUSH email on iOS, while GMail, Rackspace, and most other providers don't offer this because it requires custom integration with Apple's Push Notifications service.
tl;dr yeah, the average provider might promise the moon but can they actually deliver when the shit hits the fan? will they actually strive to please their users and make the internet a better place? probably not.
Actually the Iceland bit is out of date. Network there was too unreliable. We're in Amsterdam and LA as well as New York these days. Amsterdam has the hardware that used to be in Iceland including full replicas of all email.
Amsterdam was also hit by last night's ddos. They were hitting mx, and our secondary mx is in Amsterdam.
Something i'm realizing more and more... What the hell do I really need remotely hosted mail for?
We all know mail is insecure. Unless you look really really hard, you aren't sure if the mail you received was spoofed or modified, a child can spoof mail and any MitM can modify it. So in general you can't trust your mail anyway, even if it's received by a reputable company. Sending mail is almost just as subjective... a random ISP's mail smarthost is just as good for getting your mail delivered as a hosted mail provider.
All I really need is a way to get my mails, once. Once you have the mail, you can back it up to an infinite number of places (Git repository, anyone?) if in the future you need to search it.
So really, the only thing I need is 1) to receive mail, 2) to filter the spam, and 3) to keep a backup of my mail somewhere.
Considering this, why do we even need domain-specific mail? Like, myusername at Gmail dotcom, for example. I don't need it sent to GMail... I need it sent to me. I don't care what server receives it. I don't even need to store my mail there once i've read it - I can keep it offline, and back it up to remote repositories to search. With a format + protocol like Git, this would be fast, efficient, reliable, secure, and compatible.
So really, if we just had a distributed decentralized peer-to-peer mail network, a unique address system, and a retrofitted mail storage protocol (IMAP5?), we could send mail anywhere, receive it anywhere, store it anywhere, and spam could be filtered by whatever product or company was hosting your Git backup. With the new address system we could even build in personal crypto keys and teach people how to send real, honest-to-god, secure mails, potentially even anonymously.
Now somebody tell me how someone already thought of this and how it won't work :-)
Urbit seems to try to provide a centralized, personalized model for application services. I don't think centralizing is the way to go, mainly because my entire digital life is dependent on de-centralizing all my services. They try to make it out to be like some kind of container you can put anywhere, but that's like saying we should all use one kind of bag for everything we ever need to carry in our daily lives. There's good reasons I have 10 different kinds of bags at home.
Fighting spam (solely) by analyzing its contents turns out to not really work that well. You also need to be very picky about which SMTP servers you will accept mail from.
In practice, this means that if you do not route your outbound mail through a reputable hosted mail provider, it is significantly likely that messages will not make it through to their recipients. Even if you do SPF/DKIM/etc correctly. SMTP servers are not interchangeable. They have reputations, and the IP address blocks they reside in have reputations.
That doesn't always mean Fastmail/GMail/Live, etc. Businesses do run their own Exchange servers successfully. They do so from high-quality IP address space which is closely guarded, not handed out to any idiot with a cable modem (most recipients blacklist all residential IP address space) or $5 (cheap hosting providers are similarly suspect).
>I can keep it offline
You are referring to POP, which predates IMAP - the server just gives you the new emails since last checkin, and then they are gone (from the server's perspective). If you use multiple computers, or a computer and a smartphone, this really sucks. Most people are not going to hack together Git and rsync; they want the same view of their email on whatever device. So we get IMAP.
I'm not referring to POP or IMAP. I can keep my mail offline in any format I choose, anywhere I want. It's just text records. I can keep it in a .pst file, in a Maildir, in an mbox, behind an MUA or any mail retrieval protocol/service. My point was that using a repository like Git, you can simply store your mails (after you receive them, using any mechanism you choose) and then replicate them anywhere, and use any method you choose to interface with the repository later. Think of Git as your mbox or Maildir, and IMAP5 as the protocol+service to interface with it.
The reputation system is a relic of domain-hosted mail. We don't need to rely on it, it's just necessary for the current mail paradigm.
Crypto signatures from/to recipients could help this. You could register your personal signature with a notary, and the network could validate it before sending or receiving. You can then extend the cryptographically-secure identity into any other system you want to determine whether you want to accept the mail.
--
Example 1:
User A wants to send mail to User B. User A and User B both have crypto signatures distributed through notary services on the network. User A connects to a New-SMTP (NSMTP) service called "ZomboCom" and initiates sending a mail. The ZomboCom service checks the signature on the mail, and it's valid on the network. It also checks the recipient signature, and it's also valid on the network. It then accepts the mail. Then, ZomboCom can route the mail, or advertise it for delivery.
User B uses three different NSMTP services - "Flooz", "Webvan", and "eToys". User B has registered their signature with them all, and each validated the signature, and all of them agree to receive mail on behalf of User B.
An additional service can exist on the network which routes users to registered services - a Message Routing service. Flooz, Webvan and eToys can all register themselves with the Message Routing network as valid recipients for User B. When ZomboCom wants to deliver a message to User B, it can simply query which providers are listed as recipients for User B. It can verify they are real providers by cryptographically verifying the records against User B's public crypto key, which is in a different network service, the Public Key service.
So, ZomboCom connects to each of Flooz, Webvan, and eToys, and sends them a notification that there is a message waiting to be delivered for User B. Each one determines they do indeed receive mail for User B, and each then connects to ZomboCom and initiates receiving the mail. They each check User A's signature against a list of revoked signatures, and a separate list of "bad senders". If there is no bad flag, they receive the mail and store it.
Flooz, Webvan and eToys all store the mail in a Git repository for the user. The user can then retrieve their mail by merging each service's repository into their own. They can purge their repositories, edit history to remove old mail, whatever. Since every message is cryptographically signed/hashed, duplicates are easy to identify and ignore. If any one provider stops working, the other will have received the message already.
Example 2:
Spammer wants to send mail to User B. Before the message can make it into the network, they need a crypto signature to sign their mails with.
Even if they run their own NSMTP server, all the recipient servers will be verifying Spammer's crypto signature before accepting. Each server can in turn subscribe to lists of revoked signatures, so if Spammer abuses the privilege of sending and is flagged as a spammer, these different recipient servers are just not going to accept their mail.
Of course, we're still using IP addresses on the low level, so you can always reject connections from certain IP blocks if you want.
--
I should note that almost all of this could be implemented using existing infrastructure. The Git repository and and I...
Is this part of a more general attack on Internet infrastructure today in the U.S.? http://downdetector.com/ has been showing many sites with issues (Google, Outlook, etc.)
Just going off the public numbers for one of google's ASNs[0], with the amount of public peers they have (and likely far more private peers, and I'm just assuming google doesn't buy transit and ignoring that), it's unlikely that a DDoS would ever cause an outage for more than a small subset of people. You may be able to saturate a link or two, but that's not difficult to route around generally.
That is of course only for network saturation DDoS, I'm sure there are ways google could be ddosed at the application or server level, but they likely have enough infra in place to be able to eat the attack without anyone outside of google noticing.
Given the timeframe, I commend them for keeping the notices open and public. It's nice to see. When I went through the A(zure)pocolypse a few years back, didn't see anything for about 15-20 minutes... though admittedly if I weren't in the middle of testing something may have not noticed for a while either.
All said, you can't mitigate all DDoS easily, and it's nice to see that they were pretty responsive and open... Also, while email can be very important, it shouldn't be eminently critical.
42 comments
[ 2.0 ms ] story [ 127 ms ] thread1.) Run your own SMTP infrastructure. Setup SPF/DKIM/DMARC. Realize your outbound emails still don't always reach their destination. Also you have to fight inbound SPAM.
2.) Use gmail or Google Apps. Things just work. Cede control to Google.
Google is not without its problems. Just now on HN, in fact: https://news.ycombinator.com/item?id=12008365
I'll stick with Fastmail is very reliable and independent. They can't really be blamed for a DOS attack.
That doesn't really hold when their competitors have teams of engineers working on DDoS mitigations and successfully handle most of these attacks. Falling to a DoS is a technical failure just like other causes of outages.
Until they don't, and then good luck with getting ahold of someone who can actually do anything to fix your problem(s).
In my experience, Google support is _excellent_ for their paid products.
Also, you can buy your own domain, and have Google merely act as the SMTP relay and temporary storage. They can also forward your mail to another server for you.
It gets more difficult if you are providing a service that has to have uptime guarantees or are providing mail to many users but if you take the time to learn and educate yourself on current standards, hosting your own mail for fun and for profit is a doable thing that more developers and admins should do.
We are at a time when we actually have relatively easy to use software to managing mail servers, so us it.
For what it is worth, I have not had problems with Google, Microsoft, Yahoo, or domains that use their services whether it comes to sending or receiving. Sometimes a server is stuck in an SPAM prevention queue or I might have to whitelist a particularly silly server, but that doesn't happen very often.
And if someone wants to DDoS you, you're a lot more vulnerable than a major provider like Fastmail.
Personally, I use a hybrid solution: I use Gandi's SMTP servers for outbound and inbound mail, but I run my own IMAP server for unlimited storage under my control.
They do something similar with their webmail service, but the data is encrypted so it can't be read by a third party.
https://www.fastmail.com/about/reportabuse.html (last paragraph)
EDIT: Fastmail is fairly priced (for me) and i like the features they offer but i wish they wouldn't do this (or rather, i wish they would do the same for the SMTP service as they do for the webmail service)
Personal users aren't of interest to DOS ransomers since a call to the ISP will drop their traffic at the border. Without SLAs costing me money, as would be the case with a big provider, I coulf outlast the DOS. Just inconvenient and annoying.
So here's what I know about Fastmail that I want to know about any provider I host with:
- I know Fastmail will refuse any requests from the US govt to access data because they're Australian and legally the request has to come from an Australian court
- I know Fastmail's servers use encrypted storage
- I know the specs of Fastmail's servers (they post them on their Help/FAQ)
- I know Fastmail's actually trying to make email better, obvious by their invention of the JMAP protocol
- I trust Fastmail to be able to recover from any serious issues because they actually have on staff developers of Cyrus, the open source IMAP software they use. This means their admins have actual recourse when Bad Things^TM happen, vs the usual when an admin runs out of options, eg, "let's just post on mailing lists and hope we can find an answer"
- I know the fine details of how their spam filtering works, because it's publicly documented. (and it's quite well integrated with some tricks I couldn't employ at my last ISP job as I didn't have developers to assist)
- I know their infrastructure is primarily hosted in NYI with the backup in Iceland.
- I know they are serious about security, as they've been proponents of full SSL/TLS vs STARTTLS which could be MITM and downgraded (yes, many MTAs will let you require STARTTLS, but there are always possibilities of client bugs that could be exploited when you let an attacker intercept plaintext and inject data before the upgrade to a validated TLS session)
- I know how their backups work, because it's documented and I also have the ability to undelete emails which almost no provider gives the end user.
- I know their support is responsive and competent, as they've actually fixed Webmail bugs and put them into production for me within 48 hours
- Fastmail does PUSH email on iOS, while GMail, Rackspace, and most other providers don't offer this because it requires custom integration with Apple's Push Notifications service.
tl;dr yeah, the average provider might promise the moon but can they actually deliver when the shit hits the fan? will they actually strive to please their users and make the internet a better place? probably not.
Amsterdam was also hit by last night's ddos. They were hitting mx, and our secondary mx is in Amsterdam.
We all know mail is insecure. Unless you look really really hard, you aren't sure if the mail you received was spoofed or modified, a child can spoof mail and any MitM can modify it. So in general you can't trust your mail anyway, even if it's received by a reputable company. Sending mail is almost just as subjective... a random ISP's mail smarthost is just as good for getting your mail delivered as a hosted mail provider.
All I really need is a way to get my mails, once. Once you have the mail, you can back it up to an infinite number of places (Git repository, anyone?) if in the future you need to search it.
So really, the only thing I need is 1) to receive mail, 2) to filter the spam, and 3) to keep a backup of my mail somewhere.
Considering this, why do we even need domain-specific mail? Like, myusername at Gmail dotcom, for example. I don't need it sent to GMail... I need it sent to me. I don't care what server receives it. I don't even need to store my mail there once i've read it - I can keep it offline, and back it up to remote repositories to search. With a format + protocol like Git, this would be fast, efficient, reliable, secure, and compatible.
So really, if we just had a distributed decentralized peer-to-peer mail network, a unique address system, and a retrofitted mail storage protocol (IMAP5?), we could send mail anywhere, receive it anywhere, store it anywhere, and spam could be filtered by whatever product or company was hosting your Git backup. With the new address system we could even build in personal crypto keys and teach people how to send real, honest-to-god, secure mails, potentially even anonymously.
Now somebody tell me how someone already thought of this and how it won't work :-)
> distributed decentralized peer-to-peer network
> a unique address system
> personal crypto keys
Funny. You just described a perfect fit for Urbit.
In practice, this means that if you do not route your outbound mail through a reputable hosted mail provider, it is significantly likely that messages will not make it through to their recipients. Even if you do SPF/DKIM/etc correctly. SMTP servers are not interchangeable. They have reputations, and the IP address blocks they reside in have reputations.
That doesn't always mean Fastmail/GMail/Live, etc. Businesses do run their own Exchange servers successfully. They do so from high-quality IP address space which is closely guarded, not handed out to any idiot with a cable modem (most recipients blacklist all residential IP address space) or $5 (cheap hosting providers are similarly suspect).
>I can keep it offline
You are referring to POP, which predates IMAP - the server just gives you the new emails since last checkin, and then they are gone (from the server's perspective). If you use multiple computers, or a computer and a smartphone, this really sucks. Most people are not going to hack together Git and rsync; they want the same view of their email on whatever device. So we get IMAP.
The reputation system is a relic of domain-hosted mail. We don't need to rely on it, it's just necessary for the current mail paradigm.
Crypto signatures from/to recipients could help this. You could register your personal signature with a notary, and the network could validate it before sending or receiving. You can then extend the cryptographically-secure identity into any other system you want to determine whether you want to accept the mail.
--
Example 1:
User A wants to send mail to User B. User A and User B both have crypto signatures distributed through notary services on the network. User A connects to a New-SMTP (NSMTP) service called "ZomboCom" and initiates sending a mail. The ZomboCom service checks the signature on the mail, and it's valid on the network. It also checks the recipient signature, and it's also valid on the network. It then accepts the mail. Then, ZomboCom can route the mail, or advertise it for delivery.
User B uses three different NSMTP services - "Flooz", "Webvan", and "eToys". User B has registered their signature with them all, and each validated the signature, and all of them agree to receive mail on behalf of User B.
An additional service can exist on the network which routes users to registered services - a Message Routing service. Flooz, Webvan and eToys can all register themselves with the Message Routing network as valid recipients for User B. When ZomboCom wants to deliver a message to User B, it can simply query which providers are listed as recipients for User B. It can verify they are real providers by cryptographically verifying the records against User B's public crypto key, which is in a different network service, the Public Key service.
So, ZomboCom connects to each of Flooz, Webvan, and eToys, and sends them a notification that there is a message waiting to be delivered for User B. Each one determines they do indeed receive mail for User B, and each then connects to ZomboCom and initiates receiving the mail. They each check User A's signature against a list of revoked signatures, and a separate list of "bad senders". If there is no bad flag, they receive the mail and store it.
Flooz, Webvan and eToys all store the mail in a Git repository for the user. The user can then retrieve their mail by merging each service's repository into their own. They can purge their repositories, edit history to remove old mail, whatever. Since every message is cryptographically signed/hashed, duplicates are easy to identify and ignore. If any one provider stops working, the other will have received the message already.
Example 2:
Spammer wants to send mail to User B. Before the message can make it into the network, they need a crypto signature to sign their mails with.
Even if they run their own NSMTP server, all the recipient servers will be verifying Spammer's crypto signature before accepting. Each server can in turn subscribe to lists of revoked signatures, so if Spammer abuses the privilege of sending and is flagged as a spammer, these different recipient servers are just not going to accept their mail.
Of course, we're still using IP addresses on the low level, so you can always reject connections from certain IP blocks if you want.
--
I should note that almost all of this could be implemented using existing infrastructure. The Git repository and and I...
- Available anywhere
- Guaranteed notarized sending, guaranteed total ordering of messages
- Privacy and security built into the protocol.
The incentives aren't there -- what would miners get in this scenario? Also doing it in a way that retains perfect forward secrecy will be tricky.
Could be a startup idea.
That is of course only for network saturation DDoS, I'm sure there are ways google could be ddosed at the application or server level, but they likely have enough infra in place to be able to eat the attack without anyone outside of google noticing.
[0] https://www.peeringdb.com/net/433
All said, you can't mitigate all DDoS easily, and it's nice to see that they were pretty responsive and open... Also, while email can be very important, it shouldn't be eminently critical.