> on August 8th, our Security Operations team was made aware of a customer who claimed their password had been reset, without their initiation.
> One of the first discoveries was a non-DigitalOcean email address that appeared on a regular email from Mailchimp on August 7th.
> Soon after we discovered an issue with our Mailchimp account on August 8th, we initiated contact with Mailchimp, both via traditional support channels and other escalation methods. On August 10th, we had our first actionable response
> both via traditional support channels and other escalation methods
I wish our industry could just be honest that many meaningful escalations have to happen via discriminatory back-channeling, contacting friends, family, former co-workers, ANYONE who might have an in with the organization.
Its discriminatory because if you don't know someone you can be SOL.
Mailchimp/Mandrill have specially bad customer support and business user experience. At one point they closed the account of a company I was working at without any previous notice and without any recourse. They just sent a blanket email "Your account has been suspended". They left us scrambling for an alternative (we had both transactional and marketing email). We happily migrated to SES for transactional email and SendGrid for marketing.
This is because of fraud. Everyone either does this, or enables the fraudsters. The only third option is to get rid of email, except the fraudsters would just move on to the next thing.
Assume that you’ll have availability problems with email and engineer with that in mind.
Claiming that you either need to ban users without recurse or suffer from enabling fraudsters is why people start feeling like corporations don't have any human employees anymore. If you care about your customers, you could absolutely reach out and verify if the user you are about to/have blocked is a fraudster or not, and then act accordingly.
The reason most just say "You're banned, bye" is because they don't want to do that work, and subsequently don't care about their users one bit.
They ban thousands of accounts per day. Every day. If you can solve this problem you will have discovered a license to print money, so by all means try. But the problem is much harder than you are making it sound.
On the other hand, how many do you think would reach out when they get blocked? Most actual fraudsters wouldn't. Of those that do, you can take a look at what they are sending and quickly see if they are actually doing fraud or not.
I could say the same thing to you, you're making it sound harder than what it is. But if you're set to the mindset of saving money, I understand minimising work makes sense.
Of course they do. It’s just email/twitter/telegram/WhatsApp, and fraudsters are literally professional messaging automators. They open support cases, email execs, I have seen cases of them paying call center workers to call support lines and complain. Fraud is a serious business run by serious people, many of whom have a decade plus of experience.
You cannot attempt to stop fraud with manual effort if you operate at any kind of scale. Full stop.
> Assume that you’ll have availability problems with email and engineer with that in mind.
That goes for all dependencies, not just email. And not just third-party dependencies either.
If PyStorm died today, I could carry on in VIM. I could rebuild my entire architecture on DO if AWS sours. If git takes a dump, I have four machines from which I could copy the code and migrate to e.g. mercurial. And if I get hit by a bus, my code is well documented and testable.
> Its discriminatory because if you don't know someone you can be SOL.
Thanks for explicitly explaining that part. Now that you've put it that way, it's a pretty apt term but I probably wouldn't have been able to figure out because that term is usually used for things like gender, race, etc. and my brain immediately jumped to that. Maybe I am just stupid, but sometimes it shouldn't be that hard to understand what people mean and it helps to be more explicit.
Good opportunity to talk about it. An awful lot of people are extremely solitary. I’d wager this is more of a problem than racial discrimination because:
- Lonely people aren’t visible, by definition,
- Solitude increases racism, so it would be worth solving,
- They often end up creating companies and being one’s boss.
Especially when you generate one-off passwords with enough entropy to busy an attacker until the heat death of the universe. OTPs are not meaningfully increasing security in that scenario.
A password is something you know, but password reset is a bypass for that factor. They could drop the option of password reset and thereby eliminate that vulnerability, but I presume they don't because that would inevitably result in people getting locked out of their accounts.
That's not something fundamental to passwords. It's just that the culture of security in the industry seems to be more willing to lock people out of their account when they lose the factor they have than when they lose the factor they know.
> It's just that the culture of security in the industry seems to be more willing to lock people out of their account when they lose the factor they have than when they lose the factor they know.
It could be that losing what you have is easier for people to remedy than a forgotten password, but somehow I suspect that it probably comes down to which one generates the fewest angry customers/calls/support tickets. Maybe it's easier for us to blame ourselves if we forget to carry a token or our cell phone?
To anyone who reads this, works in Mailchimp and happens to be in a leadership position at the company, sincerely, it would be nice to see you go bankrupt and fuck right off the face of the landscape. You don't even have the sheer size pretexts of Google, for example, to justify such terrible responsiveness, shitty customer service and generally awful account suspension and service practices. Is it simple laziness or a sharper sort of contempt for your own customers? Disgusting tendency among too many tech companies that reach any moderate size.
> I would prefer that they fix their problems instead. They have 800 employees. I don't want those people to lose their jobs.
This thinking is how one rationalizes "bailing out" companies which have objectively failed in the market and should suffer the consequences.
Never has it been so easy to form a business and employ people. Gone are the days of a business being some kind of precious thing worth preserving because forming one is so involved.
Let the failures experience negative consequences, it's how we improve as a society.
What is the "current market" that you're talking about?
When a company goes bankrupt they give maybe 2 paychecks. How long until these people get new jobs? Maybe the company gives 4 paychecks? Still, how long is that going to last?
EDIT:
I'm seeing now that nobody on HN has ever been on the receiving end of your company going bankrupt. Carry on.
(Sincerity not snarkiness, and wasn't a downvoter) Do you not have an emergency fund? It was always important, but hopefully 2020 was a wake-up for most people.
> You don't even have the sheer size pretexts of Google
There is no excuse. None. Not for google. Not for anyone. In fact the other way is far more justified. When you're as massively profitable and market dominating as google if you don't respond well and fast you need to be broken up. Now. And Google clearly do need to be broken up. But for many, many other reasons too.
Just to hell with this excuse of "we're so successful we don't need to try because you're nothing to us!" Straight to hell. "You make so much money and dominate the market so comprehensively we must hold you to a lower standard." Seriously.
I completely agree with you on this, my point in my original comment was that Mailchimp doesn't even have this ambiguous and very shifty pretext of it being difficult to handle customer service with a humen response when your users number in the billions. Google has no excuses either at its enormous size and scope, and with annual revenue of $258 billion for 2021 (and a very healthy percentage of that being profits) they could at least make a sincere effort. It is however even less justified in a company like Mailchimp, whose users actively pay for the expectation of decent service and number in the low millions instead of billions.
Quite the opposite. Logistics matter. even to a Google, scaling human responses for user problems is extremely difficult when those users are in the billions. There will always be lots of difficulties unless AI gets radically better. I still think we fundamentally agree and Google should definitely ensure real responses at least to anyone who spends over a certain amount with it, but to me Mailchimp has far fewer excuses. These are paying customers of a multimillion dollar company and only a fraction of such people require support at any given time. Yet they still treat these users like shit sometimes for no good reason.
edit: and I have dumped Google. I use it for exactly nothing important in my life. It is possible and doable.
Mailchimp is a multi billion dollar company with 15 million + customers now. How many support people do you think it takes to respond to hundreds of thousands of customers? I think you're asking for India style call center customer service instead of their local customer service. I imagine the amount of people who use Mailchimp and need support is much larger than you think. The name of company and brand probably attracts the majority of their customers. What sort of technically adept customers do you think make decisions based on something looking fun?
> I use it for exactly nothing important in my life. It is possible and doable.
This is actually delusional. Email. You don't get a choice there. And your ad-blocking is unlikely to be 100% effective. How many websites you visit use google stuff? Never clicked on an amp link? That's just for starters. The decisions of others prevent you from living google free unless you leave modern society and find a nice place in the wild.
But more importantly, _No!_ Logistics do NOT matter at all in responsibilities. It doesn't matter how difficult it is. When you're the lynchpin in something and you don't deliver, screwing up someone's business or life to a non-trivial degree you are 100% responsible for that. You can't do it, GET OUT. "The logistics of driving drunk are really hard so it's ok when it goes wrong." Nope. It's not ok. Don't do it. The end. If you can't take responsibility for providing a critical service because you want huge scale you have zero business doing it.
Having more tolerance for smaller players trying to compete is fair. Having tolerance the market dominating force with the decreasing long run average total cost curve making billions is just madness. Break them up. Why are they in phone os development? Why isn't mail spun off from search? Why isn't 3rd party advertising spun off to a separate entity from those two? And so on.
"Because it suits their market dominance" has always been an argument for breaking up this kind of market power not in support of it.
MailChimp has been bleeding talent for years. They've had bad move after bad move, and then got acquired by Intuit. Their longtime CEO just stepped down. It's essentially just a brand owned by a megacorp at this point.
Happy for a service which only lets you look at 3 days worth of logs until you pay more just to see more logs?
Besides, their technical skill is pretty poor when their site shows "page not found" of some sort on log in process for a split second and when you try to search through the logs, they will quickly show you that I've made excessive access after less than 10 searches.
They had an incident on themselves and I asked them to resend the emails that they failed to send and support couldn't do that and that got us off of SendGrid.
Large free plan limit is the only good part about SendGrid.
> their technical skill is pretty poor when their site shows "page not found" of some sort on log in process for a split second and when you try to search through the logs
This is typical of single page apps. They have a default state of "no data" and then they update it when they get a response.
Wildbit is fantastic but unfortunately the new owners are known for very poor products and service. Fingers crossed though because Postmark has been the beacon of hope for a couple of years now.
Mailgun still didn’t automatically rotate DKIM keys last time I looked, otherwise I’d rate them much higher. That one thing creates a detail I don’t want to have to worry about.
> Mailgun still didn’t automatically rotate DKIM keys last time I looked, otherwise I’d rate them much higher.
Not only do they not automatically rotate them, there's no functionality to manually rotate them either. The only thing you can do is delete the configuration for your domain entirely and recreate it, which of course nukes your Event log. Hopefully they are working to address this...
They had another product called Mandrill that did transactional email, they discontinued it, but also allowed old customers (which also paid big $$$) to keep using Mandrill, perhaps that is what DO used?
We left Mandrill because they had a DB failure which took them a long time to recover, and we felt that all their focus were on newsletters, and that transactional email didn't get any attention.
You’re right, they mention Mandrill in the article.
I didn’t realise they’d discontinued it. I remember there being an uproar some years back when they ditched the free tier and made entry level pricing $20/mo.
Since then Mailgun also went paid-only, but it’s still practically free. They don’t send an invoice unless it’s over a certain amount, and mine never are.
I was evaluating Mailchimp for my newsletters at that time. They basically put all the features I was using behind a paywall without notice forcing me to pay that fee. Since I only have a small number of subscribers, It was at that point I started to rethink using a SAAS solution for that purpose and went back to sending newsletters from my Mail client.
Tried Mandrill/Mailchimp transactional email earlier this year and it was trash, we already had an established marketing account and they insta banned me for sending a single test email to myself before we even started using it. They flagged the account for "suspicious access" with no way to resolve even though I followed all their steps.
It seems that most of these transactional email companies concept of "100% delivery guarantee" excludes you from that guarantee... they aggressively and pre-emptively protect their servers from being blacklisted by autoblocking you first based on some heuristics and then very slowly or possibly never unblocking you manually later through useless support staff. Guilty until proven innocent.
Settled on Postmark (after trying pretty much all the popular transactional mail services without success), which has so far been golden, customer support will actually help you resolve an issue if there is one, plenty of forewarning with transparency before any blocking, the front end is exemplary of clear usable UX. It also lets you sandbox transactional mail into "servers" so that one bad source can't affect another.
They have a fairly extensive API, I used it in a small business, my main problem with them was they kind of just sat on their laurels and didn't keep up with other companies in the space. Once you get a few thousand contacts (who in our case were very casual "want to keep up but don't want to buy a lot" low-dollar-value types), you're paying a large sum for a lot less than competitors give you with a full marketing automation suite. We've since frozen the account.
As an indie developer, these increasingly frequent security disclosures (although yes good) are getting very frustrating.
I want to say this is due to the threat landscape expanding by the day but some part of me suspects that when a service provider becomes 'comfortable' (mailchimp, Heroku, Twilio, etc.) they becomes complacent/cut costs in the security department.
The other day I got a clear phishing SMS from REVOLUT! Crazy!
You would think Digital Ocean being a cloud hosting provider they could do email themselves, secure customer data and not provide it to third parties. Great write up though.
> We have migrated our email services to another provider and are completing thorough security reviews to confirm our vendors’ security posture.
Must suck for Mailchimp to lose a big account, but I guess that's not suprising. Mailchimp is going down by a thousand cuts - they could have stayed a great company if they wouldn't have focused on growth so much (I mean they now offer online ship builder and appointment scheduler products).
I do wish DigitalOcean would support WebAuthn/FIDO2. Meaning I could use my Yubikey and other hardware tokens I have.
Instead they only supported TOTP (Google Authenticator is one implementation) second factor which is vulnerable to phishing attacks. But still better than SMS or nothing at all.
On a similar note, Azure only supports U2F with yubikey on select Windows / macOS environments, but Firefox ESR on Debian is not supported at all. Every other service I use supports U2F just fine on Linux, but Microsoft wants Linux users to live slightly more insecure.
Ah Mailchimp. The king of terrible customer service.
I was their paid client some years ago, never have I treated as badly (though Convertkit came a close 2nd). People keep recommending Mailchimp, when they are one of the worst companies for support.
And funny to see big million dollar corps are treated the same way us plebs are-- at least Mailchimp dont discriminate!
91 comments
[ 3.9 ms ] story [ 147 ms ] thread> One of the first discoveries was a non-DigitalOcean email address that appeared on a regular email from Mailchimp on August 7th.
> Soon after we discovered an issue with our Mailchimp account on August 8th, we initiated contact with Mailchimp, both via traditional support channels and other escalation methods. On August 10th, we had our first actionable response
I wish our industry could just be honest that many meaningful escalations have to happen via discriminatory back-channeling, contacting friends, family, former co-workers, ANYONE who might have an in with the organization.
Its discriminatory because if you don't know someone you can be SOL.
I no longer recommend or use Mailchimp.
Assume that you’ll have availability problems with email and engineer with that in mind.
The reason most just say "You're banned, bye" is because they don't want to do that work, and subsequently don't care about their users one bit.
I could say the same thing to you, you're making it sound harder than what it is. But if you're set to the mindset of saving money, I understand minimising work makes sense.
Of course they do. It’s just email/twitter/telegram/WhatsApp, and fraudsters are literally professional messaging automators. They open support cases, email execs, I have seen cases of them paying call center workers to call support lines and complain. Fraud is a serious business run by serious people, many of whom have a decade plus of experience.
You cannot attempt to stop fraud with manual effort if you operate at any kind of scale. Full stop.
If PyStorm died today, I could carry on in VIM. I could rebuild my entire architecture on DO if AWS sours. If git takes a dump, I have four machines from which I could copy the code and migrate to e.g. mercurial. And if I get hit by a bus, my code is well documented and testable.
Thanks for explicitly explaining that part. Now that you've put it that way, it's a pretty apt term but I probably wouldn't have been able to figure out because that term is usually used for things like gender, race, etc. and my brain immediately jumped to that. Maybe I am just stupid, but sometimes it shouldn't be that hard to understand what people mean and it helps to be more explicit.
- Lonely people aren’t visible, by definition,
- Solitude increases racism, so it would be worth solving,
- They often end up creating companies and being one’s boss.
[1] https://news.ycombinator.com/item?id=32398773
[2] https://news.ycombinator.com/item?id=32392935
Interesting write up and 2FA by default sounds like a sensible move.
If you’re the type of user to have a DO account, you should be perfectly capable of using 2FA.
A number of accounts would have been accessed were it not for 2FA.
The best password won’t save you if a reset email is intercepted.
Something you have plus something you know should be what we cone to expect from any service that deals with anything of importance.
If both the email and DO account are behind strong, unique passwords, that won’t happen from a mere DO email leak.
Now, if the same, weak passwords are used for both accounts, sure. That’s terrible practice. But it’s also not what the GP was talking about.
That's not something fundamental to passwords. It's just that the culture of security in the industry seems to be more willing to lock people out of their account when they lose the factor they have than when they lose the factor they know.
It could be that losing what you have is easier for people to remedy than a forgotten password, but somehow I suspect that it probably comes down to which one generates the fewest angry customers/calls/support tickets. Maybe it's easier for us to blame ourselves if we forget to carry a token or our cell phone?
I would prefer that they fix their problems instead. They have 800 employees. I don't want those people to lose their jobs.
This thinking is how one rationalizes "bailing out" companies which have objectively failed in the market and should suffer the consequences.
Never has it been so easy to form a business and employ people. Gone are the days of a business being some kind of precious thing worth preserving because forming one is so involved.
Let the failures experience negative consequences, it's how we improve as a society.
You're thinking everything is some kind of an existential crisis.
You're the one thinking a company going bankrupt and 800 employees being out of work is some kind of existential crisis.
Companies fail, employees lose jobs get humbled and go find new work with lessons learned. It's how this is supposed to work.
What is the "current market" that you're talking about?
When a company goes bankrupt they give maybe 2 paychecks. How long until these people get new jobs? Maybe the company gives 4 paychecks? Still, how long is that going to last?
EDIT:
I'm seeing now that nobody on HN has ever been on the receiving end of your company going bankrupt. Carry on.
The one with all the layoffs and hiring freezes?
There is no excuse. None. Not for google. Not for anyone. In fact the other way is far more justified. When you're as massively profitable and market dominating as google if you don't respond well and fast you need to be broken up. Now. And Google clearly do need to be broken up. But for many, many other reasons too.
Just to hell with this excuse of "we're so successful we don't need to try because you're nothing to us!" Straight to hell. "You make so much money and dominate the market so comprehensively we must hold you to a lower standard." Seriously.
Dumping mailchimp? Ok. Dump google? Good luck with that one. You see it?
edit: and I have dumped Google. I use it for exactly nothing important in my life. It is possible and doable.
This is actually delusional. Email. You don't get a choice there. And your ad-blocking is unlikely to be 100% effective. How many websites you visit use google stuff? Never clicked on an amp link? That's just for starters. The decisions of others prevent you from living google free unless you leave modern society and find a nice place in the wild.
But more importantly, _No!_ Logistics do NOT matter at all in responsibilities. It doesn't matter how difficult it is. When you're the lynchpin in something and you don't deliver, screwing up someone's business or life to a non-trivial degree you are 100% responsible for that. You can't do it, GET OUT. "The logistics of driving drunk are really hard so it's ok when it goes wrong." Nope. It's not ok. Don't do it. The end. If you can't take responsibility for providing a critical service because you want huge scale you have zero business doing it.
Having more tolerance for smaller players trying to compete is fair. Having tolerance the market dominating force with the decreasing long run average total cost curve making billions is just madness. Break them up. Why are they in phone os development? Why isn't mail spun off from search? Why isn't 3rd party advertising spun off to a separate entity from those two? And so on.
"Because it suits their market dominance" has always been an argument for breaking up this kind of market power not in support of it.
It seems SES or mailgun are the primary options these days.
Besides, their technical skill is pretty poor when their site shows "page not found" of some sort on log in process for a split second and when you try to search through the logs, they will quickly show you that I've made excessive access after less than 10 searches.
They had an incident on themselves and I asked them to resend the emails that they failed to send and support couldn't do that and that got us off of SendGrid.
Large free plan limit is the only good part about SendGrid.
This is typical of single page apps. They have a default state of "no data" and then they update it when they get a response.
Mailgun still didn’t automatically rotate DKIM keys last time I looked, otherwise I’d rate them much higher. That one thing creates a detail I don’t want to have to worry about.
Not only do they not automatically rotate them, there's no functionality to manually rotate them either. The only thing you can do is delete the configuration for your domain entirely and recreate it, which of course nukes your Event log. Hopefully they are working to address this...
It's kind of funny that Mailchimp treats a company as large as Digital Ocean as if they're a one person newsletter.
We left Mandrill because they had a DB failure which took them a long time to recover, and we felt that all their focus were on newsletters, and that transactional email didn't get any attention.
I didn’t realise they’d discontinued it. I remember there being an uproar some years back when they ditched the free tier and made entry level pricing $20/mo.
Free options seem to be drying up.
It seems that most of these transactional email companies concept of "100% delivery guarantee" excludes you from that guarantee... they aggressively and pre-emptively protect their servers from being blacklisted by autoblocking you first based on some heuristics and then very slowly or possibly never unblocking you manually later through useless support staff. Guilty until proven innocent.
Settled on Postmark (after trying pretty much all the popular transactional mail services without success), which has so far been golden, customer support will actually help you resolve an issue if there is one, plenty of forewarning with transparency before any blocking, the front end is exemplary of clear usable UX. It also lets you sandbox transactional mail into "servers" so that one bad source can't affect another.
Why wasn't two factor authentication required to reset the password? This is Security 101: Greater risks need greater authentication.
I want to say this is due to the threat landscape expanding by the day but some part of me suspects that when a service provider becomes 'comfortable' (mailchimp, Heroku, Twilio, etc.) they becomes complacent/cut costs in the security department.
The other day I got a clear phishing SMS from REVOLUT! Crazy!
Also some nonsense in there about Crypto scammers?
Must suck for Mailchimp to lose a big account, but I guess that's not suprising. Mailchimp is going down by a thousand cuts - they could have stayed a great company if they wouldn't have focused on growth so much (I mean they now offer online ship builder and appointment scheduler products).
Instead they only supported TOTP (Google Authenticator is one implementation) second factor which is vulnerable to phishing attacks. But still better than SMS or nothing at all.
Afaict, the bug is that Firefox doesn’t support FIDO2. AzureAD also doesn’t support U2F, which is unsurprising.
I was their paid client some years ago, never have I treated as badly (though Convertkit came a close 2nd). People keep recommending Mailchimp, when they are one of the worst companies for support.
And funny to see big million dollar corps are treated the same way us plebs are-- at least Mailchimp dont discriminate!