Beware Scaling on AWS in Early Days

31 points by Kalanos ↗ HN
During my first month on AWS, I used AWS Batch a lot.

AWS decided to bill me mid-month (for more than I utilized I should add) even though the cycle ends on the 31st.

The transaction failed. Instead of contacting me to resolve the issue, AWS completely locked me out of the account. Card is also now locked due to "fraud."

I've been trying to resolve this for a week now (closer to 3 weeks if you include quota-related tickets). It's Friday, and I have a demo with a Fortune 100 company next week. In order to get results for this demo, I need to run a lot of Batch jobs - not great.

I shudder to think about what would happened if this had been a launch with paid customers.

From their perspective, I get that crypto miners probably sign up and cancel their credit cards, but I was just using the cloud as intended. I had multiple tickets related to resource quotas that explained why I was using the cloud.

20 comments

[ 4.1 ms ] story [ 49.9 ms ] thread
You should really tell us who you are and your support ticket numbers, you are on the hacker news frontpage complaining about a digital service provider (support) failure, which means you are eligible for free premium support. At least this works for cloudflare, where it goes like „hi this is the cloudflare CTO, I will personally take care of this, this shouldn‘t have happened OwO“.

> The transaction failed.

Why? What would have been different half a month later?

> for more than I utilized I should add

Do you have details?

Hello, this is the Cloudflare CTO.
In this situation what would you do ?

Would you just spot OP some billing credits to make the account current while sorting out issues ?

While I have a host of projects dependent on AWS and Firebase, I'm always worried about being locked to a vendor. What if a vendor just decides they don't like me, what if I'm not able to migrate...

> What if a vendor just decides they don't like me

Why would that happen? Vendor lockin is a real concern, but worrying that AWS will decide they don't like you is not rational.

No vendor wants to earn a reputation of being that hostile to its own customer base, and so unless you are doing something that is likely going to raise eyebrows you'll be fine.

While I agree with you on principle, Google has earned just that reputation and I would never use Google Cloud for anything critical.
Okay that is a fair point, see Blogger, Stadia, Google Reader etc.
Those aren't example of Google deciding they don't like a specific customer and booting them off the system.
Right, but it was example of why GP would be unlikely to use Google Cloud or recommend Google cloud to others.
This happened to me with Twilio/Sendgrid. I used their email service on all my projects, never a spammer, only transactional. In fact I used it so little that I think they decided I was not worth it. So someday they closed my account. No email, I just noticed I could not login. Contact support and they ask me to provide a bunch of kyc info, it was a very annoying email thread, but basically all my email integrations were broken for no stated reason.

My account had 100% reputation, no billing issues and nothing in my account details was unverifiable.

Then the support rep told me that the account was closed by the system, maybe by mistake, but there was no way to reopen an account so I’d have to signup again and then open a support ticket for that account (I think it was to migrate data, no longer remember).

Annoyed, I try to signup again, open a ticket and the response was:

> We have reviewed your account and have determined that Twilio's services cannot support your account at this time due to one or more of the following:

> 1. Violations of Acceptable Use Policy 2. Violations of Twilio SendGrid Email Policy 3. Accounts with unresolved billing issues 4. Inability to verify provided information

> Thank you for your interest in Twilio SendGrid, best of luck with your future endeavors.

This was from an account with 100% reputation. What I think happened is they were purging low spending accounts. So yes, a company can definitely stop liking you. Gladly I was able to move very quickly to SES.

Did your use case violate 1,2,3,4?
It violated exactly zero of those policies.
Thanks for sharing your story.

Big companies also just make mistakes. They can accidentally ban you and not feel like fixing it.

Have you considered fixing support via Cloudflare instead of offering free diamond tier support to the lucky users whose complaints you read?

My company recently started to offer premium support for more money, how about that?

"Pay 100$ to receive a response within 12 hours!"

"Pay 500$ to receive a response within an hour!"

More money for you, less frustrating and potentially business-ruining outages for customers.

Last example was comically quick to find by the way.

https://news.ycombinator.com/context?id=39317352

> The transaction failed.

Why did it fail?

> AWS decided to bill me mid-month (for more than I utilized I should add) even though the cycle ends on the 31st.

There are specific items that are billed immediately, such as domain registrations.

+1 to this sentiment. You'll also run into similar problems if you need to request quota increases. For companies that AWS takes seriously they'll approve the requests in a few minutes. For startups they'll literally take days reviewing the request before some random person at AWS decides whether to increase your limit or not.
This is the type of stuff that keeps me using smaller cloud providers. Even medium sized companies don't need the scalability and flexibility of a full cloud offering like AWS (for the most part).
Which smaller players are you referring to, DigitalOcean and Linode?
Those and things like Heroku, Render, etc.

I remember spending months trying to get Google Cloud to increase a quota while I was forced to turn sign ups off in my product while I waited. I also remember asking Digital Ocean to increase my droplet quota and got a blind yes the next day.

I'm unwilling to invest in building a business on a platform where the lack of customer service can be a blocker.

How are you going to fix it? Create a second account? Azure/GCP?