Tell HN: I accidentally ran up a $1000 Heroku bill
I was experiencing a spike of traffic when my site was on national news a few weeks ago, and I turned my dynos and workers way up on Heroku to handle the traffic. The spike went down, but I forgot to turn off the dynos.
Flash forward a few weeks, and I am stuck with $1300 bill on a site that I'm not making any money on. I asked Heroku if they could help me out a little... I'm a good customer, and even though I'm a cash-strapped in-the-red bootstrapper, I gladly pay them hundreds of dollars a month to run my sites, bills payed on time and in full. I was hoping they'd maybe give me a little discount... not pay for my servers, but maybe not take a profit off of my stupid mistake.
They shot me down. No refunds available, even though these were virtual servers, and didn't contribute any real load to any of their machines. Although obviously they provisioned machines for me, so they had real costs.
Part of me wants to be upset, but obviously I was the one who screwed up. Still, it leaves a bad taste in my mouth. I guess a smaller host might offer more personal care, but I wouldn't get Heroku's world-class engineering baked into their platform, which I really, really appreciate.
What do other folks think? Am I just a dumbass who lost a chunk of paycheck, or is this poor customer care?
117 comments
[ 0.32 ms ] story [ 68.3 ms ] threadI imagine that tons of their user base (especially for small hobby sites that only occasionally, if ever, see high traffic or run background tasks) look for this, and yet there's nothing pointing to it other than old posts on the mailing list. At least put information and resources for autoscaling in the Docs section of the site.
Acquiring resources only as long as you need them, and then releasing them when you're done, should be the standard service - complete user control should be the advanced feature set. They already do this for websites on a single dyno: they spin them down when they're idle for a while. But they don't do it with workers (what effect does the latency of starting a backup job up really have? it's already running in the background).
But my question stands: why isn't this stuff very visible - i.e., on Heroku's DJ doc page: http://docs.heroku.com/delayed-job , or at least linked from there? Or published as an official piece of code on the Heroku github page?
I'd certainly be happier paying more per pro-rated unit of time that I actually use than I am feeling as though I'm in an arms race to not be fleeced.
Is there really that much revenue to skim off the top? Either way, they're paying Amazon out of those charges.
But I'm not so cynical as to think that their success is reliant on this income.
I'd be extremely surprised if Heroku actually spools up EC2 instances the moment you move the slider. It'd be a far more intelligent (and profitable) design to simply make sure there are sufficient resources to maintain the desired service levels with a buffer to deal with spikes in demand.
It should be relatively straightforward to provide service levels that are functionally identical to actually starting each "dyno" or "worker" with a greatly reduced cost structure.
Relying on customers overspending due to technical deficiency seems like a pretty poor business model.
I didn't mean to say they're relying on it, simply that it's adding to the bottom line and it would be greatly missed.
just because it is the cloud doesn't make it the IT version of big rock candy mountain.
There comes a time in a young man's life when "haha, whoopsie, I'm a kid who is not responsible for his own decisions" is no longer sufficient for a do-over. I am not sure when that is, but if you're running a business, well, welcome to being an adult.
So, the adult business owner would likely split the bill with the customer to cover his costs, fore go his profits on this particular occasion, have a very happy customer (who might even write about that!) and would apply the lessons learned to their business.
As a normative question, though, for-profit businesses do not have a generalized right to make their mistakes into others' obligations.
More broadly, one of my periodic crusades is trying to get HN people to understand that norms in the business world are very unlike the norms of cash-strapped twenty-somethings regarding things like a) the appropriateness of charging money (charge more) b) how much money $1,000 really is (a lot to you, a lot to me, rat spit to businesses with an HR department) and c) expectations for professional behavior (you're doing taxes like a boss, reading contracts like a boss, capacity planning like a boss, paying your invoices like a boss, etc).
This limits the amount of damage a customer can do to himself and to the company between billing periods.
I think Heroku is the party here that stands to benefit the most from the encounter, and by simply charging the customer they've lost an opportunity for good PR and have made half of HN more wary of dealing with them.
I understand what you're getting at, but just like there is a maturity to being 'in business' there is a maturity in dealing with your customers, and in this case both parties carry part of the blame, so the reasonable thing to do is to find a solution that lies in the middle.
Even the pros mess up occasionally, and in this case it was pretty clear that it was a mistake.
The real lesson here is that Heroku should implement a notification system when a customer has excess unused capacity or a min/max setting for their dynos with automatic increase and decrease depending on the amount of traffic a customer is currently receiving.
As you say, you 'do this all the time', so that means that you know exactly what is in your interest as a business person, and if that means that you occasionally eat a charge that must mean that you know exactly what the value of goodwill is.
Heroku, apparently has not yet learned that lesson, and 'heroku' and 'unexpected large bill due to mistake' are now two concepts that have become strongly linked. It did not need to be that way.
The OP never argued that he has the right to force the cost of his mistake onto Heroku. He's polling a community of business owners to see how they feel it should've been handled.
If I were Heroku management, I would've written the OP something like this:
(Valued Customer)-
I'm so sorry your accident created a nasty end of the month surprise. We understand, but we cannot forgive the whole amount because while they were not servicing requests, we incurred real costs keeping the dynos ready, willing, and able for you.
Since it's your first mistake of this sort, and you've been such a great customer, we're willing to reduce your total amount owed to $x.
We hope this helps, Heroku
----
so yes, I think it's his own damned fault. But I also think it's a missed opportunity to create a happier customer.
So what do you do; lose money each time this happens? Or take a hard line "we're nice guys and all but, sorry, it's costing us too".
And the happy customer might write about you; but the risk is it sets a precedent. So the next month when another customer screws up they think to try the same tactic - and if you refuse them it's even worse press ("oh, apparently I'm not good enough to be treated that way", "2 tier customer service" etc.).
It's a minefield all round.
What's more interesting is if this happened before and how they dealt with that. What's the cut-off point at which you want to stop your customers from doing serious monetary damage? $1000? Apparently not. $10,000 ? $100,000?
There comes a point where the continued existence of Heroku would be at stake, so it is in their own interest to guard against this, especially when you take in to account the number of users they've got.
They need some guardrails for this, and sooner probably better than later. Terms of service will not protect you if your customer can't pay your insanely large bill.
In that case you'll end up eating the charges anyway, so guardrails are to mutual benefit.
Heroku should almost certainly have user defined cut off points. They should require every new user to set a maximum monthly bill to something they can afford (or "unlimited").
I did exactly that for a similar type of service I worked on. One of the biggest anxieties customers had was that they'd get a massive unexpected bill, just like this guy did.
Heroku's best move would be to cut his bill in half and institute a cut-off feature. Every service like it should follow suit.
To which I answer: Yes. It is reasonable to run a Enterprise-oriented cloud scaling company which second-guesses your customers' decisions at every point. It is also reasonable to run an Enterprise-oriented cloud scaling company where a professional engineer engaged in capacity planning clicking OK to an estimate presented in 60 pt Arial is assumed to know what they're doing.
Less like a straightjacket and more like an (optional) seatbelt.
"I was the one who screwed up" doesn't like hating to me.
It's nice when you can cushion the user from the bad consequences of their own decisions, but you can't always do that. They may have been able to do that in this instance, but it's hard to know from the outside.
And then what? They shut you off? That would go over just as well! Here's a thought: understand what you sign up for and take some responsibility for your actions.
If you don't, we're haggling over the price, not over the principle.
Sure, haggle away. I see no problem with this. I don't see the need for Heroku (or anyone for that matter) to support a user's desire to haggle. I do think, however, that a warning should be given if your usage surges suddenly so that you can react appropriately.
If you set a loss limit with your broker, when you hit it they'll automatically sell all your shares and just leave the money in your account.
It's an interesting line to walk. If they give the poster a refund and he posts goodwill posts about it on Hacker News, Twitter, etc. Then, next thing you know, every customer who isn't using all of their capacity is asking for refunds and if they don't give them, then the PR backlash will be even worse (think AT&T and the free micro cells they're giving to some of their best customers but not others).
I'm guessing the majority of their profits come from people who don't use their full metered capacity -- in the same way giftcard profits come from people who never redeem them. Of course a google-like model that provides traffic-based metering would be best.
For Herokus sake I hope that that is not the case.
Typically a service like this is 'oversubscribed' to about 10 to 15%, it's like pre-paying for your parking meter, you want to pay a little bit too much because of the uncertainty in when you'll get back to the meter but not so much that you'd be cheaper off parking somewhere for a flat rate fee.
If they really do make the majority of their money on unused instances then they're setting themselves up for trouble in the longer term.
I highly doubt they did not think that one through.
The user sets up those limits, it cuts costs because they don't need techs to set it up. But the client shoulders the responsibility too. They last thing their company needs is to hire people to handle personal requests like funerals or "i forgots" which would only add more to their overhead. They above comment was perfectly honest with them and not at all condescending.
In terms of "wrong" ? You get what you paid for. Not every business model can include white glove service with freebies and "We understand".
Spoken like someone who has never had a real customer. There will always be people who expect instant response to support tickets. There will always be people who expect you to fix their webapp, or their apache config when you are selling them a 'no support' plan. People can be unreasonable. If you have a reasonable number of customers, some of them /will/ be unhappy, no matter how good your service actually is.
Now, for me, I avoid some of these problems by making people pre-pay. If you don't, I just don't turn up your server. If it's time for renewal, and you don't want to pre-pay for the next period, I shut you off. There's no way for you to get an unexpected bill, and if it turns out my service doesn't match your expectations, I offer you a refund if you are willing to leave. (You'd be surprised the number of people who don't take the refund and want to stay even after complaining very loudly and publicly.)
For my target market (companies with small revenues and hobbyists) this is absolutely the best way to do it.
On the other hand, there are cases where some businesses would prefer to just pay through the nose to survive the slashdotting or what have you, and that's where this pay as you go model comes in. The problem is that when you have expectations of a 'pre paid' customer but end up buying a 'pay as you go' service.
I've had real customers and you do what you can to keep the loyal ones and make them happy, even if that means cutting your profits temporarily. It's not just being nice, it's smart business. If he finds another hosting company and starts recommending them, that's way more money lost than the $200 they could have knocked off the $1300 bill.
My policy is that if I give one person a refund/credit (without them leaving... anyone can get the last month back for any reason if they want to leave, I mean at least once.) I give all customers who experienced the same problem the same refund or credit. A few months back when I was hit with a DoS and was down 8-10 hours? I gave everyone a free month, not just the complainers. Fourteen grand that DoS cost me.
The thing is, your complainers are expensive customers. On a low margin service, it does not take very many "fix my apache config" complaints before a customer becomes unprofitable. I mean, you want some complainers; they act as the canary in the coal mine. But keep in mind that the real money is in the silent masses who silently leave when they feel wronged.
I mean, focusing on higher maintenance customers is also a valid business model... but you need to charge more, because high maintenance customers are more expensive to deal with. I want to be clear that I am only talking about (and I only know about) lower-margin services where the actual cost of providing the service is the determining factor in price. There are markets where those rules don't apply, and in those markets, nothing I'm saying makes any sense at all (and your advice is probably pretty good... not that I'd know.)
Is the latter part of that true? (I'm not playing Devil's Advocate, honest!)
I'd assumed the lion share of Heroku's profits were coming from clever arbitrage of EC2 instances rather than markup alone.
For example, if they have 1000 users paying $108 per month for 4 dynos, they don't keep 4000 "dynos worth" of EC2 instances running constantly. Instead they can see the overall load and maintain that lower level of EC2 instances plus some "slack." When traffic increases among apps, the slack can be rapidly configured to pick it up. Heroku would then make not only a markup on EC2 but also profit from the difference between the maximum load and actual load. (Isn't this how cloud scaling systems work anyway, right? Heroku could/might be doing that over thousands of apps for collective benefit.)
Perhaps I'm over-thinking how I'd be trying to squeeze profits out of a Heroku-type system though ;-)
The way I understand it is if no one is making requests to your app they will shut down the dynos, and when you start getting requests they'll spin them back up, up to the number of dynos you're paying for. That's why if your app has been idle for a long time it takes a few seconds for the first request to complete.
This lowers their costs, and presumably the prices for their customers.
I could be wrong though, and perhaps they only turn off dynos for non-paying customers.
Since he's a good customer, they could even give it to him in the form of credit for their service, which gives him an incentive not to leave.
I believe if you take care of your customers, they'll take care of you. In general, the one thing that people will remember for life is how you make them feel.
The benefits of providing awesome customer service outweighs the cost IMO (I haven't done the math, but that's my gut feeling). I remember reading a story on reddit yesterday where a gamer experienced the almost exact same situation with Steam:
http://www.reddit.com/r/gaming/comments/dcnp7/one_of_many_re...
that provides a good example of how news of awesome customer service can be spread, and possibly grow sales (Zappos).
For example, I have several sites hosted with WebFaction.com and I'm unbelievably happy with their customer service. They respond within 10-30 minutes on any support tickets. I'll continue to use them in the future. (see how that was a good example of what awesome customer service can do?)
The example in question is more like: you make a reservation for September instead of October. In October you realized your mistake. Then you ask for your money back, after all you didn't use either the tickets or the hotel. And because the airplane and hotel weren't fully booked, you claim they didn't have any additional costs because of you. Well, good luck getting your refund.
After all, Heroku could have auto-detected idle instances easily and could have disabled them or alerted the user.
The whole 'dyno' concept without an automated scaling option is a bit strange. Why set an exact number of instances when web traffic is just about the most variable thing you could possibly want to measure. Even apache auto-scales the number of processes depending on how many workers it currently needs to serve your requests.
Auto-scaling and a web app hosting service go together like bread & butter.
I think the issue here may be more of tone. It seems from the post that Heroku just gave a cold "no". The user should really post the original exchanges. As a user, I feel a lot better psychologically when I know the other company tried its best--even if they could not help me.
Did Heroku try? I can't tell clearly from the post.
Scammers, as a rule don't go for solutions with a high up-front investment in their time, they'd be much better of using a few thousand stolen credit cards instead. Why bother asking for refunds on a real life ID when you can get away with someone else holding the bag...
Heroku should sidestep this question anyway and implement auto-scaling and notifications on certain thresholds asap.
And how could this be a scam? They can easily see the traffic on the instances in question. The scam is that I only pay them $300 this month?
They are a company. IMO posting their email is fair play. It's not an individual.
And how could this be a scam? They can easily see the traffic on the instances in question. The scam is that I only pay them $300 this month?
I never said you are a scammer. You're not. But this potentially sets a precedent for future users to be reckless and want the same favor granted to you.
On many many occasions, businesses have literally given me money even when it was my stupid mistake. Some examples just over the past few years:
- Sprint credited me $100 when I racked up a $250 bill by not watching my usage
- Bank of America credited me $100 when I left my card in the ATM and someone withdrew $100 from my account
- Hyatt Place gave me a free room when I made a reservation on the wrong date (and never showed, or called)
- Southwest Airlines gives you full credit if you cancel your flight any time
- ATT gave me all my rollover minutes when I changed plans (by rule, you lose them - i had over 2700)
Those are just some examples that I can remember...I'm not sure it's a healthy mentality to expect this, but it certainly leaves a very positive impression. Leaving your customers in a state of feeling ripped off when you can obviously help is not the right business decision.
This is especially important with a service such as Heroku, where users will be changing these settings all the time and so there's a high probability of human error.
Perhaps you intentionally provisioned a large amount of headroom for a forthcoming expected/possible traffic spike.
In that example it would be annoying to get "are you sure you want to have this extra resource running?" emails.
(sure, you should be building the system to auto-provision but that would have required a better programmer and you just got the interns to build it, etc)
Akamai comes to mind as an example of a company that has always had similar issues. If I ever were to use them in the future, I would have to factor into their monthly costs the partial FTE required to manage their usage and costs. More importantly, if there are other CDNs that offer better cost management features, I would almost certainly use them before ever using Akamai again, unless there was no other technical option.
Guess who didn't turn them off.
So he casually mentions it to me and tells me that he racked up a 500 pound bill, but his CC didn't have any money in it so it was fine. I asked "so did you turn them off?" He said "no, not really, but it's okay because they can't charge me".
So he racks up another 500 pounds, and only turns the instances off when he finds out that he didn't use the inactive CC but another, active one. Long story short, he's 500 pounds out (Amazon were kind enough to refund the first 500, which disappeared as soon as it hit his account, since he owed another 500).
Guess who forgot to close this box and after 2 hrs of exposure now has to worry about cancer and/or infertility in later life. True story.
Look at it this way: at least your friend took CS and not Physics and thus is only £500 in the hole.
loads of graduate students in the physical sciences are regularly given access to stuff that dangerous or worse. if we've got to have somebody watching over each and every one of them, graduate education will grind to a complete halt.
That's why the first couple of times somebody more experienced will check everything about your harness.
This was a Masters level student, so I don't feel like the instructor has any blame here
I like a very simple rule: Customer retention is far more important than customer acquisition. It may be a flawed rule, I'm not sure since I'm not a successful business person. But it sounds good to me. It seems vitally important to retain your good customers.
If it were me, and a good customer made a mistake, I'd eat the cost to keep the goodwill.
I didn't get any sort of break though. :( June was a lot of hotdogs and Budweiser till I could dig out of that hole.
Heroku might optimize for this usecase by detecting spikes somehow and notifying you that you still have machines running, but other than that, I don't see how any of this is their fault.
You're talking about "a few weeks": did you check your expenses in the meantime ?
I would put some reminders (google calendar etc) or habits in place to regularly check your expenses.
I suppose it depends on what sort of shop Heroku wants to run. It just seems like customer service is not a big deal to them.
Imagine how big a raving fan you would have been if they had credited the entire thing. That is a tiny cost compared to acquiring a new customer and turning them into a raving fan.
"I am stuck with $1300 bill on a site that I'm not making any money on." So that means if you make money on the site, then you'd pay? What's the difference here? You're saying that because you're poor now, you don't have enough integrity to say "oops I did something wrong" and take the responsibility?
I'm myself a boostrapper burning thru my savings and to tell you the truth, I do everything myself to cut on expenses. I'd love to use Heroku, but they are too expensive for me to run. So if you don't want to make another mistake, then start to accept responsibility and learn to take care of problems yourself.