HN Explorer

8 days downtime: Cloudflare r2 subscription bug ruins my business

76 points by hriis ↗ HN
Our website has been severely impacted for 8+ days because I can't add the R2 Subscription back to my account.

The error is: "You cannot add or modify subscriptions or services until all invoices are paid. Download each "Unpaid" invoice on your Billing page and click the "Pay online" link within each invoice to fix your account"

All my invoices are paid, I contacted Cloudflare support but literally no one is answering. I don't know what to do anymore.

They closed my old ticket then opened a "report" where I had to verify my identity with Stripe. After I did that they didn't respond anymore.

Ticket number: #17419451

44 comments

[ 0.26 ms ] story [ 94.6 ms ] thread
cpach
Can you restore your data from backups and move to another storage provider?
hriis
unfortunately not, i dont have access to the storage at all, also backups wouldn't be a good idea (storage size is 800gb, we have a 100k+ user website where users frequently upload files)
trollied
800gb is nothing. You should be backing it up.
hriis
might be a little too late now.. also idk if cloudflare even supports backups, i'd have to make my own
Nullabillity
Cloudflare-managed backups wouldn't help you against Cloudflare revoking access.
ccakes
It's a hard lesson to learn like this, but it's a learning moment nonetheless. You don't rely on your provider to make backups for business-critical stuff like this, you need to manage it (and test!) it yourself.

rclone[1] supports tons of storage providers. If you get access back and recover your business, start making backups onto other providers.

[1] https://rclone.org/

kikimora
I would say the lesson is just use AWS which has outstanding support. Building backups yourself, going multi cloud or else just erase the benefit of CF. If you do this you can just run on a bare metal and save on cloud cost.
cpach
AWS is very good, but you could get locked out there too. With no backups in another place, the ability to do disaster recovery is at risk.
FaridIO
If your business depends on it, not backing it up in some other place on a consistent basis is a bad idea. The cost of this should be factored into the price you charge people. 800GB would be less than 20 dollars on S3, all you have to figure out is the replication process, but that shouldn't be rocket science either.
RKearney
I’m currently ~6 weeks into a billing issue with CloudFlare. They typically take 5-6 days to respond, ask you for verification to do things, wait another 5-6 days, then they ask you to do it because it turns out they don’t have permission. It’s a mess.
koollman
if your business depends on someone else's entirely to work, you have to factor that into your risks and maybe plan for it.

No alternatives, no disaster recovery plan, nothing you could try for more than a week with some other provider ?

hriis
well, i never knew this could be an issue. the problem is that i dont have access to the r2 storage at all
koollman
external/independant backups are a thing. They would be part of a disaster recovery plan (and you should test the restore process and result from time to time, else they might be worthless when you actually need them).

I understand this is not helpful right now, but there are best practices and many ways to plan for this kind of stuff for good reasons. Some things can and will go wrong, and sometimes that may cost your business a lot, which means you have to anticipate (or accept) the possible costs and risks.

It is part of the job

vorador
This is very unhelpful. There's a difference between "someone fatfingered a command and brought this mysql machine down" and "our cloud provider shut down a core feature for our account".

When you design a new system do you plan for S3 to go down for more than a day? Do you have a fleet of offsite machines to smoothly transition to? If not, why not?

misiek08
If data on S3 is crucial for business to be alive? Of course! I would even think about writing data to 2 separate storages in real time to be able to switch easily. 3-2-1 backup is a no joke, especially when it's your business, not a hobby project. Having DR too. And it's not a legacy, 20 years old system, so it's much easier to do such plans and test them.
vorador
Do you run a multicloud system too? In case the AWS account gets shut down by billing issues?
viraptor
Running multicloud (at the same time) is for availability not disaster recovery. Just having another copy of the data outside is good for DR, even if you have to redeploy everything manually in a hurry.
koollman
You don't have to run it a entire duplicate of your current infrastructure, just plan for it to happen (depending on impact to your business). Then it means you can think about doing it after a few days of downtime, maybe earlier depending on difficulty/impact.

But, yes, you do have to plan to run it elsewhere and consider wether various features may make it harder for you to move on. vendor lock-in is a known business risk

koollman
Well, it is about things to prepare before being in this situation, I acknowledged already that it was unhelpful now, since it is too late. It is advice for other business deciders looking at this, or for this one for changes to implement in the future.

Sometimes the plan is "well, this business had a good run, now it is over".

Sometimes it can be as cheap as "well, here's the documentation to run all my stuff elsewhere, starting from my external backups"

Depends on how much money you lose from a failure and how likely the failure is. As an additional point, you may not plan for s3 going down, but even a price hike on egress traffic may put a business in trouble if trying to move without already having external backups. So, as often, you have to do some cost/risk analysis. But having backups not controlled by your main provider is often considered a good idea.

Technical problems, human error, cost changes, various disagreements between provider and customer, can all be made easier if you have a plan B rather than being stuck with a single provider or solution

Brian_K_White
"There's a difference between "someone fatfingered a command and brought this mysql machine down" and "our cloud provider shut down a core feature for our account"."

No there isn't. Those are both exactly the same problem with exactly the same solutions.

trollied
You appear to be lost, anyway. This isn't a random support site for cloud providers.

People will no doubt downvote me for saying this, but if everyone posted their woes on here this site would be next to useless.

It's also quite normal for such pleas to leave out lots of vital information, and just present a good looking view, rather than the actual history.

ceejayoz
> You appear to be lost, anyway. This isn't a random support site for cloud providers.

It often functions this way. Quite a few times with CloudFlare. It's not the first place to go, but it has historically been a helpful last-resort option.

For example: https://news.ycombinator.com/item?id=34639212

trollied
I am well aware of that. It doesn't change the fact that this isn't a support forum, and others being successful should not encourage the behaviour.
ceejayoz
Easier to say when one’s business isn’t at risk, I suspect.
Brian_K_White
The old couple who lived across the street from me for the entire 20 years I've been here both died within a few weeks of each other only a few weeks ago. I'm really sad because I liked them. I know this is relevant to you because to say otherwise would be easy to say when one's own friends didn't just die.
ceejayoz
If posting about loved ones dying to HN had a long history of successfully resurrecting people, this would indeed be a great analogy.
Brian_K_White
The fact that doing something wrong sometimes gets you what you want doesn't make it right. That gain came at everyone else's expense.

Everyone that ever left a grocery cart in the middle of the parking lot instead of putting it away "successfully" got something they wanted.

iJohnDoe
Victim blaming needs to stop.

After 8 days of downtime, most people get desperate and look for all avenues for support.

trollied
I'm not blaming the victim. I am telling the OP this is hacker news, and not a support forum.
viraptor
This is effectively a support forum, because you get visibility and issues do bubble up higher than the standard support queue. The system is what it does.
Brian_K_White
After 8 hours of downtime, I have already been back up some other place, some other way, in some other fashion, for 6 hours already.

There is no excusable way to explain being down for 8 days no matter WHAT any service provider does. How do you even explain the 2nd day let alone the 8th! Holy cow.

viraptor
CF CTO comes here to do damage control for uncontroversial issues. It's actually likely to get fixed this way if he sees it
koollman
I do think some exemples of "stuff that can happen to you" are useful (without becoming a support site, of course).

It may help others consider what they should be doing to not let this happen to them or imagine what they could do before it happens to avoid being stuck.

1over137
Honest question: why do you need CloudFlare, why not self-host?
hriis
well, r2 was cheap and "good" so i went for it, but i also use cloudflare to proxy my backend ip.
ivanvanderbyl
I have a similar issue with R2 billing, but related to enabling bucket events. Says I need a subscription, which I already have. I opened a ticket in ~March while they were having a huge billing outage. After a month of pinging them every 3 days for an update they closed the ticket without replying marking it as resolved.

I still don’t have access to bucket events.

I want to assume it must be an absolute mess internally where they can’t even reply to a ticket.

hriis
if this happens to me then i can just directly shut down my business, this is unbelievable
koollman
Do read contracts, service terms and SLA/guarantees (and other customer experiences, if available)
LinuxBender
The CTO's email is sortof in his profile [2]. I assume he would want to know if there is a business impact on someone assuming in good faith everything stated in the submission is accurate.

CEO & co-founder of CloudFlare. [1] Cloudflare CTO [2] Not listing their individual contributors.

[1] - https://news.ycombinator.com/user?id=eastdakota

[2] - https://news.ycombinator.com/user?id=jgrahamc https://jgc.org/

hriis
you really think he would respond if i email him?
LinuxBender
All I can say for certain is that he will not reply if you do not email him. Just keep it professional and state only the facts, is the only thing I can offer.
jgrahamc
Yes, I would. My email is jgc@cloudflare.com.
spaceheater
Why is this flagged? When cloudflare announces R2 there are hundreds comments praising their announcement-ad. Now that we get an actual review of what may happen, it gets flagged?