Speaking of racketeering, it's an enlightening experience to search for "stresser" or "booter" providers (euphemisms for DDoS-as-a-Service) and look up their NS records to see who helping them ward off competitors DDoS attacks and keep their origin servers hidden. 9 times out of 10 it's Cloudflare, with the few exceptions being DDoS-Guard, who more or less specialize in facilitating crime.
I wouldn't be the least bit surprised if DDOS protection providers were using DDOS on their prospective customers via proxy. Problem, reaction, solution. Did you pay your "protection" money this month, Luigi?
What’s especially shocking is how closely coupled sales and engineering are. Like I get they talk, but for a sales call to end in engineering pulling the plug…
To me it's pretty clear: Trust & Safety (NOT engineering; they don't appear to be involved) likely raised an alarm saying "Customer X is breaking TOS - no immediate resolution available, but something might be possible given extensive legal & engineering review. Recommend switching to enterprise so we can study how to make it work."
In that light you can see why Sales would be sent as the messenger. But I agree they shouldn't have been involved. Sending a T&S representative would have been better. But then again it looks like from screenshot #2 that they kind of did that? They just didn't have a direct call with T&S.
Sounds like OP is a casino and plays domain games to avoid regulatory interest. Recommend reading article carefully before reacting to the headline. Hopefully Cloudflare provides a perspective.
The point is more that the author is an unreliable narrator and you need to apply a little salt to the rest of the story. Cloudflare absolutely shouldn't be taking bribes to permit regulatory evasion. But if they are, I want more evidence than a substack post.
It also seems strange they dont know their Traffic Numbers.
>Note that 80TB is the number they tried to sell us, I don’t know if it is accurate since they removed all our access to historical analytics.
I mean you dont need accurate Data but surely most would know by heart their traffic in rough figures? Or am I the old dog where every new Web Dev are so used to Cloud and Serverless they have no idea what they are using?
Eh, your traffic is a total cost you pay per month. That's how I would look at it. The one figure I know best of all is annual revenue, and how our annual revenue this year is on track to do compared to last year's.
As far as exact volume of QPS or TB/month or whatever, I really couldn't say.
Over 90% of our traffic is cached, since it is static assets. I can look up how much traffic reaches our origin, but the main factor is the number of static files hit. We used Cloudflare Analytics (part of the business plan) to track this, and since it didn't really impact our tech much until now I don't have an exact overview. I mainly know which (uncached) endpoints are hit how much. Fastly is currently saying 15TB per week which seems roughly the same range as Cloudflare's 80TB / month number.
People seem to have a very laissez faire take on egress which I’ve never understood given the really impressive markups the cloud providers charge on it. But yeah, it seems like the attitude is that as long as you’re using “cloud-native” services (AKA locked-in proprietary offerings) then cost is low and doesn’t matter anyway because it’s opex, not capex.
I spend a lot of time wondering if the Emperor is wearing any clothes.
Depends on your scale. I would probably know the traffic for the project I looked at last, but the whole account? No way. Half of it I've never touched and would have to talk to different teams. I'd only look at that when discussing the contract again. Or if their TAM flags us crossing some threshold.
It would be completely different for a small project of course, but once you're counting in TBs... it's less important.
Do you have a suggestion to make that not possible? It doesn't seem fair to punish them so aggressively because that might happen. The "may" there isn't a statement of intent.
I'm not defending Cloudflare's exact actions in this scenario, but it seems reasonable that there are cases where yes, for $10k Cloudflare is okay.
Risk can be mitigated, especially if you take care to know what the risk is, but risk mitigation and the salaries of the risk mitigation teams are not free.
The answer of "no, we will not host you unless you pay us enough money to hire people to make sure we're not breaking laws by hosting you" makes plenty of sense, and an online casino that is likely dubiously legal in many countries is definitely a place where you might use that answer.
I'd also expect there are cases where Cloudflare enter into enterprise agreements with customers, get a good hard look at exactly what's happening, and then tear up the agreement and walk away.
And all of that is fine when communicated properly. Even if OP is an unreliably narrator are we to believe they also left out some of CF's emails?
To me it looks like https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr... is entirely the wrong email to send in the situation and if you are as old as I am and come from where I come from, you will have flashbacks to "reading between the lines" of the party daily in the 1980s. The real content is at the bottom:
> As we have a very short window to report back to Trust & Safety team, please let me know if you can make time tomorrow
Big red flashing lights: the right questions are 1) why is T&S involved at all 2) What are their concerns which forces such a hurried deadline? 3) What are the consequences of missing this deadline.
The right email would start with something like this:
> Providing services to your business constitutes serious legal risk to Cloudflare. We are happy to work with you in the future if you are buying an Enterprise plan. As we need to commit significant resources to accommodate you, we need an annual commitment. Otherwise, with much regret we need to terminate our services provided to you as it is our right per Terms on date/time. ("We may at our sole discretion terminate your user account or Suspend or terminate your use or access to the Service at any time, with or without notice for any reason or no reason at all.")
T&S departments generally exist for one reason: to manage reputational risk. This sometimes involves legal risk, but it usually just means preventing relentless hit pieces about your company enabling something portrayed as horrible. This can result in customers and even employees leaving if the media is relentless enough.
Companies take risks if the reward is considered good enough. In this case, that reward is income from the customer (who can still be dropped if the hit pieces start getting published).
If it's legal but burdensome (somehow) to host a particular industry, requiring more money to deal with the increased burden seems reasonable. For instance, if their legal department needs to deal with complaints from various countries, that probably costs more than $250/month.
That being said, I doubt that's the core issue in this case.
A reasonable scenario to me seems to be: An automatic "upgrade to the enterprise plan" requirement was triggered, and then in the process of the sales calls to make that happen, Cloudflare got serious eyes on the customer for the first time (whereas at a paltry $250/month previously they wouldn't have), and realized exactly what line of business the customer was involved in, and decided to fire them.
I was rushing to judgment until I heard this... pretty plausible.
In support of your theory particular is I don't think enterprise sales "ragequits" a conversation when the customer is mid-evaluation based simply on the idea that they are considering multiple options.
Why would they walk away at this point, let alone ban the customer.
From the write-up I bet CloudFlare had it as a "60% to close" in their CRM at this moment. It doesn't make sense for them to drop the ban hammer in this moment.
PS: explanation or not, this is deeply shady behaviour from CloudFlare. Just perhaps a little less so.
> In support of your theory particular is I don't think enterprise sales "ragequits" a conversation when the customer is mid-evaluation based simply on the idea that they are considering multiple options.
> Why would they walk away at this point, let alone ban the customer.
It wasn't just that they were considering multiple options. Looking at the timeline, this was about a month after their initial soft gloves approach/enforcement action and they drug their feet the entire way through it.
Once CF got to the top of the leadership chain at their company and it was clear that all the relevant decision makers were involved in the conversation but were unwilling to pay, they just folded their cards, resumed the initial enforcement action, and moved on with their day.
If this was a small account they probably wouldn't have even blinked twice with just striking down the user for causing reputation harm and violating TOS but since they were a large account CF clearly went out of their way to meet with them multiple times and try to find a solution. But after a month of little to no progress while the account continues causing reputational harm and is unwilling to budge, they just called it quits and moved on.
It seems like the sales team went out of their way to try and land a $10k/mo deal. Then when they heard there was a second potential suitor in the mix they got upset and said “well we never wanted your $10k anyways!” and destroyed any chance of reconciliation. Very sour grapes/ no second date on tinder type of reaction.
If there is a TOS issue I’m not listening to a sales pitch on it. You better tell me what the issue is upfront in the first email instead of dicking around with the commission based workers. Like very low level stuff here imo
That's not true at all. That line of argument gets close to "if this product is free for open source, why is it not free for me? either it costs something to operate or it doesn't." You don't get to price the service.
I can reason my way into it, I think objectively. To protect their IP reputation, CF required BYOIP. This costs them something, and de-jure requires an Enterprise plan. Which for the customers usage costs $X. Is it right? Ehhhhhh. Does it follow corporate logic? Yeah. (Sales logic? YES)
If you think something your client wants could explode into a liability, you can turn them away or you can just make sure their bill covers your exposure.
If it's a legally questionable service, there's likely to be plenty of abuse contact, or they're going to be a big target of crime, they're going to end up paying more. This is the same reason why some industries (eg porn sites) have always paid more for card processing.
It's not just 10k a month. it's 10k a month for the plan that allows you to BYOIP (Bring your own IP addresses). That was cloudflare's issue.
Their business was causing IP reputation damage and all plans but the enterprise BYOIP plans share the same IP pool.
Essentially it was "use your own IP pool and pay us for the cost of maintaining that pool for you or GTFO".
This wasn't just a normal sales rep hitting them up. This was trust and safety (i.e. the moderation team) coming to them with a compromise that would allow them to stay on the platform. They chose against that and were dragging their feet.
The timeline of the article also really makes this clear. This wasn't over the course of 24 hours. This started a full 4 weeks prior with sustained back and forth. They only included a few images of emails from the discussions but the article makes clear that there was more discussion happening.
And to quote the article. After receiving the ultimatum, they got an entire extra week to deliberate.
> We managed to buy a week of time by letting it escalate to our CEO and CTO and having them talk directly with Cloudflare.
Then finally when they told CF that they were just buying time while looking to move elsewhere, CF dropped their act of goodwill and the moderation team resumed the moderation action they would have taken in the first place had this been a smaller account.
----
So yeah it sounds bad from the snippets but this was basically "hey you are a big customer and you are breaking rules we would normally ban anyone else for but if you can compensate us we'll spend the labor hours and infra to let you keep operating in your own little quarantine box.". So this really should be seen as an act of goodwill rather than malice.
You can't start the timeline from the first email, because clearly Cloudflare didn't communicate the actual issue to the customer. (Yes, the customer could be lying about what was said in that meeting, and they could have been told what the problem was rather than it being just Cloudflare trying to upsell them the enterprise plan without telling why. But then the "omg, we just discovered a problem with your site during a routine inspection!" email sent two weeks later wouldn't make sense.)
They also were clearly lying in those email messages: The second email says that domain rotation is strictly forbidden, but a few days later in the third email they're explicitly selling features for rotating domains more effectively.
And sorry, but a company selling "we'll override the Trust and Safety team if you pay us $$$" is absolutely unacceptable. There are only two options, both bad. Either they're not running a real TnS operation, but just pretend-staff one in order to run these kinds of shakedown operations. Or they're running a real TnS team that found a real problem but are letting sales people override the TnS team's honest judgement.
For $10k / mo paid 1 year in advance, your cloud provider does a legal review of the situation and figures out how to make your problem work on both the technical and legal level. It's not a "special plan", it's consulting.
Edit: "How do you know?" -- I don't know it's actually what happened, but when switching to enterprise, you don't go from 10% margin to 98% margin. The added costs actually represent added budget for the provider to deal with your "special case". ALL enterprise pricing tiers are disguised consulting contracts.
Assuming that is what was happening, why would CF suddenly be okay with an illegal site if they pay more? Might as well call it the criminal enterprise plan then.
It's 10k a month for them to set up a dedicated IP address pool so that they could BYOIP and buy their own IP addresses instead of getting the IP addresses in cloudflare's main IP address pool repeatedly banned or reputation harmed.
i.e. it's a $10k fee for maintaining the infrastructure for a quarantine around their services
They did. Repeatedly. You see it mentioned in the few emails the OP chose to share. But they also didn't share the other communication they had over the month long discussion they had with cloudflare.
> BYOIP also costs nothing to produce.
That's not really accurate. Cloudflare is entirely built around one big unified anycast network. If you want to provision an entirely separate network that maintains all the same features they are using from the main cloudflare network, that's going to require provisioning a lot of cpu time and routing table slots at a lot of different sites plus whatever admin and engineer overhead comes from maintaining this quarantined service.
This is literally all written communication they gave us. The only other thing was calls. Not calls with anyone that had any knowledge about what any issues were, but calls with sales.
The costs for announcing the customer's IPs and making sure that only they are the ones actually sitting on them would be minuscule. Nowhere near $10k/mo. Maybe $100/mo.
Anycast/unicast changes nothing here. Anycast simply means that the same prefix is announced in multiple different PoPs.
>If you do not use byoip with your service you cannot be a customer, and also pay us $120k upfront while we do not tell you why, it's only a 40x price increase. You have 24h.
As a network engineer I'm well aware of what it costs to add prefixes to what you announce over bgp, some miniscule additional CPU overhead, likely unmeasurable. Even slow mips processors could process full table bgp.
I mean, sure, they’re probably doing some sketchy regulatory dodging or whatever. Which part of this can Cloudflare solve by having them pay $120k/year to them?
Over-charging is a legal way to effectively deny service. When I'm offered jobs I don't want, I sometimes tell them my salary is 3x what I really need.
Hmm. My take is the casino structured its business to comply, not to evade interest. Further, I don't see how Cloudflare benefits by taking on the risk to charge more to help a customer avoid scrutiny. More like: they know it's a humming business and want a piece.
> We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries.
Evasion:
> Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available.
This is more like one gang hitting up another for "protection" payments. I had to laugh when they called it "Trust & Safety".
The way I read the screenshots of the emails from the articles seemed to suggest that something the authors company was doing was causing issues with IP reputation on CloudFlares range.
Them very aggressively highlighting the BYO IP feature and then even suggesting third parties to rent IPs from strikes me as a significant detour from their normal “script” (having dealt with their AU sales team before).
this is exactly what is happening. Cloudflare uses an anycast network, so IPs are shared by default.
this customer is damaging Cloudflare IP reputation which hurts other customers. Cloudflare can either fire the customer to protect other customers using Cloudflare IPs, or force this customer to use their own IPs and damage/manage their own IP reputation.
unfortunately this is expensive and OP is mad they can't do their legally fraught gambling operation on Cloudflare's addresses for free
They're mad that cloudflare cut them without real warning. And they should be! Anyone can get on a big company's bad side, and if there aren't extremely important messages being withheld by the author this makes it scary for anyone to use cloudflare.
If a custom IP is going to be mandatory, they need to say that and give a deadline, at the very least.
The IP-reputation damage is immediate. Cloudflare is choosing to pass the hard landing directly onto their customer instead of forcing their other customers to share the damage.
As a CF customer, I am happy that Cf is preventing another business from damaging mine.
The ToS doesn't say anything against gambling sites. Even if there was IP reputation damage, it's not appropriate to cut them off so immediately. Especially when they're a long-term paying customer.
If this is what's happening, the right behavior is to say that and terminate OP's service. Even if OP is in the wrong, Cloudflare did such a bad job communicating with them that they come off as extortionate.
Cloudflare could've just said so. Cloudflare also chose to make BYOIP expensive.
They could've explained the problem ("your gambling business is a problem for our IP reputation") and offered a solution ("we can switch you over to BYOIP so this won't be a problem"), but instead they sent in an army of sales reps that demanded an upfront payment for a product tier that they only needed one small part of, to the point of sales people pretending to be part of other teams.
It makes business sense to kick out casinos, but OP got fucked over by Cloudflare's shitty practices.
CF calls and says there is a problem with traffic. They want to push an enterprise plan. Customer says no.
CF calls and says there is a problem with domains. They want to push an enterprise plan. Customer wants to solve problem, dropping domains, making changes. CF says, only enterprise plan will remedy the situation.
There is obviously a sales script involved.
“get back to Trust & Safety"
Heard that story several times, it's always another team, e.g. "Licensing" that need to be satisfied, or that if you don't pay up, that team will be off the leash. Also heard the pay-for-a-year-upfront for several large vendors who pull this. The reason is, some sales reps need to make numbers, so they shake the tree and see who falls down:
"Cloudflare has absolutely no information on when they will force you into custom billing, but when they start "urgently" needing to talk to you you're probably not going to get out until you have a juicy custom contract with them."
> My take is the casino structured its business to comply, not to evade interest
It's impossible to say what's going on since it's an anonymous post with no details.
Maybe it's all 100% true.
Maybe there are some key details being left out. Wouldn't be the first time I've seen one of those outrage posts that seriously misrepresented things.
Whatever the case, obviously the author is not an unbiased party. These posts do well because "zomg Cloudflare bad!", and maybe they are, but I sure as fuck don't trust some casino guy either.
Thing is, this is also not the first time when Cloudflare T&S team has been disrupting their customers out of the blue. The post even has some links to other HN stories.
So the thing that stands out in the article, is that cloudflare's initial communications (and the final communications, when they moved to ban) implied issues with their behavior (trust and safety team, terms of service violations), but in between it sounds like the didn't talk about ToS at all, just sales team asking them to buy enterprise. Though it's possible OP is omitting some explanation given by as why enterprise plan would alleviate ToS issues.
Yeah, there's some pretty key info being left out. I don't doubt that Cloudflare communication sucks especially when dealing with their sales team (aka bizdev which is what OP was originally contacted by), but the second screenshot is pretty damning.
My guess: Their account fell out of the non-enterprise TOS for some reason which is being obscured in the post (probably domain rotation related). Their T&S team proposed moving to enterprise for a custom resolution. OP's company refused, their account was purged because they had gotten several warnings about it.
I'm sure this sounds frustrating to the average HN dev who runs a legitimate startup with cloudflare on top and is now biting their nails worried to death about what will happen to them. But "online casino" immediately raised a million alarm bells in the post.
I did mention the multiple-domains issue in the post. It would not have been amazing for us to remove our secondary domains, but we would have been very happy to do it if it had resolved the issue. We asked them again and again but they would not give us any detail or options apart from their 120k/year package. Note that BYOIP (which I guess they could reasonably have required to isolate us even if we only use a single domain) is available for a fraction of the cost elswhere (e.g. fastly).
Since we already left Cloudflare the only reason I finished writing this article is to warn others. I think it's still relevant to many companies regardless of what you think of casinos, since very unprofessional sales tactics (unprofessional as in business threatening) seem common place with them. Do look at the linked other posts and comments here from other people affected that don't have anything to do with casinos.
For me, the worst part is blackmail and account ban.
If you had legal presence in EU then new Digital Services Act[1] might a help for you. I am not sure if you could sue them based on that law, but you could maybe lodge a complaint.
"Now this needs a bit of context on what they are talking about. We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries. For example, many games are only available in some countries. Some countries we block completely. Then we have a few different domains that remove certain game groups or site features - for example our social features (chat, user tipping / interaction) or our sportsbook. Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above."
Looks like they COMPLY with regulatory interest, to me.
When it comes to laws and taxes, "comply" and "evade" tend to be synonyms.
"In order to comply with tax regulations and donor laws, we had to structure our activities in order to make it possible for political donations to be classified as regular consulting income".
At least from discussions I've had over the years with my accountants, comply and evade are very different. Evasion is when you are doing something that's explicitly illegal. Optimization and compliance is when you comply with the law while trying as much as possible to reduce your tax. In some cases, there's a bit of a grey area where you use multiple structures that according to your accountant should comply with the law but in a way that has never actually been tested legally. That last part tends to be named "optimization" rather than "compliance"
This is in part why good laws are more concerned with what “it” is rather than what “it” is called. Good laws define something and then name it, so that you can test a pattern of facts and determine whether “it” is indeed what the law covers.
For me, my corporate tax professional errs on the side of well tested strategies or those that the IRS has ruled on administratively, and that is perfect for my risk tolerance.
Well HN is the unofficially official Cloudflare Support forum. I think we will hear from them soon. From past experience normally their response time for anything Cloudflare on HN is within 2-3 hours.
If that's the take, that means Cloudflare is okay with 'breaking laws' so long as they can take a heavy cut of the ill gotten gains? </sarcasm>
Let's not try to find reasons to harm the messenger and stick to the facts -- a paying customer was suddenly extorted for hundreds of thousand of dollars out of nowhere.
Taking a step back, why would they even care if their platform is supposedly neutral and not responsible for the content ?
If they can indeed stop providing services to a casino, why cannot they shutdown a website spreading pro-war propaganda, or a website selling illegal services ?
It means they are making editorial choices, instead of just being the technological provider and being a neutral "internet pipe".
Not sure it's really in their best interest to self-police in the end, as they could lose their DMCA safe harbor provision ?
It’s not “editorial” to choose to stop serving (or charge more money to) a customer whose actions pose legal difficulties or risks to your business. If some country’s vice division contacts cloudflare legal due to a customer’s illegal online gambling presence, I guarantee that cop does not care about a US/EU copyright law.
The same thing is true for IP reputation, just without an external official complaining. If other CF customers are negatively impacted by one customer’s action, CF isn’t violating safe harbor by booting that customer or passing on the costs of mitigating that impact. That’s just running a business, not exercising editorial review of hosted content in violation of safe harbor provisions.
> Taking a step back, why would they even care if their platform is supposedly neutral and not responsible for the content ?
Because their main network all uses one big IP address pool and the blocks by various regions/countries against their site were probably not just DNS blocks but also IP address blocks.
So they now have an account whose activity is getting their IPs banned in countries where they operate.
So they told the account owners they needed to pay for an enterprise account and a dedicated IP address pool maintained by cloudflare. That's why CF kept talking about BYOIP in the emails.
i.e. "Pay for us to build you a quarantine with your own IP pool or leave ASAP"
This may indeed be the motivation but, at least in those emails that are presented in the linked post, there's no evidence that Cloudflare did at any point clearly communicate that this is indeed what the problem is.
I do encourage you to read the whole article cause there is some fine details in there. The main point is that we were happy to remove any domains apart from our main domain (which gets > 95% of our traffic) but Cloudflare did not give us that option or any other detail on the supposed issue.
This has been my experience with 80+% of these loud complaints about services, especially regarding "losing Google traffic". Dig into it just a little and you find out the complainer was doing something extremely shady that the service is often too polite/proper to call out in a public forum.
If 90 or 95% of their traffic comes from a single domain (and presumably has for a while), that still doesn’t make OP sound guilty. If there was a legal issue Cloudflare legal should’ve stepped in, not their sales team.
That was the part that bugged me. This workflow is very busted from a user standpoint, though I'm sure it works very nicely to Cloudflare!
It smells like the "problem" was detected by automation, but instead of being able to reach anyone technical to work through it, you can only call sales teams.
Using localized versions of your services to comply with regional laws and enhance user experiences (i.e. make money) is SOP for practically every international $bigco. Online gambling is regulated and legal in ~50%-70% of the world; without actual evidence to the contrary, it’s completely reasonable to assume that this is a legitimate business. I’m really struggling to agree with the “two sides to every story” replies being left here about how there’s likely shady activity going on behind the scenes, when to me the post read as candid and transparent about the nature of the nature of the business, the admitted legitimacy of CF’s TOS violation claims against them, and the content of the communications with CF.
My 2c: It’s scummy that CF did this. It looks like they were disingenuous about the severity of the violations and used it as an excuse to get more $$$ from an already paying customer to make the manufactured problem go away.
This shouldn't matter, in general Cloudflare responds to complaints about allowing illegal content with "we're a neutral utility, we forwarded your complaint to the site's webhost". To me, the article showed that Cloudflare was being extremely aggressive with selling the customer on an enterprise plan and repeatedly invented excuses to get them on the phone with their sales team. They then took the site offline and locked them out of their account when the customer started talking with other CDNs.
Nah, you have different domains so you can track and maintain flows, also the regulations might even stipulate having domains in the locale, the headline is very much accurate after reading the article.
Is the casino illegal in the jurisdiction they're based out of?
It doesn't seem so, so there is at least a valid reason for Cloudflare to keep them as a customer as they're not violating the laws where they have their business in.
Except Cloudflare position here is not to ban them but they want to get paid for it. You are shaming the OP and his business but the reality is that Cloudflare has acted in a worse manner and that should be highlighted.
Cloudflare was the company that went viral for the firing of an account rep not hitting her goals. I wonder if it’s overall indicative of not a great culture in terms of relationships with enterprise customers.
> Cloudflare was the company that went viral for the firing of an account rep not hitting her goals.
How is that something worth going viral over? Salespeople get fired all the time for not meeting their sales goals. Engineers similarly get fired all the time for not meeting their productivity goals. If you don't do your job well, don't expect to keep it.
And if I recall correctly, in this particular case, she was a green employee who hadn't even made a single sale yet! What more obvious of a layoff target is there than that? Would you keep a green unproven salesperson over a proven veteran salesperson who's landed 9 figures in sales?
While I agree with you, I think the call is for companies to be less psychopathic and stop onboarding people within 90 days of mass layoffs.
Especially in a world where people pick up their whole lives and relocate for jobs. Recent joiners aren't getting any sustainable kind of severance either. The idea is if you're hiring them you have a minimum commitment to support their success.
Yes she was an obvious fire, but it's also the organization's fault. Enterprise deals also take way longer to close than that...
All that said, salespeople can and do move jobs a lot. I'm sure she'll be fine.
It takes around that long to plan and for public companies they don't just suddenly come to that decision within a single quarter, but also that's missing the point.
If you're a public company and you go from healthy to mass layoffs within a single quarter then your investors (and employees) should be a lot more concerned.
>If you're a public company and you go from healthy to mass layoffs within a single quarter then your investors (and employees) should be a lot more concerned.
1. According to the CEO the layoffs in question were for "~40 sales people out of over 1,500 in our go to market org"[1]. Is that really a "mass layoff"?
2. Did you not remember that the layoffs coincided with reversal in macroeconomic conditions? Specifically, the reversal from "inflation is transitory" to "inflation is persistent and the central bank will hike interest interest rates".
90 days isn't enough time to close enterprise deals. The whole point is that the firings demonstrate that she was never set up to succeed by the organization.
>90 days isn't enough time to close enterprise deals
I'm not sure why you think she was fired/assessed on the basis that she wasn't able to close enterprise deals within 90 days. The same tweet seems to refute this by saying "we can often tell within 3 months or less of a sales hire [...] whether they’re going to be successful or not". Presumably they're looking at various indicators (eg. size/composition of the sales pipeline, reviewing her sales calls and/or emails) and using that to predict performance.
That wasn't how I interpreted this phrasing. I read it as "it is worth being critical of an org that does mass layoffs and then goes on to hire new people to fill vacated roles shortly after the layoffs were finished".
That timing shows that it's not just implementing headcount and budget reductions, which are somewhat defensible if still tragic. It was instead a forced turnover, which in some cases can be a route to wage suppression.
>That timing shows that it's not just implementing headcount and budget reductions, which are somewhat defensible if still tragic. It was instead a forced turnover, which in some cases can be a route to wage suppression.
Apparently the person in question was fired within 3 months of being hired[1]. If this is true it seems like a stretch to characterize it as "forced turnover" or "wage suppression".
She did say she was around for only 6 months and enterprise sales cycles can last 12+. Though I guess if you’re engaging in scammy behavior it can be much less… maybe she wasn’t willing to do that.
Sorry, but if you‘re onboarded in a Sales Team you‘re at first getting small fry customers to get used to your Job. Cloudflare is massive and a complete no-brainer for 95% of companies. Not only that, she couldn‘t even land a single enterprise upcharge. Cold calls are hard, but upsells are much easier because companies are pretty much vendor locked with CF.
If you can‘t land a single customer in 6 months under these circumstances, you‘re likely just not a good fit.
Source: I worked at a Fortune 500 company in b2b sales and solution/ integration engineering. I worked with anywhere between 10 mil up to a billion YoY customers.
You can even google my name to confirm this info. This is pretty much exactly like it went when I started.
In both examples you provided it was less "banned" and more "switch to a higher priced plan or we'll kick you off". In both cases they seem to be bandwidth related, and in the second case specifically they mentioned having hundreds of terabytes bandwidth but were upset for being told to upgrade to the $200 plan.
It's unclear as it shouldn't be possible to be cheap. That said, in a world of data, Cloudflare being a massive man in the middle (MITM) probably means something [1].
[1] Cloudflare decrypts your traffic, reads it, and then forwards it. They see all encrypted data going to and from your website, in plaintext.
I actually like Cloudflare very much from a technical perspective, so I wouldn't want to hurt them.
In fact, the company vision and values (as the team grew) may have changed over the time, but originally it seems it was somewhat of a different spirit (and closer to a data collection network).
* Cloudflare operates at a scale where its caching saves a lot of bandwidth, which saves ISPs money, which makes Cloudflare an attractive partner for peering and co-location.
* CDN is a platform on top of which Cloudflare can offer a lot of additional services that used to be expensive dedicated middleboxes.
I don't know how those situations work though. The customer is obviously allowed to say whatever they want, but if a Cloudflare employee or CEO disagreed, would they be allowed to provide their own version of the facts? Wouldn't that go against privacy rules in some way, showing the details of someone's account? I would think they would only open themselves up to legal trouble.
As far as I can see, the author was careful to redact their domain from all screenshots.
The recommendations to, basically, not keep all your eggs in one basket and have backups of config are surely good ideas. But if you have to plan on dropping cloudflare in some arbitrary 24 hour window, perhaps it’s CF that’s the problem. This sort of stuff and other recent articles about CF are so worrying that it’s now being run by the finance team (hence why every email they got in this article was from sales teams rather than any technical folks).
Also; if not registering domains on CF does anyone else do at-cost or otherwise super cheap pricing?
We already have DDoS protection from our colo provider and it wasn't clearly advantageous to switch to Cloudflare. We were mainly interested in zero trust but I don't know what we're looking at as of now; we still use various VPNs which, while not exactly fashionable, do work for us.
Wisdom of the ancients: Always make sure your domain services are completely independent from your other service provider(s), regardless of whether Cloudflare is involved. (Sorry, no recommendations at this time.)
This is a really important lesson here. Don't put your eggs in one basket, and if network delivery/etc. is core to your revenues and livelihood, don't trust a random third party host to look out for you.
10TB/80TB at 120k/yr, either way, Cloudflare is taking you for a ride.
If you aren't self hosting, you're really doing it wrong.
What is "don't put your eggs in one basket" here? Your DNS has to point to something... and changing it will go through propagation delays, during which it will be down if you are banned suddenly.
It's not like you can have your domain/DNS somewhere else and point to Cloudflare IPs (to not put DNS and CDN in same basket). Cloudflare does not allow that setup.
You can't protect your website from your DNS provider or hosting provider suddenly kicking you off. You are going to be offline for a couple of days.
> You can't protect your website from your DNS provider or hosting provider suddenly kicking you off. You are going to be offline for a couple of days.
Sure you can. Colocate in two or three places. You're your own DNS provider and your own hosting provider. If one of your colocation companies doesn't like what you're doing for whatever reason, you use the other two until you replace that provider.
"Do it all yourself" is a far cry from "don't put all your eggs in the same basket". The latter principle I agree with, the former not so much.
And you are still unprotected from your DNS registrar kicking you out, directing your domain to some "customer terminated" page until you can find another registrar, and have new NS records propagate (days).
You're not really making any points, or at least none that're relevant to this discussion.
Let me summarize: businesses do silly and sometimes stupid things for irrational reasons, or for reasons they never care to divulge. To avoid what happened that led to this discussion, the most suitable solution is to not be at the whims of companies that don't communicate well.
The legal requirements of domain registrars are clearly spelled out, unlike the TOS from, say, Cloudflare which leaves tremendous amounts to the imagination. These are not the same at all.
Im reading between the lines here but it seems like the traffic amount, the saas subscription tier, and the actions required to remidiate some issue were all unaligned.
1. Its quite possible thar CF having this site on some multi-tenant infrastructure could be threatening. Not unreasonable at least to ask them to have their own IP block.
2. If thats the issue then a clear explanation should have been provided. Routing to sales is inexcusable. Someone isnt being transparent.
3. If it’s a pure cost / revenue issue then say that, set a deadline and negotiate. This is bad karma and even though CF is clearly the market leader, what they do isnt rocket surgery. Not worth it.
What stands out as odd to me is that CF seems to be pushing away a $10k/month customer. No business can reasonably be expected to accept sudden price changes like that, even if they'd paid, they would've moved away within a year.
Given that the article is an online casino that seems to be using potentially ToS violating domain rotation, and that they pay so little per month for apparently millions of users, I for one will not form an opinion on CF based on this article before CF has a chance to defend itself.
My thoughts here are also all speculation, but when you mentioned multi-tenant issues my mind immediately went to a situation I've seen all too many times before:
- a companies ops team identifies a tenant that is too heavy/burdensome for multi-tenant infra and is causing issues. These issues can cost a serious amount of money if you factor in dev/ops times to resolve, other customers impacted, etc. Certainly more than what a hypothetical single multi-tenant customer could be paying
- they escalated internally and need the tenant moved to enterprise asap to resolve
- the only reason the tenant was on multi was because sales sold them the wrong thing, so now it's on sales to explain how to fix this
- improper handling internally results in this landing only on sales, with no backup, and with their task being to get them to take enterprise
- when the customer refuses enterprise they go "we've tried nothing and we're all out of ideas"
Again, this is totally speculation and I'd hope CF has more mature practices than this but this is a scenario I've seen before in much smaller orgs.
> This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above.
And the very next paragraph begins with:
> In any case, we receive >95% of our traffic through the main domain that’s been unchanged since our founding, and were happy to resolve this issue in whatever way...
And then they complain about paying up?
The only issue I see here is around the aggressiveness on the CF side. But, I was not in those meetings and the way above reads tells me that I might have been slightly mad so perhaps the CF was just taking it out on them?
- "This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above."
Attorneys love it when people put everything in writing like this.
If a country A decides to block twitter.com but forgets to ban x.com which remains available ... is Twitter engaging in illegality / violation of CDN terms of service?
Like most things in the legal system, it depends on intent. It's pretty obvious that twitter's rebrand to x.com was an actual rebrand and not some way of evading domain bans.
Right, I'm not arguing that they're guilty, just that the legal system doesn't operate off pure black and white rules like "if you have two seemingly unrelated domains then you're trying to evade bans".
>and they give a traffic number that proves they're not doing "domain rotation".
Are you talking about the part where they said "95% of our traffic through the main domain"? Without additional context behind that number it's a stretch to claim that "proves" they're not trying to evade bans. For instance if their country is banned in country A, but country A is a small country that only makes up 3% of their overall traffic, they can confidently claim "95% of our traffic through the main domain" while still theoretically using alternate domains for ban evasion purposes.
This has always been my concern about establishing a presence online. I've considered blogging about my experiences at work or the cool stuff that I've built and it feels impossible for me to know when I've crossed a legal boundary. How do I know for sure if I'm sharing proprietary stuff or confidential stuff. The lines of legality seem to get blurry real quick.
> Make backups of your configuration on Cloudflare. It's an unexpectedly large pain to recreate all those configurations
Better yet, configure CloudFlare through terraform, so all your config exists in your own repo all the time. It also helps day to day since it's not that hard to accidentally flip some switch in the dashboard.
But yeah, do research alternatives. CF has too much power already and will either ignore issues, destroy you, or pay lawyers to protect people trying to get you murdered, depending on their mood. There are better options.
$250/month sounds like nothing at all for a site with a claimed 4M MAU. The enterprise rate of $10k/month sounds a lot more reasonable. If everything presented here is accurate, I'm not understanding the sharp discrepancy in pricing tiers. If anything they should've already been paying more than $10k/month for massive traffic on the basic plan and then be able to save money by paying for massive scale when negotiating rates for the enterprise plan.
Also this sounds like an online gambling site of questionable legality, knowingly serving customers in jurisdictions where it's illegal, so I can't say I have too much sympathy, and I feel like Cloudflare effectively fired them as a customer when they realized what they were up to.
The amount shouldn't matter, it's the unprofessional response from CF that's of concern. Also, the author doesn't necessarily say that they would be unwilling to pay more or even negotiate a higher price. The author has made it explicit that CF were more or less unwilling for any practical conversation. That, to me, is the problem. A lack of professional courtesy, communication, and transparency. Obviously there may be details from this exchange which have been omitted but if I were in this position, I would be equally upset.
This doesn't pass the sniff test. From the actions Cloudflare took and their communication, it's very clear that there was something about the way their services were used that they were unhappy about. The post doesn't include what that problem was, but I have a very hard time believing that the author was not in the know and just got their account nuked without any further commentary, especially after being in a number of calls with real human beings from Cloudflare. Surely they'd have plenty of room to both ask and tell to figure out what the issue is. More than anything, this sounds like they knowingly did something shady and are now trying to shift the blame.
Indeed. Based on what has been shared by OP, they could have a case.
If OP’s business was in fact illegal, CF should have stated it. Now it seems CF is an evil sales driving monster. A monster that grew so big it thinks it can do whatever it likes.
The sad part is that, assuming OP is not leaving out critical parts, multiple people play parts of this evil machine. I’ve seen how sales people think. But this is next level toxic culture. The second customer threats of leaving for the competition, they freak out and pull a bigger lever to destroy them. And the fact that a company allows this to happen…
I would never do business with CF. Good thing i don’t right now. Cause i will definitely take it elsewhere.
After reading this article carefully I have a few thoughts. Firstly you were knowingly in violation of TOS after they pointed it out to you. Violating their TOS is a fast way to have your account suspended.
Secondly I'm a little confused why they would require you to pay a year upfront? I would like to hear from cloudflare as to why they required this? It's pretty fair for them to ask you to pay a year in advance because of the risk that you carry as a gambling company.
Cloudflare needed you to have to enterprise plan to remove liability from them. It's not even a big request, they have specific pricing plans for a reason.
There is a single mention of html in their terms, and it just restricts wrapping cloudflares software. Could you link me the section where they restrict anything to html-only?
> I'm a SysOps engineer at a fairly large online casino. We have around 4 million monthly active users. We had been happy Cloudflare customers since 2018 on the "Business" plan which has some neat features and costs $250/month for "unlimited" traffic.
Sorry to be “that guy,” but, you’re serving 4 million people at a casino and paying $250 a month for shared multi tenant infra, and you’re SURPRISED you have problems? Really?
To be honest, I’m glad these sorts of businesses get kicked off Cloudflare because it causes problems for others sharing the same IP space and infra. I’ll let someone else with experience discuss how many times a day the network would see a hacking or DDoS attempt against the online casino, which is by far the favorite target of hackers. But in general, I just don’t want any of my infra touching the same stuff as these guys.
Like another person here, I am assuming that Cloudflare ops told someone “tell these guys to get their own IPs and upgrade,” and then the message went to Cloudflare’s (utterly lousy!!!) sales people to try to fix before shutdown, and then it all turned into the mess we see here.
The true moral of the story, I think, is, if you’re running an online casino on a shoestring budget, expect bad things to happen to you. Of all kinds.
This reads more like a shakedown than anything. Even if the casino was being dodgy, CF went in asking for more money, not demanding that they stop doing whatever it is they were doing.
This is the nail on the coffin, but make no mistake, Cloudflare has been a liability. It's a massive Man In the Middle decrypting all traffic, including OkCupid and 4chan for example. Imagine all those 4channers learning they aren't actually anon.
This is my first post on Hacker News as I primarily just browse. This situation kept me intrigued, wanting to know how it would unfold.
The Google Cloud situation and all these little happenings, including the proliferation of Gen AI into everything, make me long for the days when companies had their mainframes onsite, in closets or separate rooms, away from CDNs and cloud networks. It seems like a better idea to use these cloud networks as a separate off-site backup rather than for primary use.
I’d love to learn more about what will happen next in this saga. I’ve seen a post where a Cloudflare exec has posted here on HN before. They probably won’t say anything for legal reasons, but what repercussions can Cloudflare expect for this? Will they be, or can they be, sued for this downtime and the related expenses?
Unfortunately it's difficult and expensive to scale those traditional solutions to the modern world of billions of internet users located all over the world. It's still quite slow to access a server on the other side of the globe. It's less noticeable for an american user accessing an american company's resources.
Does it matter for 90% of business?
I will say no, the vast majority of business is rather localized for very good reasons (as a starter, language/culture/legalese) so it is an unnecessary "feature" most of the time.
The reality is that manager drank the Kool-Aid and wants to pretend that they are Google/Facebook or the likes but almost no one else has this kind of scale.
Outside of tech behemoth nobody need that cloud bullshit.
The OP does not give their company nor domain name. I wonder if this is related to recent efforts by the Dutch to collaborate with Cloudflare to prevent online gambling companies operating in the Netherlands.
482 comments
[ 2.9 ms ] story [ 358 ms ] threadShocking how often "gatekeepers" fall to the temptation.
Holy shit.
In that light you can see why Sales would be sent as the messenger. But I agree they shouldn't have been involved. Sending a T&S representative would have been better. But then again it looks like from screenshot #2 that they kind of did that? They just didn't have a direct call with T&S.
>Note that 80TB is the number they tried to sell us, I don’t know if it is accurate since they removed all our access to historical analytics.
I mean you dont need accurate Data but surely most would know by heart their traffic in rough figures? Or am I the old dog where every new Web Dev are so used to Cloud and Serverless they have no idea what they are using?
As far as exact volume of QPS or TB/month or whatever, I really couldn't say.
I spend a lot of time wondering if the Emperor is wearing any clothes.
It would be completely different for a small project of course, but once you're counting in TBs... it's less important.
> if a country DNS-blocks our main domain, a secondary domain may still be available
Risk can be mitigated, especially if you take care to know what the risk is, but risk mitigation and the salaries of the risk mitigation teams are not free.
The answer of "no, we will not host you unless you pay us enough money to hire people to make sure we're not breaking laws by hosting you" makes plenty of sense, and an online casino that is likely dubiously legal in many countries is definitely a place where you might use that answer.
I'd also expect there are cases where Cloudflare enter into enterprise agreements with customers, get a good hard look at exactly what's happening, and then tear up the agreement and walk away.
To me it looks like https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr... is entirely the wrong email to send in the situation and if you are as old as I am and come from where I come from, you will have flashbacks to "reading between the lines" of the party daily in the 1980s. The real content is at the bottom:
> As we have a very short window to report back to Trust & Safety team, please let me know if you can make time tomorrow
Big red flashing lights: the right questions are 1) why is T&S involved at all 2) What are their concerns which forces such a hurried deadline? 3) What are the consequences of missing this deadline.
The right email would start with something like this:
> Providing services to your business constitutes serious legal risk to Cloudflare. We are happy to work with you in the future if you are buying an Enterprise plan. As we need to commit significant resources to accommodate you, we need an annual commitment. Otherwise, with much regret we need to terminate our services provided to you as it is our right per Terms on date/time. ("We may at our sole discretion terminate your user account or Suspend or terminate your use or access to the Service at any time, with or without notice for any reason or no reason at all.")
> This plan would also include these features:
Companies take risks if the reward is considered good enough. In this case, that reward is income from the customer (who can still be dropped if the hit pieces start getting published).
That being said, I doubt that's the core issue in this case.
It doesn't matter the reasoning - its the execution wherein lies the issue - this is an extortionary business practice plain and simple.
By the way, it appears gambling sites are fine on CF [1].
[1] https://community.cloudflare.com/t/using-the-services-for-on...
In support of your theory particular is I don't think enterprise sales "ragequits" a conversation when the customer is mid-evaluation based simply on the idea that they are considering multiple options.
Why would they walk away at this point, let alone ban the customer.
From the write-up I bet CloudFlare had it as a "60% to close" in their CRM at this moment. It doesn't make sense for them to drop the ban hammer in this moment.
PS: explanation or not, this is deeply shady behaviour from CloudFlare. Just perhaps a little less so.
> Why would they walk away at this point, let alone ban the customer.
It wasn't just that they were considering multiple options. Looking at the timeline, this was about a month after their initial soft gloves approach/enforcement action and they drug their feet the entire way through it.
Once CF got to the top of the leadership chain at their company and it was clear that all the relevant decision makers were involved in the conversation but were unwilling to pay, they just folded their cards, resumed the initial enforcement action, and moved on with their day.
If this was a small account they probably wouldn't have even blinked twice with just striking down the user for causing reputation harm and violating TOS but since they were a large account CF clearly went out of their way to meet with them multiple times and try to find a solution. But after a month of little to no progress while the account continues causing reputational harm and is unwilling to budge, they just called it quits and moved on.
If there is a TOS issue I’m not listening to a sales pitch on it. You better tell me what the issue is upfront in the first email instead of dicking around with the commission based workers. Like very low level stuff here imo
I don't see an unwillingness to fix TOS issues anywhere. Just an unwillingness to buy the enterprise tier. Those should not be treated the same way!
I'm not saying cloudflare can't do it, I'm just saying it's wrong.
If you think something your client wants could explode into a liability, you can turn them away or you can just make sure their bill covers your exposure.
If it's a legally questionable service, there's likely to be plenty of abuse contact, or they're going to be a big target of crime, they're going to end up paying more. This is the same reason why some industries (eg porn sites) have always paid more for card processing.
Their business was causing IP reputation damage and all plans but the enterprise BYOIP plans share the same IP pool.
Essentially it was "use your own IP pool and pay us for the cost of maintaining that pool for you or GTFO".
This wasn't just a normal sales rep hitting them up. This was trust and safety (i.e. the moderation team) coming to them with a compromise that would allow them to stay on the platform. They chose against that and were dragging their feet.
The timeline of the article also really makes this clear. This wasn't over the course of 24 hours. This started a full 4 weeks prior with sustained back and forth. They only included a few images of emails from the discussions but the article makes clear that there was more discussion happening.
And to quote the article. After receiving the ultimatum, they got an entire extra week to deliberate.
> We managed to buy a week of time by letting it escalate to our CEO and CTO and having them talk directly with Cloudflare.
Then finally when they told CF that they were just buying time while looking to move elsewhere, CF dropped their act of goodwill and the moderation team resumed the moderation action they would have taken in the first place had this been a smaller account.
----
So yeah it sounds bad from the snippets but this was basically "hey you are a big customer and you are breaking rules we would normally ban anyone else for but if you can compensate us we'll spend the labor hours and infra to let you keep operating in your own little quarantine box.". So this really should be seen as an act of goodwill rather than malice.
It's called "extortion"
You put yourself in a bad spot. We can either kick you out or work (for a price) to help you.
Extortion ? Hardly. Nobody work for free, you know.
They also were clearly lying in those email messages: The second email says that domain rotation is strictly forbidden, but a few days later in the third email they're explicitly selling features for rotating domains more effectively.
And sorry, but a company selling "we'll override the Trust and Safety team if you pay us $$$" is absolutely unacceptable. There are only two options, both bad. Either they're not running a real TnS operation, but just pretend-staff one in order to run these kinds of shakedown operations. Or they're running a real TnS team that found a real problem but are letting sales people override the TnS team's honest judgement.
Edit: "How do you know?" -- I don't know it's actually what happened, but when switching to enterprise, you don't go from 10% margin to 98% margin. The added costs actually represent added budget for the provider to deal with your "special case". ALL enterprise pricing tiers are disguised consulting contracts.
The only questions that come to mind: how do you know? If that was the case why didn't they tell the customer?
Except the "place" isn't Mom and Pop's bodega, it's a casino dodging countries blocking its main domain.
i.e. it's a $10k fee for maintaining the infrastructure for a quarantine around their services
They did. Repeatedly. You see it mentioned in the few emails the OP chose to share. But they also didn't share the other communication they had over the month long discussion they had with cloudflare.
> BYOIP also costs nothing to produce.
That's not really accurate. Cloudflare is entirely built around one big unified anycast network. If you want to provision an entirely separate network that maintains all the same features they are using from the main cloudflare network, that's going to require provisioning a lot of cpu time and routing table slots at a lot of different sites plus whatever admin and engineer overhead comes from maintaining this quarantined service.
Anycast/unicast changes nothing here. Anycast simply means that the same prefix is announced in multiple different PoPs.
>Here's the features you get with enterprise
And
>If you do not use byoip with your service you cannot be a customer, and also pay us $120k upfront while we do not tell you why, it's only a 40x price increase. You have 24h.
As a network engineer I'm well aware of what it costs to add prefixes to what you announce over bgp, some miniscule additional CPU overhead, likely unmeasurable. Even slow mips processors could process full table bgp.
> We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries.
Evasion:
> Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available.
This is more like one gang hitting up another for "protection" payments. I had to laugh when they called it "Trust & Safety".
Them very aggressively highlighting the BYO IP feature and then even suggesting third parties to rent IPs from strikes me as a significant detour from their normal “script” (having dealt with their AU sales team before).
this customer is damaging Cloudflare IP reputation which hurts other customers. Cloudflare can either fire the customer to protect other customers using Cloudflare IPs, or force this customer to use their own IPs and damage/manage their own IP reputation.
unfortunately this is expensive and OP is mad they can't do their legally fraught gambling operation on Cloudflare's addresses for free
If a custom IP is going to be mandatory, they need to say that and give a deadline, at the very least.
As a CF customer, I am happy that Cf is preventing another business from damaging mine.
They weren’t protecting you or any of their customers. This is a mafia style shakedown
This is directly contradicted by the contents of the article, perhaps you should re-read it.
>$120k up front for one year of Enterprise
Doesn't sound like a reputation problem.
But only after getting ~US$100k up front first, just because you can.
They could've explained the problem ("your gambling business is a problem for our IP reputation") and offered a solution ("we can switch you over to BYOIP so this won't be a problem"), but instead they sent in an army of sales reps that demanded an upfront payment for a product tier that they only needed one small part of, to the point of sales people pretending to be part of other teams.
It makes business sense to kick out casinos, but OP got fucked over by Cloudflare's shitty practices.
CF calls and says there is a problem with domains. They want to push an enterprise plan. Customer wants to solve problem, dropping domains, making changes. CF says, only enterprise plan will remedy the situation.
There is obviously a sales script involved.
“get back to Trust & Safety"
Heard that story several times, it's always another team, e.g. "Licensing" that need to be satisfied, or that if you don't pay up, that team will be off the leash. Also heard the pay-for-a-year-upfront for several large vendors who pull this. The reason is, some sales reps need to make numbers, so they shake the tree and see who falls down:
"Cloudflare has absolutely no information on when they will force you into custom billing, but when they start "urgently" needing to talk to you you're probably not going to get out until you have a juicy custom contract with them."
It's impossible to say what's going on since it's an anonymous post with no details.
Maybe it's all 100% true.
Maybe there are some key details being left out. Wouldn't be the first time I've seen one of those outrage posts that seriously misrepresented things.
Whatever the case, obviously the author is not an unbiased party. These posts do well because "zomg Cloudflare bad!", and maybe they are, but I sure as fuck don't trust some casino guy either.
My guess: Their account fell out of the non-enterprise TOS for some reason which is being obscured in the post (probably domain rotation related). Their T&S team proposed moving to enterprise for a custom resolution. OP's company refused, their account was purged because they had gotten several warnings about it.
I'm sure this sounds frustrating to the average HN dev who runs a legitimate startup with cloudflare on top and is now biting their nails worried to death about what will happen to them. But "online casino" immediately raised a million alarm bells in the post.
Since we already left Cloudflare the only reason I finished writing this article is to warn others. I think it's still relevant to many companies regardless of what you think of casinos, since very unprofessional sales tactics (unprofessional as in business threatening) seem common place with them. Do look at the linked other posts and comments here from other people affected that don't have anything to do with casinos.
I'm happy to answer questions as well.
If you had legal presence in EU then new Digital Services Act[1] might a help for you. I am not sure if you could sue them based on that law, but you could maybe lodge a complaint.
Looks like they COMPLY with regulatory interest, to me.
"In order to comply with tax regulations and donor laws, we had to structure our activities in order to make it possible for political donations to be classified as regular consulting income".
For me, my corporate tax professional errs on the side of well tested strategies or those that the IRS has ruled on administratively, and that is perfect for my risk tolerance.
Well HN is the unofficially official Cloudflare Support forum. I think we will hear from them soon. From past experience normally their response time for anything Cloudflare on HN is within 2-3 hours.
Let's not try to find reasons to harm the messenger and stick to the facts -- a paying customer was suddenly extorted for hundreds of thousand of dollars out of nowhere.
If they can indeed stop providing services to a casino, why cannot they shutdown a website spreading pro-war propaganda, or a website selling illegal services ?
It means they are making editorial choices, instead of just being the technological provider and being a neutral "internet pipe".
Not sure it's really in their best interest to self-police in the end, as they could lose their DMCA safe harbor provision ?
This.
That said, we're seeing this across so many platforms, from datacenters to social network sites.
They blew their safe harbor provisions years ago and yet remain untouched despite this.
The same thing is true for IP reputation, just without an external official complaining. If other CF customers are negatively impacted by one customer’s action, CF isn’t violating safe harbor by booting that customer or passing on the costs of mitigating that impact. That’s just running a business, not exercising editorial review of hosted content in violation of safe harbor provisions.
Because their main network all uses one big IP address pool and the blocks by various regions/countries against their site were probably not just DNS blocks but also IP address blocks.
So they now have an account whose activity is getting their IPs banned in countries where they operate.
So they told the account owners they needed to pay for an enterprise account and a dedicated IP address pool maintained by cloudflare. That's why CF kept talking about BYOIP in the emails.
i.e. "Pay for us to build you a quarantine with your own IP pool or leave ASAP"
It smells like the "problem" was detected by automation, but instead of being able to reach anyone technical to work through it, you can only call sales teams.
In my opinion it's one racket vs another.
My 2c: It’s scummy that CF did this. It looks like they were disingenuous about the severity of the violations and used it as an excuse to get more $$$ from an already paying customer to make the manufactured problem go away.
It doesn't seem so, so there is at least a valid reason for Cloudflare to keep them as a customer as they're not violating the laws where they have their business in.
Not in response to the way Cloudflare came at them, anyway.
https://m.youtube.com/watch?v=7LuwPdp-_4c
How is that something worth going viral over? Salespeople get fired all the time for not meeting their sales goals. Engineers similarly get fired all the time for not meeting their productivity goals. If you don't do your job well, don't expect to keep it.
And if I recall correctly, in this particular case, she was a green employee who hadn't even made a single sale yet! What more obvious of a layoff target is there than that? Would you keep a green unproven salesperson over a proven veteran salesperson who's landed 9 figures in sales?
Especially in a world where people pick up their whole lives and relocate for jobs. Recent joiners aren't getting any sustainable kind of severance either. The idea is if you're hiring them you have a minimum commitment to support their success.
Yes she was an obvious fire, but it's also the organization's fault. Enterprise deals also take way longer to close than that...
All that said, salespeople can and do move jobs a lot. I'm sure she'll be fine.
Was there any indication that the "mass layoffs" were planned 90 days in advance?
If you're a public company and you go from healthy to mass layoffs within a single quarter then your investors (and employees) should be a lot more concerned.
1. According to the CEO the layoffs in question were for "~40 sales people out of over 1,500 in our go to market org"[1]. Is that really a "mass layoff"?
2. Did you not remember that the layoffs coincided with reversal in macroeconomic conditions? Specifically, the reversal from "inflation is transitory" to "inflation is persistent and the central bank will hike interest interest rates".
[1] https://x.com/eastdakota/status/1745697840180191501
That's why the org took heat online for it.
I'm not sure why you think she was fired/assessed on the basis that she wasn't able to close enterprise deals within 90 days. The same tweet seems to refute this by saying "we can often tell within 3 months or less of a sales hire [...] whether they’re going to be successful or not". Presumably they're looking at various indicators (eg. size/composition of the sales pipeline, reviewing her sales calls and/or emails) and using that to predict performance.
That timing shows that it's not just implementing headcount and budget reductions, which are somewhat defensible if still tragic. It was instead a forced turnover, which in some cases can be a route to wage suppression.
Apparently the person in question was fired within 3 months of being hired[1]. If this is true it seems like a stretch to characterize it as "forced turnover" or "wage suppression".
[1] https://x.com/eastdakota/status/1745697840180191501
If you can‘t land a single customer in 6 months under these circumstances, you‘re likely just not a good fit.
You can even google my name to confirm this info. This is pretty much exactly like it went when I started.
Not really nice to leave out halve of the info :)
"Small SaaS banned by Cloudflare after 4 years of being paying customer"
https://news.ycombinator.com/item?id=34639212
Also:
https://news.ycombinator.com/item?id=31336515
[1] Cloudflare decrypts your traffic, reads it, and then forwards it. They see all encrypted data going to and from your website, in plaintext.
https://www.projecthoneypot.org/cloudflare_beta.html
In fact, the company vision and values (as the team grew) may have changed over the time, but originally it seems it was somewhat of a different spirit (and closer to a data collection network).
* Cloudflare operates at a scale where its caching saves a lot of bandwidth, which saves ISPs money, which makes Cloudflare an attractive partner for peering and co-location.
* CDN is a platform on top of which Cloudflare can offer a lot of additional services that used to be expensive dedicated middleboxes.
[0] https://x.com/eastdakota/
[1] https://news.ycombinator.com/user?id=eastdakota
As far as I can see, the author was careful to redact their domain from all screenshots.
Also; if not registering domains on CF does anyone else do at-cost or otherwise super cheap pricing?
Actually had a sales call with Cloudflare in the last month and I got some bad vibes from the whole experience. We did not end up going with them.
10TB/80TB at 120k/yr, either way, Cloudflare is taking you for a ride.
If you aren't self hosting, you're really doing it wrong.
Yet everyone in "ai ai ai ai" is buying up Nvidia/CUDA like there is no tomorrow and then pooping on AMD for even trying to do anything.
History loves to repeat itself.
It's not like you can have your domain/DNS somewhere else and point to Cloudflare IPs (to not put DNS and CDN in same basket). Cloudflare does not allow that setup.
You can't protect your website from your DNS provider or hosting provider suddenly kicking you off. You are going to be offline for a couple of days.
Sure you can. Colocate in two or three places. You're your own DNS provider and your own hosting provider. If one of your colocation companies doesn't like what you're doing for whatever reason, you use the other two until you replace that provider.
And you are still unprotected from your DNS registrar kicking you out, directing your domain to some "customer terminated" page until you can find another registrar, and have new NS records propagate (days).
Let me summarize: businesses do silly and sometimes stupid things for irrational reasons, or for reasons they never care to divulge. To avoid what happened that led to this discussion, the most suitable solution is to not be at the whims of companies that don't communicate well.
The legal requirements of domain registrars are clearly spelled out, unlike the TOS from, say, Cloudflare which leaves tremendous amounts to the imagination. These are not the same at all.
1. Its quite possible thar CF having this site on some multi-tenant infrastructure could be threatening. Not unreasonable at least to ask them to have their own IP block.
2. If thats the issue then a clear explanation should have been provided. Routing to sales is inexcusable. Someone isnt being transparent.
3. If it’s a pure cost / revenue issue then say that, set a deadline and negotiate. This is bad karma and even though CF is clearly the market leader, what they do isnt rocket surgery. Not worth it.
Given that the article is an online casino that seems to be using potentially ToS violating domain rotation, and that they pay so little per month for apparently millions of users, I for one will not form an opinion on CF based on this article before CF has a chance to defend itself.
- a companies ops team identifies a tenant that is too heavy/burdensome for multi-tenant infra and is causing issues. These issues can cost a serious amount of money if you factor in dev/ops times to resolve, other customers impacted, etc. Certainly more than what a hypothetical single multi-tenant customer could be paying
- they escalated internally and need the tenant moved to enterprise asap to resolve
- the only reason the tenant was on multi was because sales sold them the wrong thing, so now it's on sales to explain how to fix this
- improper handling internally results in this landing only on sales, with no backup, and with their task being to get them to take enterprise
- when the customer refuses enterprise they go "we've tried nothing and we're all out of ideas"
Again, this is totally speculation and I'd hope CF has more mature practices than this but this is a scenario I've seen before in much smaller orgs.
> This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above.
And the very next paragraph begins with:
> In any case, we receive >95% of our traffic through the main domain that’s been unchanged since our founding, and were happy to resolve this issue in whatever way...
And then they complain about paying up?
The only issue I see here is around the aggressiveness on the CF side. But, I was not in those meetings and the way above reads tells me that I might have been slightly mad so perhaps the CF was just taking it out on them?
Anyway. I don't think this is a CF foul.
Attorneys love it when people put everything in writing like this.
If a country A decides to block twitter.com but forgets to ban x.com which remains available ... is Twitter engaging in illegality / violation of CDN terms of service?
Right, I'm not arguing that they're guilty, just that the legal system doesn't operate off pure black and white rules like "if you have two seemingly unrelated domains then you're trying to evade bans".
>and they give a traffic number that proves they're not doing "domain rotation".
Are you talking about the part where they said "95% of our traffic through the main domain"? Without additional context behind that number it's a stretch to claim that "proves" they're not trying to evade bans. For instance if their country is banned in country A, but country A is a small country that only makes up 3% of their overall traffic, they can confidently claim "95% of our traffic through the main domain" while still theoretically using alternate domains for ban evasion purposes.
Better yet, configure CloudFlare through terraform, so all your config exists in your own repo all the time. It also helps day to day since it's not that hard to accidentally flip some switch in the dashboard.
But yeah, do research alternatives. CF has too much power already and will either ignore issues, destroy you, or pay lawyers to protect people trying to get you murdered, depending on their mood. There are better options.
Also this sounds like an online gambling site of questionable legality, knowingly serving customers in jurisdictions where it's illegal, so I can't say I have too much sympathy, and I feel like Cloudflare effectively fired them as a customer when they realized what they were up to.
CF was more than happy to help, but .25k wasn’t a number that solved all the problems.
If OP’s business was in fact illegal, CF should have stated it. Now it seems CF is an evil sales driving monster. A monster that grew so big it thinks it can do whatever it likes.
The sad part is that, assuming OP is not leaving out critical parts, multiple people play parts of this evil machine. I’ve seen how sales people think. But this is next level toxic culture. The second customer threats of leaving for the competition, they freak out and pull a bigger lever to destroy them. And the fact that a company allows this to happen…
I would never do business with CF. Good thing i don’t right now. Cause i will definitely take it elsewhere.
Secondly I'm a little confused why they would require you to pay a year upfront? I would like to hear from cloudflare as to why they required this? It's pretty fair for them to ask you to pay a year in advance because of the risk that you carry as a gambling company.
Cloudflare needed you to have to enterprise plan to remove liability from them. It's not even a big request, they have specific pricing plans for a reason.
They know this
Sorry to be “that guy,” but, you’re serving 4 million people at a casino and paying $250 a month for shared multi tenant infra, and you’re SURPRISED you have problems? Really?
To be honest, I’m glad these sorts of businesses get kicked off Cloudflare because it causes problems for others sharing the same IP space and infra. I’ll let someone else with experience discuss how many times a day the network would see a hacking or DDoS attempt against the online casino, which is by far the favorite target of hackers. But in general, I just don’t want any of my infra touching the same stuff as these guys.
Like another person here, I am assuming that Cloudflare ops told someone “tell these guys to get their own IPs and upgrade,” and then the message went to Cloudflare’s (utterly lousy!!!) sales people to try to fix before shutdown, and then it all turned into the mess we see here.
The true moral of the story, I think, is, if you’re running an online casino on a shoestring budget, expect bad things to happen to you. Of all kinds.
> I'm a SysOps engineer at a fairly large online casino
Oh no, a casino losing the trust of its customers? Those places are normally so scrupulous!
Caching, detecting+modifying headers, routing traffic, ...
The Google Cloud situation and all these little happenings, including the proliferation of Gen AI into everything, make me long for the days when companies had their mainframes onsite, in closets or separate rooms, away from CDNs and cloud networks. It seems like a better idea to use these cloud networks as a separate off-site backup rather than for primary use.
I’d love to learn more about what will happen next in this saga. I’ve seen a post where a Cloudflare exec has posted here on HN before. They probably won’t say anything for legal reasons, but what repercussions can Cloudflare expect for this? Will they be, or can they be, sued for this downtime and the related expenses?
When big cloud goes down, you get a few days of credit. That's it.
What's the Google Cloud situation?
[1] https://news.ycombinator.com/item?id=40313171
[2] https://news.ycombinator.com/item?id=40304666
[3] https://news.ycombinator.com/item?id=40466810
https://igamingbusiness.com/legal-compliance/legal/cloudflar...