Twilio blocked our account and with it hundreds of our customers [resolved]
I'm the founder of a tech startup in the field of telephony and we use Twilio for parts of our service.
A little over 3 hours ago our entire account was suspended without reason. We received an email stating that fact without further explanation.
Since then, we are trying to get in contact with them through every channel we could find: email, Twitter, LinkedIn, and I even tried emailing Jeff Lawson, their CEO. They don't have a support number we can call.
We are not a small customer and we have been with them for more than 6 years. It's deeply frustrating that despite that fact, we don't seem to matter to them at all.
If you work at Twilio or if you can help us get in touch with them, I'd be indebted to you forever!
164 comments
[ 3.0 ms ] story [ 262 ms ] threadI take it you've tried (844) 814-4627 ?
Thank you!
Thank you! We are still here with the team trying to find someone who can help us.
The FCC has defined these SLAs and if you ever have a losing provider that is close to breaking the SLA on a port request, search for the providers local number portability escalation list and call each contact until resolved.
Voxbeam and Anveo Direct are solid choices for international DIDs.
Bandwidth.com has a poor reputation FYI, they pissed off a few Public Utility Commissions and were blacklisted from getting new phone numbers in some states.
Unfortunately (with the exception of SIP dialling, which is extremely niche) there is no DNS for telephony. If your provider cuts you off, you can’t transparently move away, and number porting is slow.
I'd assume the same courtesy would be extended to any customer, provided their ToS breaches aren't significantly worse (read: illegal) and it's not a billing-related suspension.
Unpopular opinion.
You see, I dont want assumption. I want guarantees. I also dont want a platform owner to remove you without some days if not weeks notice. I mean your landlord and court give you time for eviction. At least I hope that is true, I am not sure if time to vacate is universal across the world. They also give little to no time to solve any dispute.
I also dont like it when media spin that AWS is at least helping them move their Data off the platform as if it is something good. I dont know why it is perceived that way in the US, but that was absolutely the least they need to do and expected in EU or UK.
It is increasingly a worry sign. I have been yearning for an iOS Time Capsule for more than a decade. And the recent event makes it even more important. I want to own my things.
Spin up your own servers, etc. I don't really get why so many people here think AWS is the only way to run a website nowadays; you can put metal in a closet anywhere that has a good enough internet connection. If you're planning on doing shit that'll get your cloud accounts suspended, why on earth are you using a cloud provider?
The only way to _guarantee_ that you'll have access to your data no matter what is to own it, so own it!
Back to your eviction policy- it's common in my area for leases to have a clause stating that landlords may recoup delinquent rent by entering your apartment and taking your property. Legal? Don't know, but my lease as well as the leases of most of my friends have this clause in them. I also know that university housing will kick you out with < 1 week notice at my uni if you're in violation of your housing contract.
Maybe we just have very different points of view/experiences, but in my eyes you can't "own" anything in a cloud provider. This is fine for most circumstances (i.e. my personal site is on Netlify but if I get kicked off I still have all the source), but if you need to own something just put it on your own server.
That isn’t the case for PSTN. It’s outright impossible.
In fact there is: https://www.networkworld.com/article/2332977/lan-wan-what-is...
Only most telephony providers sabotage it.
But let's say there is a contract or TOS that says they won't, and they do - what are you going to do about it? Go to court for 2 years?
I use both Twilio and Plivo, and they're both fine for sms.
(Twilio is running amok because there was a recent court case that providers could be held liable for actions of their clients, so last month you would have received update ToS from several cloud providers in your email.)
If so, does using Twilio's subaccounts feature do anything to prevent your whole account being suspended for the actions of one of your customers?
Sure there were "fines" but as usual that was pocket change them..
Clothes manufacturers have been notorious for mistreating their employees, though; e.g. Triangle Shirtwaist.
Ha, check out “unsafe at any speed” the book that launched Ralph Nader and the government’s role in consumer safety.
Or “The Jungle”, the book that launched food safety regulations (and some worker safety) a century ago.
Certain politicians and certain companies decry “regulation” generically but there are good reasons behind almost all of it.
Any small company ignoring privacy laws would get into trouble. If you're as big as Twitter or Facebook you can just ignore laws and get away with it..
Report them to your local ICO.
I understand what you're saying but, speaking specifically to this case (which is a little different than YouTube censorship), I think this highlights the dangers of outsourcing your critical infrastructure to third-party SAAS companies without a mitigation plan.
As companies, we have to get into the habit of putting together mitigations plans for times when the outsourced SAAS disruption has significant business impact. Here, Twilio blocked the account of this vendor, but Twilio could have gone bankrupt, or they themselves could have had a significant disruption and the end-result would have been the same. Was there a plan put in place for that? If Twilio isn't responsive with support, are any of their competitors better? Maybe it was a better idea to go with a smaller or more expensive competitor but one who gives you a dedicated account manager you can call anytime if something goes wrong. Twilio fucked up here, but ultimately the responsibility for business continuity rests with OP. They can't offload that responsibility on Twilio because if Twilio fucks up, Twilio doesn't suffer the consequences.
It's incredibly hard to have multiple vendors if you're using specific or advanced functionality, and if you're hosting a failover yourself then you might as well just use that instead.
I'm not suggesting any specific solution and I didn't even suggest they run their own infrastructure. There are good reasons to outsource these kinds of operations. But what happened was the company got the rug pulled underneath their feet and realized they didn't know how to contact their critical supplier. That's not good and that's their failure in this mess.
I don't understand the resistance to identifying and mitigating business risks. Honestly.
What happened in this situation is that Twilio for one reason or another decided to block access to their service, ostensibly halting their business. OK. That's not good. You know what made it worse? The CEO realizing that they don't have any line of communication to Twilio and that Twilio had shit tech support and therefore he had to frantically trawl Twitter and HN and Reddit and emailing Jeff Lawson (Twilio) CEO. Are you telling me they couldn't prevent this? You sure there isn't an Account Manager assigned to their account? Twilio holds conferences, there are plenty of opportunities to forge some relationship with someone at Twilio so you could at least backchannel issues like this (in fact, some Twilio dev here at HN noticed this and escalated it). By the way, Twilio is not the only telephony provider. If Twilio has shit support, OP can find another critical supplier, maybe a smaller or a more expensive vendor that is more responsive.
How about this: I looked through OPs comment history and he mentioned that their support numbers is provided by their service (i.e. they dogfood their product). Is that not a risk? If they are down, or Twilio is down, their customers can't reach a human either.
Very few of us control our circumstances, but we can certainly control our response to them. Outsourcing isn't the problem here. It has benefits and detriments. Lack of planning, and foresight is the issue here. This could have been a 10 minute outage, instead of a full day outage.
"In countries with strong rule of law:
"1. Property rights over land, equipment, and personal items are clear and protected by law.
"2. Contracts between people, businesses, and the government are effectively enforced by the legal system.
"3. Political accountability is high and corruption is low.
"4. Business regulations are clear and enforced in a transparent manner.
"In such environments people make long-term investments and build large organizations. In contrast, if the property rights and contracts are not enforced and the business regulations are not clear, most of the economy consists of small family owned firms with little modern equipment. A high-tech, prosperous economy would not develop.
"Effectively, there are no contracts anymore in the digital economy. There is no predictability anymore. There is no accountability. There is no responsibility. There are no requirements for performance anymore. In sum, the US digital economy is rapidly becoming the equivalent of a third-world economy, complete with crony capitalism and digital robber barons."
Phone number on the page and an email address that may work for you. Let me know if that helps.
UPDATE: Your account should be active now.
I plan to update everyone on what happened as soon as I get some sleep first. I've been up since 7 am and it's 2 am now. The incident started at 7 pm.
I'm kinda sad that the thread is about Twilio and not Google - my GMail account is blocked and I was hoping I could jump on the bandwagon.
Someone here should make a startup out of it.
Exactly. I'm locked out of a google account I have which links to a yt channel and a twitter account. The only truly viable option for me to get these back is work my way up the social media infulencing ladder and then complain how I lost access to my accounts. This is insane.
Got in contact with a real human (i think..) in chat in under a minute, which helped me get it resolved via filling out like 4 different forms.
No idea if this would work for you though, but worth a try?
That never, ever happened and I had to resort to a nicely worded note to a corporate VP on LinkedIn to resolve it. Even then, I'm pretty sure that "unpermitted gas line work" was the real driver in getting it resolved, since that's a big deal.
If the problem really gets kicked up in the media and left unaddressed, then eventually politicians will be forced to add some real regulation to force companies to do right by their customers.
That'd be expensive. So instead, ignore customers who can't kick up social attention and quickly resolve those that can before it gets legs. No one cares about the unknown, disconnected hordes of betrayed customers who lost their access for unknown reasons.
And I'm sure no company is covertly doing this to their tiny, would be competitors, killing the baby in the crib so to speak.
In fact, if you're an employee that actually cares about your customers, you'd do well to not internally raise these issues. Let the PR hit the executives harder and make them squirm.
After I shamed them in twitter/github the issue got solved in 1 day.
The world is has become too wild tbh.
Support cannot be an afterthought. If a long time customer has a critical incident like this, it needs to be caught and forwarded to the relevant people within a few hours at most. This type of PR fallout is unacceptable and completely avoidable. I, for one, will avoid Twilio just based on this story alone.
Can you imagine running a business and having a critical aspect of it just entirely stopped without warning or recourse?
People could lose their jobs during a pandemic due to ineptitude and systematic arrogance?
If you factor these kinds of risks into business equations, it does not look good.
Depending on Tech is starting to feel like US Health Insurance: you're 'covered' until you're 'not' in which case you're going bankrupt.
I understand what you're saying, but OP cannot alleviate their responsibility in this. You can't outsource your critical infrastructure to a third-party SAAS vendor and not have a plan for when things go tits up. OP didn't even know who to call. That's on them because they should have identified this as a risk long time ago.
But it doesn't so much seem like this is the case, having tried to reach out through several channels.
There are two issues that stand out:
1) The power asymmetry - some large SaaS vendors provide critical infrastructure to customers - like financial service providers to small business. There are tons of regulations in that industry because that.
2) The lack of recourse as standard practice in new high tech. If your bank magically stops you from collecting VISA at your corner store - there's generally a human you can speak to, pretty much right away. You may be put on hold. I get relatively immediate customer service from my bank as a tiny retail customer. The lack of provisions here by the industry is a systematic problem.
It was too late now. When things are on fire, you're in panic-mode now. The reality is that if Twilio had shitty support today, they had shitty support last week and 6 months ago (etc.). That should have been identified by OP as a risk to their business and rectified (either by forging a relationship with someone at Twilio so you can backchannel support request, or identifying your account manager and making sure they are responsive, or moving to another vendor with better support). Twilio is a critical supplier for OP's company. You can't just assume they care about your business as much as you care about your business.
>he power asymmetry - some large SaaS vendors provide critical infrastructure to customers
OK. So? There are lots of things you can plan for and not control. None of us are fully in control of our circumstances. We can't control how trillion-dollar companies behave. We can't control the weather. What you can control is your actions, your planning and ultimately your response to things you can't control.
For example, I peeked at OPs comment history and he mentioned that their support line was provided by them (they provide telephony services and therefore they dog-food their products) ... was that the right move though? Because if they are down (or Twilio is down), their customers will need to talk to someone and won't be able to reach them. Maybe it makes sense to have another provider handle their support line (or at least have a backup).
The point is that, sure, Twilio screwed up - fully agree with that. Maybe OP can recover some damages, and maybe not. Maybe it isn't worth chasing Twilio through courts for years and spending thousands of dollars. Regardless, ultimately it is OP that suffered the consequences of Twilio's screw up, so OP should prepare themselves for this in the future. And don't tell me that there's nothing OP could have done. That's bullshit.
For the love of christ will you please give us an 'email' verb in twiml ?
Or do you need sendgrid signups for some business metric and, therefore, this dead simple integration has to be avoided ?
Twilio is also the only Pay as you Go SaaS vendor that doesn’t have spend limits on accounts and advises customers to implement their own any fraud measures themselves.
While this was going on, we got an email from their sales rep, congratulating us for increasing our spend by more than 8000% and wanted to have a chat. So I guess they do have the systems to monitor unusual patterns, they just turn a blind eye.
You’d never know, someone in Twilio might even be selling customer lists on the black market to the fraudsters so they know who to target.
For example, all UK mobile phone numbers begin 07 (i.e. +44 7), fixed/special rate 08, and premium rate 09.
But SMS also has "short codes", which is probably what's being abused here. 5 or 6 digit numbers, usually used for things like "Text 12345 to donate €2 to this charity" or "Send 'CXM' to 12345 and receive a message telling you when the next bus comes to this stop, costs 10¢".
https://en.wikipedia.org/wiki/Telephone_numbers_in_the_Unite...
https://en.wikipedia.org/wiki/Short_code
AFAIK there's no easy way to always, securely detect premium numbers. The fraudster can setup a forward from a normal number to a premium one anyway. So checking number prefix is useless. One could listen for the "After the beep, this call will be billed at X c/min" recording that premium numbers in honest countries have. The fraudsters manage to find less honest premium number providers that skip these.
This is a well enough known issue that Twilio has a page about it: https://www.twilio.com/learn/voice-and-video/toll-fraud
If Twilio wanted to stand up for their customers they would refuse to pay for any premium number that's not in an ITU registered premium range. It will cause a few lawsuits with telecom carriers, but then probably ends this problem once and for all.
Twilio claims this isn't possible, but most mobile operators offer an option to disable calling premium numbers. So either the providers are very customer friendly and eat the cost (unlikely, it would open them up for huge amounts of fraud) or the mobile operators figured out how to block it or push the cost back to the upstream provider.
Would the forwarding party not have to pay the premium rate part in this case?
Doesn't this mean that your call from Twillo to the normal number would be charged at your usual rate, and the fraudster would pick up the cost for the forwarding?
At this point we're entering criminal territory. Enriching yourself when you should reasonably know something criminal is occurring (like someone being defrauded), and helping the perpetrator by providing your service anyways, can make you a criminal yourself very fast.
As far as judges will be concerned, you're now a perp yourself.
I have a very hard time seeing the criminality of this. Can any lawyers provide more insight?
That doesn't sound automated.
Isn’t this a major selling feature of pretty much every CRM?
One young prosecutor seeking to make a splash in a jurisdiction with a twilio customer.
Twilio might win in court, but they’d lose in the big picture.
EDIT: With AWS SNS SMS you can set a spending limit and your bill will never go over that.
Source: use it and twilio.
https://docs.aws.amazon.com/sns/latest/dg/sms_publish-to-pho...
Looks like we'd need https://docs.aws.amazon.com/pinpoint/latest/userguide/channe...
Which isn't bad, but is much more limited than what Twilio provides.
Lesson learnt, always keep a backup for crucial services and move away from Mailgun
I’ve always been a big advocate for using 3rd party SaaS, but with the level of customer service I’m seeing lately, I’m starting to recommend avoiding it where possible.
I'd rather spend 100 hours upfront to have something that I know and can maintain than using a service, running into trouble and spend 50 hours chasing the support around to see whether they'd fix it, or worse, rely on their system, have it fail or be banned, and then having to spend 100 hours building a replacement, often while things are exploding left and right.
Relying on SaaS feels incredibly fragile to me.
Tell FAANG: You blocked me, again
FAANG Support: Important please call back ASAP
I'll call it Support As A Service.
1. If someone works at a company, they can sign up on our website as a support agent.
2. We will only list companies with which we have an authenticated contact, maybe require people to verify they got our authentication token on their employee email?
3. When someone contacts us about an issue with a company we support, we verify the request is genuine. Perhaps charge a small fee to block spam. (This is the problematic part. How do we know a problem is genuine without asking for potentially confidential information?)
4. Add all vetted requests to a private channel that all the people who volunteered on step one. (Can we pay our volunteer support agents? How? Does paying make things worse? Maybe gamify this step and send our volunteers gifts?)
5. Monitor performance of support agents to avoid giving false hope to our customers.
IMHO, most emphatically *yes* - this is a completely unserved/unserviced area of the B2B and B2C markets.
I noted in the linked thread that such a service would likely start by leveraging individual contacts, and building a reputation for unwaveringly high signal.
It definitely does seem to me that the vacuum between corporate and customers represents a giant sleeper industry that fundamentally won't shift very quickly.
Which means you're unlikely to reach the usual core-problem-is-solved entrenchment/stagnation point that tends to happen when you ramp up on a complex-but-solve-once problem, then find yourself stuck with a bunch of intricate sausage machine bits that aren't doing anything...
Having been a customer since our launch [1], and very early after theirs, it has been really sad and frustrating to see that friendly distrupter energy zap out of the service.
I get why businesses go through that changes like this during growth, but it feels like all the things to let slide, your persona is not one of them!
[1] https://mailosaur.com (somewhat related to the thread we let you test SMS messages)
Two weeks ago, all of the analog lines stopped working, which are local numbers, but our main 800 number forwards to those numbers.
Calling the local provider was futile. The provider’s office was also experiencing the issue and could not receive calls.
Although we do have the 800 number on business cards, we quickly updated our website and Intercom docs to use a recently provisioned 800 number from one of the VOIP providers.
It took 2 days for the other lines to begin working again. This is the first POTS outage we have experienced in several years.
I intend to avoid Twilio in a wide range, don't need that kind of a "business partner" in my life.
- long term we really should move to an industry wide due process system with legislation for compensation and other parameters. once there is some more structure there can also be insurances or similar ways to prevent something like this to ruin a company.
- there maybe is a market for more intermediaries that bundle twilio and messagebird or similar providers and have an official strong communication channel to the providers to resolve issues and provide failover
Tons of platforms seem to operate like this, and this kinda thing happens all the time. HN is turning into a desperate last resort as a PleaForHelp-as-a-Service (hoping that bad PR or a company insider is the trigger).
But by now business decision-makers should really wake up and acknowledge the risk of choosing such platforms. Of course they should demand better service, but this is also something to research upfront ("is there a support desk and at least a phone number.") and then decide if they want to pay the extra money, or go somewhere else that addresses their needs.
Take this a learning experience. Once you get your business back, sit down with your team and come up with a mitigation plan because this may happen again. It's possible you may need to switch to another vendor, even one more expensive or smaller, but one with a dedicated support line and/or an account manager and stronger SLAs. Your business heavily relies on Twilio and you should have someone you can call at Twilio outside of the general support line. Maybe you need to attend their conferences and create a personal relationship with their developers or other employees so in cases like this you can backchannel your issue.
If you're going to outsource critical business infrastructure to third party SAAS vendors, make sure you have a plan for when things go tits up, because things will go tits up. It looks like Twilio screwed up here, but Twilio isn't going to suffer the consequences of this like your business will.
As far as telephony is concerned (things like SMS are a different story), a phone provider cannot legally do this, even if you signed some ToS agreement. According to the FCC, phone providers, including voip providers, have legal requirements around when they are allowed to terminate service: https://www.fcc.gov/consumers/guides/when-your-telephone-com...
They’ve turned spamming into a positive ROI business.