I'm surprised/confused: Why is it hard to detect premium rate numbers, or at least set a flag to not allow sending to them? Like, I can't think of a time when twillo should ever be sending to a premium rate number; why is this even possible?
There are 200+ jurisdictions in the phone network and everybody has their own conventions on what a "premium" number is.
For comparison, imagine if each domain in the world could set its own rates for much doing a DNS query would cost you, and governments regulated this only by designating a few second level domains as "premium". That's pretty much the scale of the problem.
Edit: To be clear, this is a very well known problem and Twilio should be doing much better at it. But it's by no means an easy problem, and all the other side needs is one (1) number to exploit.
Depends what you mean by "premium rate." Every number costs money to call in Twilio. Some numbers cost more, in lots of these frauds numbers in ordinary ranges are used (Is a rural number in Chile that costs $0.20/minute to call premium rate/fraud? Because that's what it looks like a lot of the time. How about $0.05 a minute in Austria?). IRSF, the industry term for this kind of fraud causes billions in losses a year and there is no easy answer but Twilio should probably have more infrastructure in place to reduce massive surprise bills.
Block any number that costs more than 25th percentile would be a start and so on... I can come up with plenty of heuristics that would be better than nothing.
I don't mean to do Twilio's work of defending them, but in my experience it's possible they actually don't know how much to bill the customer. What they may know is the generalized per-minute or per-session rate they've agreed with another operator alongside a general "premium rate numbers will be settled at a later date" kind of clause.
My employer got bit by this several years ago, purely on calls within the +1 country code. Before this practice was largely banned, some small carriers were allowed to designate certain rate centers as higher cost. So our VoIP carrier would say that a call to a given area code was $0.003/minute but the calls would later settle out at $0.25/minute because of a 1,000s block of numbers being (unknowing to us our our carrier) as higher cost and being settlement billed back at the higher rate.
Twilio could agree to carry some or all of this risk for its customers as part of their value-add and fees. That way, Twilio has the incentive to make the proper changes for its customers and would have the experience of looking at all of the return billed rates for all of the calls or messages across its entire customer base to help prevent toll fraud.
This is the case, the telephone billing system is perhaps the most complicated pile of softwareshit you have ever seen in your LIFE - and some of it is insane.
My favourite was getting charged for an sms my iPhone sent which was a phone home to an Apple headquarters short code for iMessage. iOS hides these from the user. Most providers don’t charge for this, but some do.
Really sucks when you carefully load 10 EUR of credit to buy a 10 EUR prepaid plan for the month and see 0,05 deducted despite being incredibly careful to not do anything that would incur a charge before buying the plan.
Apple DOES say that, when you set up FaceTime and iMessage!
There is a pop up that says "Your carrier may charge for the messages used to activate iMessage and Facetime" You can choose to not activate and do it later.
That warning actually depends on the “carrier profile”, a configuration file the phone silently fetches (or has cached in firmware builds) based on certain attributes of the SIM like the ICCID or MCC/MNC.
There’s a field in there that configured whether that warning should be shown.
Correct, and it didn't appear for carriers which were whitelisted (who zero-rated the iMessage activation SMS).
My memory, which may be wrong, is telling me that the first major version of iOS which included iMessage did not include the warning at all, and that it was added for non-whitelisted carriers (aka those which did not sell the iPhone) to prepare the user for the possibility that they will be billed, based on user feedback precisely like the comment to which I was replying.
When I was an intern, I was working for a big company on a project to optimize some call center management. Basically put mainframe reports on an intranet.
The company had made a change of some sort where whatever EDI connection between the telco and the company stopped working. I learned this when and angry facilities guy came up looking for my boss’s boss to sign for a delivery. I was the only person there, so I did. 15 minutes later, two pallets of bankers boxes came up - thousands of single sided pages of itemized call details.
Fun fact: +1 is not a country, but all of North America. For a long time it was entirely possible to dial a perfectly ordinary looking +1 258 xxxxxxx number and get charged up the wazoo because (258) is Antigua and Barbuda, not New Jersey.
Pissed me off how American Airlines wouldn't send me SMS updates to my Canadian number, even though it should cost only slightly more than than a USA number. I guess they got burned sending updates to some caribbean island number in the past.
Surely they can aggregate this across all customers though.
If Twilio cops an unexpectedly high settlement for sending an SMS to +1234567890 in January, can they assume that a separate customer sending an SMS to that number in February will end up in the same boat?
I'd be very surprised if the toll fraudsters weren't using the same numbers to hit multiple Twilio accounts.
Twilio works with phone companies across the globe; this is not something that would be that difficult for a company of their size to implement (even if it means one employee whose job it is to keep this up to date). Consider that the timezone database (a similar problem) is administered by one person (a volunteer no less)
Twilio knows which numbers will charge customers, THEY HAVE THE DATA. They can make a list of numbers that charge customers, and then have a flag that disallows SMSes to those numbers.
They also have relationships with phone providers in every market that they are in, and those providers can provide that same information and then allow a blacklist to those numbers or whatever format the premium numbers occur in.
It's not hard at all. It's a nice value-add feature and I'm sure if a competitor like MessageBird implemented something like this, it would be an easy differentiator if Twilio doesn't want to provide this.
Twilio should help their customers with this (and it looks like they do have something, but maybe not enough)... but it's also something you can do a first pass through libphonenumber metadata[1], which was pretty reasonable at my last job.
Google "international premium rate number" and you can get one in any country for free. Up to $0.7 a minute on some satellite and Albanian numbers if you opt for net 30/45 day payment in some places! This stuff happens everyday - I'm surprised no one has talked about it here till now. It is a very difficult problem to address if you want to accept international users and limit false positives.
I remember reading an article where a guy in the UK set one of these up with a bot like Lenny to make money off the scam calls. It was kind of shocking to me you could just setup any number like that, not just 900 numbers.
Wow, ok. I've been hit with this issues twice and both times, the Twilio reps failed to let me know about this. I had previously resorted to just turning off any counties. Looks like mu international app users might get functionality back.... if this feature is still live.
Do you happen to know which MaxPrice number would make sense for simple SMS? Figuring it out a reasonable value to use from Twilio's pricing pages is proving quite tricky, because of the amount of dimensions to their pricing.
When I was working in this space, all of my providers gave me a pricing feed which was essentially phone number prefix, price (and some text that my system didn't care about, I don't need a name, just a price, thanks). It looks like twilio doesn't offer that publicly for SMS, but you can see their voice price list by clicking "download voice prices (.csv)" on their voice pricing page [1]. The SMS page has a similar feed, but it only gives you price by carrier name, which isn't very helpful --- you'd need to pay to do a carrier lookup before you could use the price list; these lists don't feel complete either anyway, but it's an idea.
You can download phone prefixes with prices from them, and also set max price for an SMS (won't work if they charge just a little amount and you send thousand requests of course).
It gets better. An Ethereum address can receive any tokens, including NFTs, and has no power to reject them. A famous celebrity can have a lot of NFT spam publicly visible in their accounts, and people have no idea if they bought them or not!
As a programmatic telephone company, they're a possible (but not really probable) base for fraudulent spam calls. With KYC, and the fact that Twilio requires you call from a number you control, fraudulent calls would be easy to trace back to a person who could be charged for the calls. Much better than status quo, where it's very difficult to get to the originating phone account, and if you could, it's probably not really connected to a person.
Surely the overheads of any useful KYC are way too high for this to work? And basically nobody in this industry does KYC, so how do you propose that would meaningfully affect their interconnects?
Twilio and their users are the victim here... Twilio having KYC would not solve the problem, they do not the own the numbers the expensive texts are being sent to. Twilio should just enable their anti fraud systems by default (by the way this is no panacea, like every other anti fraud this is a cat and mouse game, there is often no clear way of telling that a number is premium rate, and many carriers are in on it too and use normal mobile number ranges).
There’s no way to set up an account such that it isn’t permitted to text premium numbers? Throttling to prevent the same number being messaged more than a certain number of times in a given window? Or throttling to prevent charges accumulating faster than a set rate?
Unfortunately, I haven't found any. There are some hacky solutions that I've bookmarked over the years, but nothing reliable enough for a production service(s). At least that I've found.
Most "alternative" SMS services a simply a façade built on top of Twilio, with the markup to prove it.
Hey! I'd naturally recommend SignalWire (as one of the founders over there.)
We have a full messaging + voice + video APIs, including a Twilio-compatible API just for people who need to switch. We're backed by companies like Deutsche Telekom, T-Mobile, and Samsung so we know how to make telecom infra!
We're also the folks behind the open-source FreeSWITCH framework that powers companies like Bandwidth, Five9, Dialpad, Zoom Voice... maybe even your own company's PBX!
I mean, seriously. Twilio could allow (verified-ownership emails only) for the simplest possible email integration into twiml for the sole purpose of alerting and paging, etc.
No spam possible since it's only verified account-controlled emails. Basically sending email to yourself from within twiml.
But no. Instead they bought sendgrid and email integration is a complete abomination of a two-company, two platform, two accounts workflow that is fragile and fails all the time.
They start out insanely great when riding high on VC cash, but over time they become the gorilla in the room and their service and support and quality begins to diminish until they're more annoying than they are helpful.
In fact, I built my own little personal telco around their services and API.
It is all slowly falling apart, however, as their failure to build out their infrastructure offerings (as opposed to their "customer engagement" offerings) and the regulatory SNAFUs[1][2] that are emerging as a result of their behavior erode all of my use-cases ...
All I wanted was more, and more useful, twiml verbs ... instead I got "customer engagement workflows".
Their support for registering 800 numbers and verifying your identity is absolute 100% trash. I've filled out their verify form many times, every time to be prompted to do it all over again months later. Over and over again.
I was a HUGE Twilio booster back in the day. I encouraged its use heavily in my very large org because it was so much better than our entrenched phone provider. I was (maybe still am) featured on their website. But things have just gotten more..business-y, less developer focused, just..generally worse.
I don't know what to do with these guys anymore. I understand they're trying to grapple with the mess of regulations that got dumped on them, but their messaging and support around all this is BAD. There's no answers, just walls of canned text.
The fact the Twilio is allowing toll numbers at all is clearly their fault, not that of their customers. Turning around and claiming that customers should be paying for twilio’s bad choices is BS
These numbers are usually not premium in the 1-900 sense of the word. It's more like they are international numbers and there are various intermediaries who work with mobile/landline operators in a bunch of countries to set up these kind of numbers and split the revenue from incoming calls/texts if they can deliver lots of minutes to them. One way of doing so is by getting a bunch of 2fa texts sent.
Isn't this something Elon Musk brought up a few weeks ago when Twitter SMS 2FA stopped working in some countries? (India? I think?). On a Twitter spaces he said they were losing millions to SMS fraud for years and found out that some Telecom companies were complicit so they just cut off all SMS traffic to those companies until they re-negotiated terms.
Last time I tried to sign up for Twitter it demanded I verify my account with text messages. Actually, virtually all services do this now when creating an account. The worst (Microsoft for example) let you sign up and use the account for a bit (possibly purchasing some items tied to the account) and then extort the phone number out of you later to maintain access.
It is sort of amusing that these companies hitched their wagon to the now scam laden telephone network to track users and ended up getting scammed themselves.
TBH I would consider this type of fraud of a more Robin Hood variety. Companies that still encourage weak security practices like sms 2fa (or even worse, just hoover your PII under the guise of it) should be defrauded of their money as much as possible.
This is illegal under GDPR. After someone has signed up for a service you can't then demand additional personal information as a condition of continuing to supply the service.
Yep, I remember getting pinged by a coworker asking why is our Twilio bill so high all of sudden. It turns out to be Toll Fraud through 2FA messages. Malicious actors sign up new accounts and setup 2FA number and just keep requesting 2FA through SMS to profit.
Twilio managed to convince everyone that SMS based auth was a good idea but it's always been a bad idea. Drop twilio and go back to using passwords and use a different 2fa method.
Were falling through the computer literacy gap between SMS MFA and authenticator/Yubikey MFA at the moment. While an IT person can do password managers (with secure backups) authenticators, passkeys, and biometrics, all with half-decent opsec, the average user can barely do more than a couple of passwords for everything, and SMS MFA.
It's absolutely critical that the companies we support vastly improve their security, but there's no way to get there from here with their staff, lack of any established processes, and zero training infrastructure.
Your only power to encourage them to fix this is to do the thing they're begging you not to: dispute the charges.
If a threshold of Twilio customers dispute charges, Twilio loses the ability to process credit cards at a lower risk rate, then with all but high risk processors, then may lose the ability to process them at all.
If enough of their customers are getting burned, and enough dispute, Twilio would no longer be able to accept credit cards. They are terrified of that, so begging you not to dispute charges for their lack of fraud prevention.
You accepting anything less than full refund of all fraudulent use they're cascading back on you is a gift to them. You accepting less than a full refund, while not dinging them at all with a chargeback is also a gift to them. If they don't want to give you the full refund for misuse they should be preventing, dispute it, as is your right.
The correct course for Twilio is for Twilio to refund these charges no questions asked while fixing the problem.
To lose the ability to process MasterCard or Visa credit cards takes a few months. You can rack up big fines during that time though. If they get put into the probation period that would raise red flags with some execs, assuming people are communicating these things.
Further, the customer has the right to dispute credit card charges thanks to the agreement between customer and card provider and between card provider and merchant.
Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
> Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
This isn't true. Visa/Mastercard care about your chargeback rate. You can block a customer who's done a chargeback. I'm sure the card networks have rules around what you cannot do as a result of a chargeback but you can stop providing services to a customer who has done a chargeback.
Disclosure: I engineered and delivered a high risk payment processing gateway to a firm specializing in being a cc processor of last resort, also working with merchant banks of last resort.
To be clear, my views on this are not legal counsel, they are simply from having worked in this area for a decade before becoming CTO at a global bank.
>Your only power to encourage them to fix this is to do the thing they're begging you not to: dispute the charges.
I'd check their TOS to see if they offer some kind of arbitration option. As noted in other threads, triggering that process can be a surprisingly effective way to make someone from the company actually engage with the issue. Disputing the charges is always a nuclear option. They may never do business with you after that.
Is there anything in the arbitration clauses forbidding them from cancelling your account if you invoke arbitration (regardless of whether you prevail or fail?).
At the individual dispute level but in the long run arbitration means you don't lose your risk level which will almost always cost you a lot more than whatever the actual arbitration/credit disputes cost.
I suspect (but may be wrong, I don't know how trigger-happy the risk level changes are) the absolute number of events needed to trigger a risk level loss at the scale of Twilio would also represent a catastrophic number of arbitration cases.
> Disputing the charges is always a nuclear option. They may never do business with you after that.
This is something that I think needs to be regulated. I'm not saying that this should be the case for a company the size of Twilio, but I definitely think that a company the size of Apple/Google/Samsung should not be able to ruin your life because you had temerity to stand up to them and dispute a charge.
Twilio has a neat platform, but their standards are very low or nonexistent when you experience jitter, large audio buffers or routing issues.
Providing PCAPs and reproducing routing issues doesn't result in support addressing these issues. Many other IPES and CLECs will actually fix these issues when documented.
As I see it, the problem is that those companies are effectively monopolies in fields most of us depend on. If Apple or Google refuses service to someone previously reliant on their services, they could be locked out of accounts on a plethora of other services, have their payments to third parties disabled, lose access to major means of communication, and more.
The last generation of big tech monopoly never had that kind of power. Microsoft couldn't even do much to block someone from using its products as most of them were sold through third parties and didn't require network services to operate.
Shouldn't have "bought" them in the first place. Buy physical books, or DRM-free books on sites like gumroad, or just pirate. Don't give in to the rent seeking business of pretending to sell you what you can't own. If you like your Kindle device, you can use KOReader to read epubs and reduce dependence on Amazon. If I sue the bookstore, they can't just take all the books away, but if you dispute a charge with Amazon, they'll do it because the ToS says they can. At the very least, try downloading and de-DRMing all the books you received from them: https://github.com/noDRM/DeDRM_tools
That Twillio doesn't protect you is bad. However, would a court agree you don't owe them the money? This recommendation seems like abuse of disputing a charge and will just get you banned from Twillio.
And then you switch to another SMS provider, which may be costly from an engineering perspective, but clearly worth it if you’re getting slammed with botnets and Twilio doesn’t care.
The court doesn't have to agree, only the card provider does.
The customer has the right to dispute credit card charges thanks to the agreements between customer and card provider and between card provider and merchant.
Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
This is why you always pay for sketchy merchants with a card, it's one of the few consumer powers you have.
Why would the card providers be in the customer's favor. The customer paid for a text to be delivered to a phone number and Twilio did that and then charged the customer for it.
If you pay someone to mow your lawn, then they mow your lawn and charge you. You can't just chargeback after the fact to get that service for free.
In your analogy it would be more like paying someone to mow your lawn because your neighbour got it done for $10, then being charged $100 because your house number is even.
It might be in the terms and conditions, but it’s bad faith to not give any warnings or controls before the services are rendered.
The people adjudicating disputes at card companies are akin to content moderators hired by social media companies, they aren't experts and do not spend much time reading up on the dispute. The decisions are more or less random, with a heavy bias towards the customer.
>The court doesn't have to agree, only the card provider does.
If it's a sufficiently large amount, Twilio will collect the money from you via other channels. Them losing a credit card dispute does not release you from liability.
>Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
This is simply not true. Chargeback blacklists are standard and not prohibited by merchant agreements.
There was an online service subscription I had several years back (not Twilio, but something somewhat similar) that I stopped needing. I forget if there either wasn't any way to this on their website or if it was just broken, but after repeatedly trying and failing, I ended up just disputing the charge the next time the monthly payment came up, and my bank stopped the transactions from that point on, so I forgot about it. A while later (probably around 6-12 months), I got an email from that company asking me to go back and tell my bank that the transactions were fine. Given that it had been so long and I only disputed the charge due to their website not giving me the ability to cancel through them, I didn't think it was worth trying to talk to someone at the bank to figure out how that would even work. It left an impression on me because it was the first time that I felt like I actually had any power in my relationship with a company as a customer rather than just having to hope that the company would chose to do right by me with nothing forcing them to.
From both a user and provider perspective I hate that individual companies are implementing 2FA at all. I don't want another database with my phone and a password in it. I wish Mozilla Persona had took off, or any other auth standard.
You chose to require an SMS OTP for your customers. It is not straightforward at all that the burden of filtering your customers would fall on your provider and not on you -- actually, if the provider you chose does explicitly not provide that filtering, it's effectively on you.
(I have to say that if I were Twilio, I would not have added the "fraud prevention" toggle, because now they can be deemed to be providing that service.)
We've been hit by this at work as well. We had to add CAPTCHA and a several other techniques to defend against this.
How it works:
1. Attacker leases 1 or more premium rate numbers in an international country.
- Attacker can lease a premium rate number for as little as $10/month
- Typically, the attacker gets to keep 70% of the money generated by the premium rate number.
2. Attacker then finds companies with OTP (One-Time Passcodes) or 2FA (Two-Factor Authentication) endpoints that require no validation and writes a script to automate the webpage or call the API endpoint
- Attacker will typically obtain a new IP address per API call using a VPN or a rented botnet from the dark web.
3. If the premium rate number costs 10 cents, then each successful text message they can send to the number generates 7 cents for them.
4. The attacker then just needs to send 150 SMS to the premium rate number to break-even on their $10 investment, not counting the cost of the VPN or rented botnet.
There is a lot of money to be made here by an attacker unfortunately. :(
Maybe on prepaid plans? Been a while since I've heard of SMS costing anything on subscription plan, outside of roaming charges. Mobile Internet effectively cannibalized that income stream for the phone companies.
In Germany the standard price for an SMS is 9 cent, of which somewhere around 2-3 cents are paid to the recipient's carrier. Unlimited plans are common, but only because nobody texts anymore (same applies for phone calls).
You’re talking about the fee your carrier charges for normal texts. These are “premium” charges, meaning the user is charged an extra fee on their bill regardless of their SMS billing plan.
I would imagine there are rules/regulations about a SMS provider blocking communications before fraudulent behavior is determined? Not saying it shouldn't/couldn't be done, but probably one of those things with a simple tech fix but a complicating social/business aspect.
It could be an option in the API call with a default in account settings. I bet most people who are trying to reduce spam accounts by requiring a phone number would actually prefer to exclude these numbers anyways.
surely not if the customer _explicitly requests_ that the communications are blocked? iirc in Aus it was possible to have your provider block messages to premium rate numbers back in the days when it was popular to buy ringtones.
I agree with other comments here. $0 is the minimum amount people should be willing to pay if they're not disputing charges or reporting fraud to the credit card networks / regulators.
No idea. I think there's a real problem with the whole design of premium numbers because I'm not sure how one is even supposed to know when payment is required or meaningfully accept it, though at least the API apparently allows this.
FWIW, I do think $0 might make a sane default, but you do understand that the user would have to change it from $0 before they could use the account, right? The whole point of using Twilio to send an SMS is because you wanted the SMS to actually be sent, which means you are going to have to pay for the SMS, and SMS is always stupidly expensive.
It isn't just as simple as 'premium rate numbers'.
Some of the criminals behind these attacks will have access to the phone network. They'll pick an expensive route, like a range of phone numbers in Georgia (the country) from the USA, and offer a cheaper route to it. The system will start using their route for those calls. They'll accept all calls to that route, get paid, and never actually connect any calls.
That gives them a range of "normal" phone numbers which helps them avoid throttling on just one number. But they can be just as expensive as premium numbers to call.
At least, this is how it was explained to me as my team fought these attacks a couple years ago. We'd see calls to a large range of a few thousand numbers. Couldn't throttle on a single number.
How is this a workable system!? Why would anyone pay them. This seems like fraud on the part of the phone networks for billing for service that was never provided or should have been provided cheaper.
while I don't agree with sanctions, this seems like the kind of time where you just block off a country/exchange entirely if you cannot have the confidence of what things cost to send there.
I think you're conflating toll bypass fraud with IRSF. A grey route that never delivered any calls or only a fraction of them would have bad ACD numbers and people would not use that route. With hacked Asterisk/FreePBX boxes people usually call the international numbers described in OP and split the termination fee with some corrupt carrier/intermediary. There is a related fraud where people use the hacked Asterisk/FreePBX boxes to terminate calls, which from what I understand these actually have pretty good quality until the unwitting owner gets a $40,000 phone bill and shuts everything off. Traditional toll bypass fraud is when countries are expensive to call internationally but have cheap local calls, so people in those countries buy a bunch of sim cards, put them in a box with a bunch of gsm modems, and use those to basically "convert" an expensive international call to a cheap local call (and profit the difference between the two rates).
Edit: Oh, you're talking about number hijacking. I think they usually aren't offering termination services though, usually it goes hand in hand with the kind of fraud described in the OP.
> Traditional toll bypass fraud is when countries are expensive to call internationally but have cheap local calls, so people in those countries buy a bunch of sim cards, put them in a box with a bunch of gsm modems, and use those to basically "convert" an expensive international call to a cheap local call (and profit the difference between the two rates).
Is this really fraud? Is it fraud to offer any VOIP service, or only when it can connect to the phone network, like Skype?
I guess I could see how it might be against the T&C's of the telecom company, to offer a service that undercuts them, but hardly a criminal act of deception.
Fraud is what the government decides it is, the governments have deemed this to be fraud.
Telecom companies don’t necessarily care about this, it’s often the governments who want to tax incoming international calls as an easy revenue source.
I consider it to be relatively harmless but how it is classified depends on the country. India is pretty cheap to call even absent simboxes but they still crack down on the practice for “national security reasons” because it makes tracking people more difficult. The UK (Ofcom) banned them outright for some reason a long time ago but that’s being appealed. In some African countries the laws are pretty vague and do not outright ban them, usually they charge people with “unregistered telecommunications business” or something like that.
I don't understand why twilio cannot simply set a flag on their phone company account saying "under no circumstances will we pay for these shenanigans", and why the phone company billing stuff cannot simply block sms messages to such scam accounts.
In particular, email (smtp) to sms gateways exist. Why doesn't twilio just use one of those (and maybe pre-arrange a flat monthly payment to avoid being blocked for going over quota).
> I don't understand why twilio cannot simply set a flag on their phone company account saying "under no circumstances will we pay for these shenanigans", and why the phone company billing stuff cannot simply block sms messages to such scam accounts.
Everyone has the idea "just don't pay for fraud" but in practice it is difficult because there are many different carriers in the typical international call chain, which means to dispute charges you need everyone to agree. Also carriers have long term agreements with eachother about billing and it is not as easy to just dispute the charges like you can with a credit card.
We've been hit by this exact issue, especially over the last month.
We tried to mitigate as cleanly as possible for our users, adding one-time nounces to signup requests, adding rate-limiting rules, locking down regions, but we still faced an onslaught of tens of thousands of fraudulent signups per day. On our tier we don't have the ability to set block rules ourselves - it requires a support request that takes 2-3 days to get a response on. Our choices are to eat thousands of dollars per day in toll fraud, or disable sign-ups until we can add more fraud prevention on top of what Twilio enables. The problem is the fraudsters are using real browsers across thousands of IPs located in dozens of different countries.
Similar to the OP, Twilio tries to say this is our fault and leaves it up to us to both pay for the issue and to try and fix it.
We were trying to avoid the use of a captcha; originally believing that our API infrastructure was the target. A captcha did end up being the solution, but is not particularly user friendly, and I was also trying to avoid pulling developers out of bed on Christmas to implement - but we're protected now!
They should be better equipped to detect and prevent the abuse. It's an order of magnitude higher request volume for phone #s located in remote regions of the world. Twilio knows full-well where those numbers go, and can see them being abused simultaneously across many customers. I don't possess the same ability to know this... unless I use Twilio to run a reverse-lookup, which would of course still incur a cost.
Yes, it would be helpful if they helped fight abuse, but that is not necessary of them. Having the capability is a competitive advantage so it would be in their interest to invest in it.
If anyone's facing this in their auth flows, we're happy to help at https://clerk.dev
We're in the same cat-and-mouse game with the attackers as everyone else, but since we're an auth company, we have full-time folks monitoring for issues and resolving when they come up.
It's worth mentioning that Twilio is in an understandably tough position here. They only receive API requests from your server, and real requests look the same as attack requests except for the phone number.
Clerk is in a better position to help because our API accepts traffic directly from the attacker (e.g. POST /verify-phone-number). We know their IP, user agent, whether they're connecting from AWS, etc, etc. We very much rely on this data to help stop them.
When I recently wrote Twilio code the first thing I did was add in as much stuff as I could to prevent this sort of thing happening. I think I put in captcha and also IP address throttling and request counting.
At the time I wondered if I was overengineering or gold plating but apparently not.
I do seem to recall that Twilio writes about this issue quite alot and includes strategies in its best practices for avoiding the issue.
We’ve had the same problems. We use Twilio for SMS based OTP login, lost lots of money to toll fraud, and spent lots of time putting up various mitigation strategies to reduce it. Now we only lose a bit of money to toll fraud, but if was lots of engineering effort and $$ down the drain.
My main suggestion would be to avoid any sort of flow, like SMS OTP login, that allows triggering SMS messages without being logged in. Just do a more traditional login, SMS OTP isn’t worth the headaches.
Haven’t tried Twilio Verify, didn’t exist when we were solving these problems ourselves. But like most fraud prevention, it’s probably far from perfect, better to just avoid fraud-prone workflows if you can.
I spent a lot of time playing cat and mouse with this type of toll fraud in 2022.
1. Rate limited SMS by number/ip: bypassed by large number of proxies/vpn.
2. Added captcha: bypassed by attacker manually signing up thousands of accounts (mechanical turks?) over months and then iterating over them for login OTP.
3. Identifying what carriers/operators are involved and blocking them asap (usually obscure ones).
4. Careful monitoring of SMS send rates and alerting of anomalies to investigate.
Good advice. By the way, the reason captcha didn't stop it is because Recaptcha is $2 per 1000 solves on 2captcha.com (or any other solving service), at $0.02/SMS this only lowers their profitability by 10%.
220 comments
[ 2.5 ms ] story [ 241 ms ] threadIf you want to read the tweet on how it works.
For comparison, imagine if each domain in the world could set its own rates for much doing a DNS query would cost you, and governments regulated this only by designating a few second level domains as "premium". That's pretty much the scale of the problem.
Edit: To be clear, this is a very well known problem and Twilio should be doing much better at it. But it's by no means an easy problem, and all the other side needs is one (1) number to exploit.
They know how to charge you for these numbers so apparently they do have that data, no?
They know how much to bill the customer, so they must know how much it costs to send to a number.
I don't mean to do Twilio's work of defending them, but in my experience it's possible they actually don't know how much to bill the customer. What they may know is the generalized per-minute or per-session rate they've agreed with another operator alongside a general "premium rate numbers will be settled at a later date" kind of clause.
My employer got bit by this several years ago, purely on calls within the +1 country code. Before this practice was largely banned, some small carriers were allowed to designate certain rate centers as higher cost. So our VoIP carrier would say that a call to a given area code was $0.003/minute but the calls would later settle out at $0.25/minute because of a 1,000s block of numbers being (unknowing to us our our carrier) as higher cost and being settlement billed back at the higher rate.
Twilio could agree to carry some or all of this risk for its customers as part of their value-add and fees. That way, Twilio has the incentive to make the proper changes for its customers and would have the experience of looking at all of the return billed rates for all of the calls or messages across its entire customer base to help prevent toll fraud.
There have been people who got printed bills from their cell phone provider for every single kilobyte of data, each individually indexed and billed: https://en.wikipedia.org/wiki/300-page_iPhone_bill
Really sucks when you carefully load 10 EUR of credit to buy a 10 EUR prepaid plan for the month and see 0,05 deducted despite being incredibly careful to not do anything that would incur a charge before buying the plan.
There is a pop up that says "Your carrier may charge for the messages used to activate iMessage and Facetime" You can choose to not activate and do it later.
There’s a field in there that configured whether that warning should be shown.
My memory, which may be wrong, is telling me that the first major version of iOS which included iMessage did not include the warning at all, and that it was added for non-whitelisted carriers (aka those which did not sell the iPhone) to prepare the user for the possibility that they will be billed, based on user feedback precisely like the comment to which I was replying.
The company had made a change of some sort where whatever EDI connection between the telco and the company stopped working. I learned this when and angry facilities guy came up looking for my boss’s boss to sign for a delivery. I was the only person there, so I did. 15 minutes later, two pallets of bankers boxes came up - thousands of single sided pages of itemized call details.
(FreeConferenceCall and similar companies lobbied heavily against this rule, but AT&T and Verizon were able to lobby it through.)
Ugh.
If Twilio cops an unexpectedly high settlement for sending an SMS to +1234567890 in January, can they assume that a separate customer sending an SMS to that number in February will end up in the same boat?
I'd be very surprised if the toll fraudsters weren't using the same numbers to hit multiple Twilio accounts.
Twilio knows which numbers will charge customers, THEY HAVE THE DATA. They can make a list of numbers that charge customers, and then have a flag that disallows SMSes to those numbers.
They also have relationships with phone providers in every market that they are in, and those providers can provide that same information and then allow a blacklist to those numbers or whatever format the premium numbers occur in.
It's not hard at all. It's a nice value-add feature and I'm sure if a competitor like MessageBird implemented something like this, it would be an easy differentiator if Twilio doesn't want to provide this.
[1] https://github.com/google/libphonenumber/tree/master/metadat...
Here’s a story from 2005:
https://www.forbes.com/forbes/2005/0919/058.html?sh=5531184c...
At the least, there should be a list of phone numbers known to not result in surprise charges so you can block all others.
[1] https://www.twilio.com/voice/pricing/us
E.g. why don't they have KYC controls during account opening? Because it would reduce the # of people who open an account.
They retain the right to any/all the upside of any risk/scale, and you retain the obligation in any downside. No questions.
It's despicable.
Most "alternative" SMS services a simply a façade built on top of Twilio, with the markup to prove it.
https://wavecell.com/sms/
While they are mostly known in Asia but they offer service in Europe/North America as well.
https://www.thinq.com/sms-mms-text-messaging/
We have a full messaging + voice + video APIs, including a Twilio-compatible API just for people who need to switch. We're backed by companies like Deutsche Telekom, T-Mobile, and Samsung so we know how to make telecom infra!
https://signalwire.com/products/cloud-messaging
We're also the folks behind the open-source FreeSWITCH framework that powers companies like Bandwidth, Five9, Dialpad, Zoom Voice... maybe even your own company's PBX!
https://freeswitch.com
Thanks
Do you have a god-damned email verb for twiml ?
I mean, seriously. Twilio could allow (verified-ownership emails only) for the simplest possible email integration into twiml for the sole purpose of alerting and paging, etc.
No spam possible since it's only verified account-controlled emails. Basically sending email to yourself from within twiml.
But no. Instead they bought sendgrid and email integration is a complete abomination of a two-company, two platform, two accounts workflow that is fragile and fails all the time.
So ... do you ?
I work with Twilio a lot and kind of agree with your last sentence but what do you mean? Are these normal business practices?
There's a very obvious pattern on the spectrum of | Asset -> Liability |, and sadly, time proves they're much closer to a liability than an asset.
Over time, tech becomes a tax.
Or, more accurately, that they have become normal, rather than being objectively normal.
Just FYI SendGrid is a Twilio product LOL, so I guess you are right.
I did as well.
In fact, I built my own little personal telco around their services and API.
It is all slowly falling apart, however, as their failure to build out their infrastructure offerings (as opposed to their "customer engagement" offerings) and the regulatory SNAFUs[1][2] that are emerging as a result of their behavior erode all of my use-cases ...
All I wanted was more, and more useful, twiml verbs ... instead I got "customer engagement workflows".
[1] https://twitter.com/rsyncnet/status/1593384850073214976
[2] 10DLC A2P makes personal/hobbyist Twilio usage basically impossible (although mine continue to work ...)
I was a HUGE Twilio booster back in the day. I encouraged its use heavily in my very large org because it was so much better than our entrenched phone provider. I was (maybe still am) featured on their website. But things have just gotten more..business-y, less developer focused, just..generally worse.
I don't know what to do with these guys anymore. I understand they're trying to grapple with the mess of regulations that got dumped on them, but their messaging and support around all this is BAD. There's no answers, just walls of canned text.
It is sort of amusing that these companies hitched their wagon to the now scam laden telephone network to track users and ended up getting scammed themselves.
It's absolutely critical that the companies we support vastly improve their security, but there's no way to get there from here with their staff, lack of any established processes, and zero training infrastructure.
If a threshold of Twilio customers dispute charges, Twilio loses the ability to process credit cards at a lower risk rate, then with all but high risk processors, then may lose the ability to process them at all.
If enough of their customers are getting burned, and enough dispute, Twilio would no longer be able to accept credit cards. They are terrified of that, so begging you not to dispute charges for their lack of fraud prevention.
You accepting anything less than full refund of all fraudulent use they're cascading back on you is a gift to them. You accepting less than a full refund, while not dinging them at all with a chargeback is also a gift to them. If they don't want to give you the full refund for misuse they should be preventing, dispute it, as is your right.
The correct course for Twilio is for Twilio to refund these charges no questions asked while fixing the problem.
I strongly encourage Twilio customers to pursue this route if Twilio is charging them for fraudulent charges.
That’s like nuclear meltdown.
Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
This isn't true. Visa/Mastercard care about your chargeback rate. You can block a customer who's done a chargeback. I'm sure the card networks have rules around what you cannot do as a result of a chargeback but you can stop providing services to a customer who has done a chargeback.
To be clear, my views on this are not legal counsel, they are simply from having worked in this area for a decade before becoming CTO at a global bank.
I'd check their TOS to see if they offer some kind of arbitration option. As noted in other threads, triggering that process can be a surprisingly effective way to make someone from the company actually engage with the issue. Disputing the charges is always a nuclear option. They may never do business with you after that.
The arbitration is painful because of the costs (financial and staffing/handling) of the process, not the outcome.
This is something that I think needs to be regulated. I'm not saying that this should be the case for a company the size of Twilio, but I definitely think that a company the size of Apple/Google/Samsung should not be able to ruin your life because you had temerity to stand up to them and dispute a charge.
Providing PCAPs and reproducing routing issues doesn't result in support addressing these issues. Many other IPES and CLECs will actually fix these issues when documented.
The west still prospered when consumer rights were given priority over business.
The last generation of big tech monopoly never had that kind of power. Microsoft couldn't even do much to block someone from using its products as most of them were sold through third parties and didn't require network services to operate.
No.
> DRM-free books on sites like gumroad
I like specific authors.
> or just pirate
I like specific authors and want them to get paid.
Rent the book on Amazon as you already are. But then download a DRM-free copy from somewhere, uhhh, free.
Absolutely, dispute the charge.
The customer has the right to dispute credit card charges thanks to the agreements between customer and card provider and between card provider and merchant.
Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
This is why you always pay for sketchy merchants with a card, it's one of the few consumer powers you have.
If you pay someone to mow your lawn, then they mow your lawn and charge you. You can't just chargeback after the fact to get that service for free.
It might be in the terms and conditions, but it’s bad faith to not give any warnings or controls before the services are rendered.
If it's a sufficiently large amount, Twilio will collect the money from you via other channels. Them losing a credit card dispute does not release you from liability.
>Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
This is simply not true. Chargeback blacklists are standard and not prohibited by merchant agreements.
You chose to require an SMS OTP for your customers. It is not straightforward at all that the burden of filtering your customers would fall on your provider and not on you -- actually, if the provider you chose does explicitly not provide that filtering, it's effectively on you.
(I have to say that if I were Twilio, I would not have added the "fraud prevention" toggle, because now they can be deemed to be providing that service.)
How it works:
There is a lot of money to be made here by an attacker unfortunately. :(Anyone with enough determination can execute a sybil attack on any service that doesn't require in-person verification.
wut, the absolutely most ordinary (in the realm of single telecom) text costs me ~6.5 cents
I'm in the US and any of the big carriers offer unlimited texting as a baseline, and we have pretty crappy carriers compared to a lot of the world.
That said, I don't care, since I literally can not remember the last time I sent an SMS. It must have been years ago.
If they can identify the premium numbers for billing, they should be able to identify them for blocking.
https://www.twilio.com/blog/2015/08/introducing-max-price.ht...
Apparently a lot of people could really use that info.
I agree with other comments here. $0 is the minimum amount people should be willing to pay if they're not disputing charges or reporting fraud to the credit card networks / regulators.
Time is money, after all.
Some of the criminals behind these attacks will have access to the phone network. They'll pick an expensive route, like a range of phone numbers in Georgia (the country) from the USA, and offer a cheaper route to it. The system will start using their route for those calls. They'll accept all calls to that route, get paid, and never actually connect any calls.
That gives them a range of "normal" phone numbers which helps them avoid throttling on just one number. But they can be just as expensive as premium numbers to call.
At least, this is how it was explained to me as my team fought these attacks a couple years ago. We'd see calls to a large range of a few thousand numbers. Couldn't throttle on a single number.
Edit: Oh, you're talking about number hijacking. I think they usually aren't offering termination services though, usually it goes hand in hand with the kind of fraud described in the OP.
Is this really fraud? Is it fraud to offer any VOIP service, or only when it can connect to the phone network, like Skype?
I guess I could see how it might be against the T&C's of the telecom company, to offer a service that undercuts them, but hardly a criminal act of deception.
Telecom companies don’t necessarily care about this, it’s often the governments who want to tax incoming international calls as an easy revenue source.
I am curious if the Twilio 'lookup' API call will identify it as such:
... which would be a very fast and simple way to validate a number before you (or your process) use it ...In particular, email (smtp) to sms gateways exist. Why doesn't twilio just use one of those (and maybe pre-arrange a flat monthly payment to avoid being blocked for going over quota).
Everyone has the idea "just don't pay for fraud" but in practice it is difficult because there are many different carriers in the typical international call chain, which means to dispute charges you need everyone to agree. Also carriers have long term agreements with eachother about billing and it is not as easy to just dispute the charges like you can with a credit card.
Hmmmm... haven't encountered that phrase before.
We tried to mitigate as cleanly as possible for our users, adding one-time nounces to signup requests, adding rate-limiting rules, locking down regions, but we still faced an onslaught of tens of thousands of fraudulent signups per day. On our tier we don't have the ability to set block rules ourselves - it requires a support request that takes 2-3 days to get a response on. Our choices are to eat thousands of dollars per day in toll fraud, or disable sign-ups until we can add more fraud prevention on top of what Twilio enables. The problem is the fraudsters are using real browsers across thousands of IPs located in dozens of different countries.
Similar to the OP, Twilio tries to say this is our fault and leaves it up to us to both pay for the issue and to try and fix it.
It would be valuable if they let you avoid texting premium numbers, but that's just a feature on top of the service they provide.
Where do you find this? I've spent 10 minutes on our Twilio account and couldn't find the toggle.
We're in the same cat-and-mouse game with the attackers as everyone else, but since we're an auth company, we have full-time folks monitoring for issues and resolving when they come up.
It's worth mentioning that Twilio is in an understandably tough position here. They only receive API requests from your server, and real requests look the same as attack requests except for the phone number.
Clerk is in a better position to help because our API accepts traffic directly from the attacker (e.g. POST /verify-phone-number). We know their IP, user agent, whether they're connecting from AWS, etc, etc. We very much rely on this data to help stop them.
At the time I wondered if I was overengineering or gold plating but apparently not.
I do seem to recall that Twilio writes about this issue quite alot and includes strategies in its best practices for avoiding the issue.
My main suggestion would be to avoid any sort of flow, like SMS OTP login, that allows triggering SMS messages without being logged in. Just do a more traditional login, SMS OTP isn’t worth the headaches.
Haven’t tried Twilio Verify, didn’t exist when we were solving these problems ourselves. But like most fraud prevention, it’s probably far from perfect, better to just avoid fraud-prone workflows if you can.
1. Rate limited SMS by number/ip: bypassed by large number of proxies/vpn.
2. Added captcha: bypassed by attacker manually signing up thousands of accounts (mechanical turks?) over months and then iterating over them for login OTP.
3. Identifying what carriers/operators are involved and blocking them asap (usually obscure ones).
4. Careful monitoring of SMS send rates and alerting of anomalies to investigate.