Ericsson said last week that "an initial root cause analysis" had indicated that the "main issue was an expired certificate in the software versions installed with these customers".
I suspect I'll never know, but I would be really interested to know what sort of certificate. Was this part of some larger public system or was it a purely internal PKI? If internal, did the certificates _do_ anything of value or did they purely make the system more fragile because somebody thought it ought to have certificates?
One of the arguments in favour of abusing the Web PKI to do other things is that tooling for the Web PKI is in a relatively good place. And a lot of other things that you might ideally hope exist actually DO exist for the Web PKI. There's independent oversight, somebody is actually reading those yawn-making audit reports, it isn't perfect but it's not just make believe either.
But on the other hand, the Web PKI is ours, and so if it suits us to change it we don't really care that this is annoying for your payment card systems, jet aeroplanes, nuclear submarines or whatever else you've duct-taped the Web PKI into. This was fun to watch with SHA-1 for example and is currently causing some fallout for names with underscores in (DNS can handle a name with an underscore in it but they're prohibited for hostnames. Lots of people ignored that, but that's their problem, not ours and they are not happy about that).
Mmm, rather than the behaviour being "removed from browsers" the important thing is that Certificate Authorities were told that issuing such certificates could cause them to be distrusted as a whole by browsers.
Part of the reason to do this is that if CAs issue the certificates then their customers buy them, and create pressure to accept them. If none of the CAs are willing to issue this never comes up.
Another is "unknown unknowns". Security systems don't play well with undefined behaviour, if we explicitly forbid everything we don't want to exist that minimises the risk of such undefined behaviour anywhere in the certificate handling code getting exploited. It's a defence in depth.
Ericsson have both their own PKI infrastructure[0], at least for software integrity checking (however that certificate is valid since March this year, and the CRL[1] it refers to is empty), in addition to using other certificate authorities for everything such as the hosting of websites to internal infrastructure[2]. I suspect it wasn't any of the above - rather it is the PKI that is used in running the IPSec networking done between carrier's RANs and other parts of core network[3], which is probably Ericsson's own internal CA.
Further expanding. GPRS Tunneling Protocol (GTP)[1] is what gets used to connected to the provider's data/voice network. This could be over any medium (wifi, GSM, UMTS, or LTE). It's likely this was the cert protecting GTP-C's ipsec tunnel[2] as without the ability to signal, pretty much everything on the network goes down.
This seems like a publicity stunt more than anything else. There's no way they actually lost 100MM here, and doing this ensures the public sees Ericsson as the real villain, and paints O2 as the victim.
I suspect business customers might have more of a stick to beat O2 over the head with, rather than average people with a cell phone contract.
e.g. Transport for London's dot matrix displays could not show bus times for the 14 or so hours the network was down, apparently iPads issued to NHS staff to manage patient reports used O2 sim cards so that couldn't work either. Probably a whole range of small to medium enterprises affected too.
Just some dirty back-of-the-napkin numbers, but theoretically, £100MM doesn't seem unreasonable.
O2 have 25MM direct customers, plus another 7MM through reseller networks — so that's 32MM people directly affected by the outage.
Presuming O2 receives an average of £30 per customer per month (which I think is a reasonable estimate), that's £940MM/month, working out to £32MM per day in a 30-day month.
O2 are refunding customers 2 days' worth of service for the outage, so that would be £64MM in lost revenue.
Add on top of that brand and reputation damage — and other factors I've no doubt forgotten to include — and £100MM doesn't seem unreasonable.
Just to clarify - MM actually just represents 'million' where a single M represents 'thousand'.
The plural part of your definition here is not really relevant.. i.e. 10M = ten thousand, 10MM = ten million, 1M = 1 thousand (not 1 million, singular)
Most people colloquially seem to use k for 'thousand' and most people would probably intuit M as 'million', but I think people still go with MM for avoidance of doubt.
One would think that a metric country would adopt "k"/kilo as the indicator for thousands and "m"/mega for millions.
Still I suppose it's better than sussing out whether someone is British and using "long" billions.
(in most of the world, "billion" = 10 to the 9th power; in older British usage still sometime seen, "billion" = 10 to the 12th power, and 10^9 is "thousand million")
Well they're french, so M stands for Mille (one thousand). MM is just a thousand squared, and MMM is a thousand qubed.
Once you understand this system you might also understand why the long system makes more sense.
1000 = Mille M
1000 000 = Million MM
1000 000 000 = Milliard MM M
1000 000 000 000 = Billion MM MM
1000 000 000 000 000 = Billiard MM MM M
in the short system the billion (nine zeros) has nothing to do with two. while in the long system it represents double as many zeros as the million. a trillion has three times the Zeros. -ard suffix? add 3 zeros
As someone who used to speak decent French, you will never convince me that the language is rational with respect to numbers. The sometimes-decimal/sometimes-vigesimal thing still makes my head spin all these years later when I hear numbers being spoken in French.
In the US, UK, and other English-speaking countries, "m" (note lower case) is used for million.
From the Guardian style guide:
"million - in copy use m for sums of money, units or inanimate objects: £10m, 45m tonnes of coal, 30m doses of vaccine; but million for people or animals: 1 million people, 23 million rabbits, etc; use m in headlines"
* Two days of an airtime contract (so e.g. if you just pay them £30 per month, that's about £2 in value for you)
* 10% discount on two types of pay-as-you-go payments with no specific limit but expect one to be announced.
The latter in particular you can expect to be heavily abused, because O2 needs the PR. I'd be astonished if that limit is any lower than £100, meaning they're giving away a £10 value.
I hadn't considered it when I wrote my earlier comment, but I wouldn't be surprised to see O2 receive their own reputational damage invoices from their reseller networks (O2, Tesco, etc.) in relation to the problems.
I think you forgot that O2 won't lose that money because they never refund to their customers. But I might be wrong. So basically the clients lose the money.
Also average monthly can't be so high maybe you're including the device cost as well.
Remember that many might have prepaid and no contracts.
Softbank's press release [0] mentions that a total of 11 countries were affected by the outage. I can't seem to find any information on what other countries were affected though.
Vietnam's second largest mobile network operator Mobifone was affected for 3 hours[1]. Now we know 3: UK, Japan and Vietnam. I'm sure it's possible track down the other 8.
Wow, Hong Kong's Smartone uses Ericsson and wasn't effected. I am surprised at how wide spread the problem is. How could Ericsson messed this up when the it has the biggest chance to grab business from Huawei.
47 comments
[ 3.3 ms ] story [ 97.7 ms ] threadOne of the arguments in favour of abusing the Web PKI to do other things is that tooling for the Web PKI is in a relatively good place. And a lot of other things that you might ideally hope exist actually DO exist for the Web PKI. There's independent oversight, somebody is actually reading those yawn-making audit reports, it isn't perfect but it's not just make believe either.
But on the other hand, the Web PKI is ours, and so if it suits us to change it we don't really care that this is annoying for your payment card systems, jet aeroplanes, nuclear submarines or whatever else you've duct-taped the Web PKI into. This was fun to watch with SHA-1 for example and is currently causing some fallout for names with underscores in (DNS can handle a name with an underscore in it but they're prohibited for hostnames. Lots of people ignored that, but that's their problem, not ours and they are not happy about that).
Can't seem to find anything on web.
On underscores: https://www.digicert.com/blog/digicert-pushes-underscore-ext...
On SHA-1: https://blog.mozilla.org/security/2015/10/20/continuing-to-p...
Part of the reason to do this is that if CAs issue the certificates then their customers buy them, and create pressure to accept them. If none of the CAs are willing to issue this never comes up.
Another is "unknown unknowns". Security systems don't play well with undefined behaviour, if we explicitly forbid everything we don't want to exist that minimises the risk of such undefined behaviour anywhere in the certificate handling code getting exploited. It's a defence in depth.
[0] https://www.ericsson.com/en/about-us/enterprise-security/pki
[1] http://crl.ericsson.net/Ericsson_Software_Deliverable_Integr...
[2] https://crt.sh/?q=%.ericsson.net
[3] https://en.wikipedia.org/wiki/System_Architecture_Evolution
[1] https://en.wikipedia.org/wiki/GPRS_Tunnelling_Protocol
[2] https://cyber-defense.sans.org/resources/papers/gsec/securin...
It's a very clever strategy, nonetheless.
e.g. Transport for London's dot matrix displays could not show bus times for the 14 or so hours the network was down, apparently iPads issued to NHS staff to manage patient reports used O2 sim cards so that couldn't work either. Probably a whole range of small to medium enterprises affected too.
O2 have 25MM direct customers, plus another 7MM through reseller networks — so that's 32MM people directly affected by the outage.
Presuming O2 receives an average of £30 per customer per month (which I think is a reasonable estimate), that's £940MM/month, working out to £32MM per day in a 30-day month.
O2 are refunding customers 2 days' worth of service for the outage, so that would be £64MM in lost revenue.
Add on top of that brand and reputation damage — and other factors I've no doubt forgotten to include — and £100MM doesn't seem unreasonable.
The plural part of your definition here is not really relevant.. i.e. 10M = ten thousand, 10MM = ten million, 1M = 1 thousand (not 1 million, singular)
Most people colloquially seem to use k for 'thousand' and most people would probably intuit M as 'million', but I think people still go with MM for avoidance of doubt.
10M, ten thousand
10MM, ten million
10MMM, ten billion
here in sec. 3: https://www.druide.com/fr/enquetes/abréviation-de-million-et...
Still I suppose it's better than sussing out whether someone is British and using "long" billions.
(in most of the world, "billion" = 10 to the 9th power; in older British usage still sometime seen, "billion" = 10 to the 12th power, and 10^9 is "thousand million")
Once you understand this system you might also understand why the long system makes more sense.
in the short system the billion (nine zeros) has nothing to do with two. while in the long system it represents double as many zeros as the million. a trillion has three times the Zeros. -ard suffix? add 3 zerosBritain does not use "long" billions. Not since the 1970s or so. Now days, billion always means 1000 million.
From the Guardian style guide:
"million - in copy use m for sums of money, units or inanimate objects: £10m, 45m tonnes of coal, 30m doses of vaccine; but million for people or animals: 1 million people, 23 million rabbits, etc; use m in headlines"
I doubt they would be compensating payment plans for devices.
* Two days of an airtime contract (so e.g. if you just pay them £30 per month, that's about £2 in value for you)
* 10% discount on two types of pay-as-you-go payments with no specific limit but expect one to be announced.
The latter in particular you can expect to be heavily abused, because O2 needs the PR. I'd be astonished if that limit is any lower than £100, meaning they're giving away a £10 value.
Puts into perspective how many customers don't change their plans when they're able to...
Also average monthly can't be so high maybe you're including the device cost as well.
Remember that many might have prepaid and no contracts.
[0] https://www.softbank.jp/en/corp/group/sbm/news/press/2018/20...
[1] (link in Vietnamese) https://congnghe.tuoitre.vn/ericsson-xin-loi-ve-su-co-sap-nh...