Absolutely. Consumer and enterprise alike, EFTPOS, internet, landlines, mobiles, roaming, all down. Hospitals and train networks heavily impacted as well.
It baffles me how they can have one single point of failure for their entire national cellular + fibre voice & data network, and for it to take so long to fix.
Some sort of DNS issue? Maybe a botched change pushed out network wide? Surely if it was a change pushed out it would be rolled back by now?
Edit: On second thought, if they've got no network & no out-of-band - it may take a while... they may need to physically visit each site to re-install the backed-up version!
I think that’s how they respond to a major issue- they take down all the radio connections while they isolate the issue. Otherwise reconnect attempts will overwhelm the systems
For Americans, this is comparable to Verizon going down. Optus is our second largest telco, with 10 million customers including many hospitals, trains, enterprises, etc… I myself am currently unable to make calls and use mobile internet.
The outage apparently began 4am AEDT. It's now 11am. For 7 hours and counting - no network at all in large parts of the country including no mobile network at all (if not all - I haven't heard of anywhere where it is working fine).
The impact is huge. As others have mentioned it's impacting financial services networks and in Melbourne, it's impacting the Metro train network (they use Optus for their backup communications).
The very widespread nature makes me think "there's gotta be more than one failure here" e.g. can't just be a DNS issue. Follow up thought - "could this be a non-accident?" i.e. a deliberate outage on behalf of actors unknown?
The 4am start time kind of makes it sound like a scheduled change gone wrong. Maybe so wrong that they lost access to the control plane and needed to send engineers on-site to fix? I think if it had been malicious we would have heard about it by now..
From an interview with ABC (Australian public broadcaster/news network) with the Optus CEO
---
ABC: Can you tell us why the system's gone down?
Optus CEO: Unfortunately, I don't have more information to give at this stage. We have had issues since 4am. The team has tried a number of parts of restoration and so far we have not had the results that we have hoped for. And we're pursuing every avenue to get everybody back online as soon as possible.
ABC: Do they know what happened though? Who's there the team that you've got working on it?
Optus CEO: Our team is still pursuing every possible avenue. We had a number of hypotheses and each one so far that we've tested and put in place new actions for has not resolved the fundamental issue. So we're still working on it. And when we have a identified root cause and a time for restoration, we'll be updating everybody as soon as we can.
---
Pretty wild to be down for over 7 hours no with no sign of a fix yet. Note they have also said they don't believe it's a 'cyber attack' (Optus had an enormous data breach last year with 9.7 million customers data leaked: https://en.wikipedia.org/wiki/2022_Optus_data_breach )
Reports from a third party ISP is that Optus are flying technicians to at least eight sites across the country, and that it’s a problem with their BGP route reflectors. Facebook faced a similar problem last year IIRC.
“An Optus source, who did not wish to be named because they were not authorised to speak publicly, said a BGP prefix flood from a peer was likely causing the issues on the telco’s core network.”
“Our on-site technician is actively prioritising establishing a console connection [a physical cable connection],” the message to Optus’ partners early on Wednesday said. “Rest assured that said technician is also being provided additional technical support remotely.”
24 comments
[ 4.4 ms ] story [ 69.1 ms ] threadEdit: On second thought, if they've got no network & no out-of-band - it may take a while... they may need to physically visit each site to re-install the backed-up version!
The impact is huge. As others have mentioned it's impacting financial services networks and in Melbourne, it's impacting the Metro train network (they use Optus for their backup communications).
The very widespread nature makes me think "there's gotta be more than one failure here" e.g. can't just be a DNS issue. Follow up thought - "could this be a non-accident?" i.e. a deliberate outage on behalf of actors unknown?
These things are hard to fix and have been known to happen in big tech.
You can see a flood of BGP announcements the time it started: https://radar.cloudflare.com/as4804?dateRange=2d
ABC: Can you tell us why the system's gone down?
Optus CEO: Unfortunately, I don't have more information to give at this stage. We have had issues since 4am. The team has tried a number of parts of restoration and so far we have not had the results that we have hoped for. And we're pursuing every avenue to get everybody back online as soon as possible.
ABC: Do they know what happened though? Who's there the team that you've got working on it?
Optus CEO: Our team is still pursuing every possible avenue. We had a number of hypotheses and each one so far that we've tested and put in place new actions for has not resolved the fundamental issue. So we're still working on it. And when we have a identified root cause and a time for restoration, we'll be updating everybody as soon as we can.
---
Pretty wild to be down for over 7 hours no with no sign of a fix yet. Note they have also said they don't believe it's a 'cyber attack' (Optus had an enormous data breach last year with 9.7 million customers data leaked: https://en.wikipedia.org/wiki/2022_Optus_data_breach )
“Our on-site technician is actively prioritising establishing a console connection [a physical cable connection],” the message to Optus’ partners early on Wednesday said. “Rest assured that said technician is also being provided additional technical support remotely.”
[1] https://www.smh.com.au/technology/what-caused-the-optus-outa...