Jump to content

2023 Optus outage

Add links
From Wikipedia, the free encyclopedia

2023 Optus outage
Date8 November 2023 (2023-11-08)
Time04:05 (AEDT)
Duration04:05 – 16:00 (approx. 12–13 hours)
LocationAustralia (nationwide)
TypeNetwork outage
CauseAutomated shutdown due to BGP event.[1]
Outcome
  • Disruption across government and private sectors
  • Landline-based Emergency 000 calls failing
  • A$2 billion stock loss to Optus
  • Investigations including Senate inquiry into the outage
InquiriesSenate inquiry

The 2023 Optus outage was an unplanned outage of all Optus internet, cellular and fixed-line services in Australia first detected on 8 November 2023 at 04:05 AEDT.[2] The outage caused disruption across government and corporate sectors including hospitals, banks, train services, EFTPOS payment systems, and calls to emergency services.[2][3] Restoration was gradual with some services being restored around 13:00 AEDT.[4] The outage directly affected more than 10 million people and 400,000 businesses across Australia.[5][6]

The cause was likely due to several routers disconnecting as an automated protective measure against routing update overload.[1] Optus stated that the outage was not due to a cyberattack.[2][7]

Background

[edit]

Optus is an Australian telecommunications company owned by Singtel. It is the second-largest wireless carrier in Australia, serving over 10 million customers in part with a 4G network that is reachable by 96.9% of the population.[8] Around 25% of Australians use Optus for their cellular service and 19% for broadband internet.[9]

Summary

[edit]

Outage

[edit]

The outage begun early morning on Wednesday, 8 November at around 04:05 AEDT following a planned software upgrade to a Singtel North American exchange.[1][10] The customer network begun disconnecting first at around 04:05 and some domain name services stopped working.

Restoration

[edit]

Restoration was a large-scale effort across more than 100 devices at 14 sites nationwide and took several hours.[11][12][1] The recovery was performed remotely and in some cases required physical access to sites. Services were restored gradually with 11% restored by 11:30am AEDT and 88% by 13:00 AEDT.[1] By 18:00 AEDT, Optus said that most of their services had been restored though connectivity was intermittent until 9 November 2023.[4][1]

Impact

[edit]

The outage impacted all internet, cellular and fixed-line services using the Optus network in Australia.[13] Public and private sectors, such as banks, hospitals, universities, and some train services, were impacted. Some people were unable to call emergency services.[14] Many businesses were not able to take digital payments via EFTPOS.[7] Dodo and Amaysim, who both rely on the Optus network as resellers, experienced outages across the country.[14] Aussie Broadband and Moose Mobile customers were also affected.[15]

A citywide Metro Trains Melbourne shutdown lasted from 05:00 to 06:00 due to drivers not being able to communicate with the control centre, causing hundreds of trains to be cancelled throughout the day on the Melbourne suburban network.[16]

Singtel lost over AU$2 billion in value over the day of the outage after its share price dropped 4.8% on the SGX.[4]

Response

[edit]

On 9 November 2023, Optus CEO Kelly Bayer Rosmarin announced that customers on post-paid mobile plans would be able to access 200 GB of extra mobile data, stating that mobile data was "much more valuable" than refunding customers. The 200 GB of data must be activated between 13 November 2023 and the end of the calendar year. An Optus spokesperson later stated that eligible prepaid mobile users are being given unlimited data on weekends for the remainder of the year, beginning on 18 November. The spokesperson noted that if any businesses were "uniquely impacted" during the outage, they should contact Optus' Business Centre directly.[17]

Despite this, user sentiment was overwhelmingly negative, with many businesses being forced to shut down for the day and incur massive losses.[18][19][20] In addition to Optus suffering an A$2 billion stock loss (more than 5%), rival telecom companies such as Telstra and Vodafone saw an increase in customers due to the mass exodus, with some industry analysts, such as JPMorgan Chase, predicting that Telstra alone could increase their earnings projections by A$125 million over the next few years if even just 2.5% of customers sever their ties with Optus.[21][22] On 20 November 2023, Bayer Rosmarin resigned as CEO.[23]

Reactions

[edit]

The outage raised concerns about the move to a cashless society and Australia's over-reliance on internet access.[24]

South Australia's government is reconsidering using Optus as its service provider.[12]

The federal government has called for financial compensation to those affected by the outage.[25] Communications Minister Michelle Rowland announced a review into the matter.[26] The Australian Senate also passed a motion to hold a public Senate inquiry into the outage.[27]

Causes

[edit]

A Border Gateway Protocol (BGP) routing problem played a role in the outage.[28][29][30] Public data from CloudFlare showed a spike in BGP route announcements from the Optus network around the time the outage occurred — over 940,000 announcements in an hour from a node that normally makes less than 3,000 announcements per hour — indicative of a BGP routing problem.[29][31]

A submission tabled to the Australian Senate Standing Committee on Environment and Communications committee describes the outage as a gradual event triggered by loss of connectivity between neighbouring computer networks.[1] The report suggests that approximately 90 edge provider routers disconnected as an automated protective measure against routing update overload. The failures occurred following a software upgrade at a North American Singtel exchange that caused one of the routers to disconnect. This, in turn, triggered Optus's routers to rapidly update its own routing tables which triggered the shutdown due to the pre-configured default threshold limits set by Cisco Systems being exceeded. The tabled report and Singtel stressed that the software upgrade was not the cause of the fault.[1][10]

See also

[edit]

References

[edit]
  1. ^ a b c d e f g h Optus Network Outage, Canberra: Senate Standing Committee on Environment and Communications (Parliament of Australia) (published November 2023), 17 November 2023, archived from the original on 17 November 2023, retrieved 18 November 2023
  2. ^ a b c "Optus outage hits millions: Telco says services fully restored after nearly 14 hours". www.9news.com.au. 8 November 2023. Archived from the original on 9 November 2023. Retrieved 9 November 2023.
  3. ^ "Optus outage affects millions of customers across Australia". amp.smh.com.au. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  4. ^ a b c "Optus says outage fixed after nearly 14 hours". www.9news.com.au. 8 November 2023. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  5. ^ Jose, Renju; Needham, Kirsty; Kaye, Ron (8 November 2023). "Optus network outage affects millions of Australians". Reuters. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  6. ^ "Optus likely to explain outage 'disaster' at senate inquiry | Daily Telegraph". Archived from the original on 9 November 2023. Retrieved 11 November 2023.
  7. ^ a b Seeto, Tamika (8 November 2023). "Optus outage: Services 'gradually' resuming for millions of Aussies but 'may take hours'". Yahoo Finance. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  8. ^ Reichert, Corinne (16 May 2018). "Optus doubles NBN customers as mobile subs hit 10 million". ZDNET. Archived from the original on 24 October 2022. Retrieved 8 November 2023.
  9. ^ Grant-Geary, Belinda (8 November 2023). "Optus outage: Is outage 'final straw' for frustrated Aussies considering 'mass exodus'". yahoo! finance. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  10. ^ a b "Singtel refutes reports that its system upgrade caused Optus outage". ZDNET. Retrieved 18 November 2023.
  11. ^ Maguire, Dannielle; Dervisevic, Hanan (7 November 2023). "Optus outage live: Company announces some services are being restored, but warns it may take a few hours". ABC News. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  12. ^ a b Wind, Emily; Cait, Kelly (8 November 2023). "Australia news live: Optus network outage; CEO says 'path to restoration' found after nine-hour blackout; SA government considers dumping Optus". The Guardian. ISSN 0261-3077. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  13. ^ Varghese, Sam (8 November 2023). "Optus hit by national mobile, broadband outage since 4am". iTWire. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  14. ^ a b Smith, Paul; Wiggins, Jenny (7 November 2023). "Optus outage cuts off millions, emergency calls". Australian Financial Review. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  15. ^ Turnbull, Tiffanie (8 November 2023). "Optus outage: Millions affected by Australian network failure". BBC. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  16. ^ "About 500 train services cancelled across Melbourne after outage". www.9news.com.au. 8 November 2023. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  17. ^ "'They're not listening to us': Business owner outraged by Optus data offer after outage". ABC News. 9 November 2023. Archived from the original on 11 November 2023. Retrieved 9 November 2023.
  18. ^ "'There's no value to data': Optus customers outraged over 200GB compensation". www.9news.com.au. 10 November 2023. Archived from the original on 11 November 2023. Retrieved 10 November 2023.
  19. ^ Businesses outraged by Optus data offer after outage | ABC News, archived from the original on 10 November 2023, retrieved 10 November 2023
  20. ^ Optus customers slam compensation offer | 9 News Australia, archived from the original on 11 November 2023, retrieved 10 November 2023
  21. ^ Lawler, Mitchell (10 November 2023). "What does the Optus fumble mean for ASX telco shares?". The Motley Fool Australia. Archived from the original on 10 November 2023. Retrieved 10 November 2023.
  22. ^ Businesses outraged by Optus data offer after outage | ABC News, archived from the original on 10 November 2023, retrieved 10 November 2023
  23. ^ Kelly Bayer Rosmarin resigns in the best intetest of Optus after natioanwide outage ABC News 20 November 2023
  24. ^ Seeto, Tamika (8 November 2023). "Cashless Australia: Optus outage 'proof' Australia could be 'doomed'". yahoo! finance. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  25. ^ Jeffrey, Daniel (9 November 2023). "Can you get compensation for the massive Optus outage?". 9 News. Archived from the original on 9 November 2023. Retrieved 9 November 2023.
  26. ^ Vidler, Adam (9 November 2023). "Optus outage: Government pledges review into network collapse". 9 News. Archived from the original on 8 November 2023. Retrieved 9 November 2023.
  27. ^ "Optus CEO to face grilling at Senate inquiry". ABC listen. 16 November 2023. Retrieved 18 November 2023.
  28. ^ Swan, David; Grubb, Ben (8 November 2023). "What caused the Optus outage?". The Sydney Morning Herald. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  29. ^ a b Davidson, John (8 November 2023). "What early signs suggest may have gone wrong at Optus". Australian Financial Review. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  30. ^ Taylor, Josh (8 November 2023). "Optus network outage may have been caused by same issue that brought Facebook down in 2021". The Guardian. ISSN 0261-3077. Archived from the original on 11 November 2023. Retrieved 9 November 2023.
  31. ^ "Overview for AS4804". CloudFlare. Retrieved 8 November 2023.
Retrieved from "https://en.wikipedia.org/w/index.php?title=2023_Optus_outage&oldid=1231110449"