2024 CrowdStrike-related IT outages

2024 CrowdStrike-related IT outages

Multiple blue screens of death caused by a faulty software update on baggage carousels at LaGuardia Airport, New York City
Date	19 July 2024
Location	Worldwide
Type	IT outage, computer crash
Cause	Faulty CrowdStrike software update
Outcome	~8.5 million Microsoft Windows operating systems crash worldwide, causing global disruption of critical services

On 19 July 2024, American cybersecurity company CrowdStrike distributed a faulty update to its Falcon Sensor security software that caused widespread problems with Microsoft Windows computers running the software. As a result, roughly 8.5 million systems crashed and were unable to properly restart in what has been called the largest outage in the history of information technology and "historic in scale".

The outage disrupted daily life, businesses, and governments around the world. Many industries were affected—airlines, airports, banks, hotels, hospitals, manufacturing, stock markets, broadcasting, gas stations, retail stores, and more—as were governmental services, such as emergency services and websites. The worldwide financial damage has been estimated to be at least US$10 billion.

Within hours, the error was discovered and a fix was released, but because many affected computers had to be fixed manually, outages continued to linger on many services.

Background

CrowdStrike produces a suite of security software products for businesses, designed to protect computers from cyberattacks. Falcon, CrowdStrike's endpoint detection and response agent, works at the operating system kernel level on individual computers to detect and prevent threats. Patches are routinely distributed by CrowdStrike to its clients to enable their computers to address new threats.

CrowdStrike's own post-incident investigation identified several errors that led to the release of a fault update to the "Crowdstrike Sensor Detection Engine":

• The channel files were validated using Regex patterns with wildcards and loaded into an array instead of using a parser for this purpose.
• In the programming language C, the length of arrays must be treated and checked separately. However, the length was not checked before access. An array with 21 fields was expected, but the channel file was in an older data format with only 20 fields.
• In the unit tests, only the happy path was tested. Regression tests for compatibility with the older data format were not conducted.
• In manual tests, only valid data was tested.
• The channel files did not contain a version number field that was checked.
• There were no staggered rollouts, but the update was distributed to all customers simultaneously. Even critical infrastructure was not specially treated.
• The software does not access the system on Microsoft Windows through a suitable application programming interface but runs as a driver in ring 0 to have elevated privileges on the operating system. However, a crash in this area leads to a blue screen of death, which stops the operating system.

Outage

On 19 July at 04:09 UTC, CrowdStrike distributed a faulty configuration update for its Falcon sensor software running on Windows PCs and servers. A modification to a configuration file which was responsible for screening named pipes, Channel File 291, caused an out-of-bounds memory read in the Windows sensor client that resulted in an invalid page fault. The update caused machines to either enter into a bootloop or boot into recovery mode.

Almost immediately, Windows virtual machines on the Microsoft Azure cloud platform began rebooting and crashing, and at 06:48 UTC, Google Compute Engine also reported the problem. The problem affected systems running Windows 10 and Windows 11 running the CrowdStrike Falcon software. Most personal Windows PCs were unaffected, as CrowdStrike's software is primarily used by organizations. The CrowdStrike software did not provide a way for subscribers to delay the installation of its content files. Computers running macOS and Linux were unaffected, as the problematic content file was only for Windows, but similar problems had affected Linux distributions of CrowdStrike software in April 2024.

CrowdStrike reverted the content update at 05:27 UTC, and devices that booted after the revert were not affected.

At 07:15 UTC, Google said that the CrowdStrike update was at fault. Within hours, CrowdStrike CEO George Kurtz confirmed that CrowdStrike's faulty kernel configuration file update had caused the problem. At 09:45 UTC, Kurtz confirmed that the fix was deployed and that the problem was not the result of a cyberattack.

The impact to companies in the Central United States was exacerbated by an unrelated outage with Microsoft Azure the previous day. On 18 July, the Azure platform had an outage that blocked some companies' access to their storage and to Microsoft 365 applications in Azure's Central United States region.

Remedy

Affected machines could be restored by rebooting while connected to the network; ideally while connected to Ethernet, thus providing the opportunity to download the reverted channel file, with multiple reboots reportedly required.

If crashes persisted, remediation required booting into safe mode or the Windows Recovery Environment and deleting any .sys file beginning with C-00000291- and with timestamp 04:09 UTC in the %windir%\System32\drivers\CrowdStrike\ directory. As this process needed to be done locally on each individual machine, it was "expected to take days" for affected businesses to restore all systems. Technical staff needed to reboot the affected computers individually with manual intervention on each system.

On devices with Windows' BitLocker disk encryption enabled, which corporations often use to increase security, fixing the problem was exacerbated because the 48-digit numeric Bitlocker recovery keys (unique to each system) required manual input, with additional challenges supplying the recovery keys to end users working remotely. Additionally, several organisations utilising local servers for Bitlocker recovery key storage could not access keys that were stored on servers that themselves had crashed.

Microsoft has also recommended restoring a backup from before 18 July to fix the issue.

Impact

Outages were experienced worldwide, reflecting the wide use of Microsoft Windows and CrowdStrike software by global corporations in numerous business sectors. At the time of the incident, CrowdStrike said it had more than 24,000 customers, including nearly 60% of Fortune 500 companies and more than half of the Fortune 1000. On 20 July, Microsoft estimated that 8.5 million devices were affected by the update, which it said was less than one percent of all Windows devices.

Widespread outages were immediately reported across multiple countries, with major global disturbances experienced by the general public sweeping from east to west from time zone to time zone. At 04:09 UTC on 19 July, the time when the faulty update was issued, it was the middle of the business day of Oceania and Asia, the early morning hours in Europe, and midnight in much of the Americas.

Some countries were less affected. China, which has striven toward self-sufficiency in IT, saw little impact to key services such as airlines and banks, although foreign businesses and luxury hotels in the country were affected. Russia and Iran—both restricted by international sanctions from using the services of American high-tech companies—reported no disruptions.

A specialist cloud outage insurance business estimated that the top 500 US companies by revenue, excluding Microsoft, had faced near $5.4bn (£4.1bn) in financial losses because of the outage, but only between $540m (£418m) to $1.08bn (£840m) of those losses would be insured.

CrowdStrike liability

CrowdStrike's own terms and conditions for their Falcon software limit liability to "fees paid", effectively a refund. Larger customers may have negotiated different terms.

In the EU, it is possible that CrowdStrike will be held liable under a GDPR regulation related to the impact of security incidents on user data. The regulation is best known in relation to data leaks but also applies to data destruction. It is unclear whether temporary loss of access to data is enough to trigger liability, or whether GDPR applies to all incidents related to security or only unauthorised access.

Further, the incident could be classed as a "personal data breach" which would be a data breach of the GDPR under Article 4 named "Definitions", paragraph 12. On 19 July 2024, a data-protection expert reported a breach of Article 32 named "Security of processing".

Air transport

Digital signage at Dulles International Airport displaying a blue screen of death during the incident

Globally, 5,078 air flights, 4.6% of those scheduled that day, were cancelled. An unrelated Microsoft Azure outage, affecting services such as Microsoft 365, compounded airlines' problems.

Oceania

Australian airlines Qantas, Virgin Australia, and Jetstar were affected. A Sydney Airport spokesperson said that the outage had affected some operations and that "there may be some delays throughout the evening". Melbourne Airport saw check-in procedures disrupted; officials advised passengers to consult with their airlines. The Adelaide, Brisbane, Canberra, Darwin, Hobart, Launceston, and Perth airports were also affected. In New Zealand, Christchurch Airport was having problems.

Asia

AirAsia passengers queue at Ngurah Rai International Airport, Indonesia, after the airline's check-in system was disrupted.

Hong Kong International Airport experienced delays during check-in, primarily for passengers of the local budget carrier Hong Kong Express, whose staff members used handwritten signs to direct passengers to check-in counters. The Hong Kong Airport Authority activated an emergency response after airline websites and automatic check-in malfunctioned. The booking systems of local airlines Cathay Pacific, Hong Kong Express, and Hong Kong Airlines were unavailable. HKExpress cancelled some flights on 20 July. Jeju Air and Spring Japan experienced problems. Jetstar Japan cancelled many (mostly domestic) flights. Some of the self-check-in kiosks in Singapore Changi Airport were affected, delaying and forcing airlines to switch to manual check-in, and Singapore Airlines and Scoot reported various levels of service difficulties throughout 19 July. Cebu Pacific and Philippines AirAsia flights were delayed. Long queues formed at Ninoy Aquino International Airport. In Taiwan, airline system disruptions were reported at Taoyuan International Airport. In Indonesia, disruptions were reported for the check-in systems of AirAsia and Citilink. In Thailand, Thai AirAsia's reservation and check-in systems were affected.

In India, the outage affected Indigo Airlines, Akasa Air, SpiceJet, and Vistara. Handwritten boarding passes were being issued during the outage. The Ministry of Civil Aviation asked and ordered the airlines as well as the airports to be compassionate and provide food as well as seats to waiting customers as needed. As of 18:14 IST (12:44 UTC), over 200 Indian flights had been cancelled; IndiGo alone cancelled 192. Airlines that relied on Microsoft Azure for their services were affected. Air India and SpiceJet said that none of its flights were cancelled due to the outage, attributing it to their robust cyber system however, minute delays were reported.

Europe

Prague Airport in Czechia, Budapest Airport in Hungary, Bratislava Airport in Slovakia, and Schiphol Airport in the Netherlands experienced problems. Planes were not allowed to land at Zurich Airport. Near Brussels, Charleroi Airport employees manually checked passengers in, but other software alleviated problems by 10:00 (UTC+2) and there were minimal delays. ENAIRE's Aena, the Spanish national airport traffic control manager, mentioned an IT outage in their website and social media. All Spanish airports reported disruptions. Charles de Gaulle Airport and Orly Airport experienced check-in problems and suspended flights. Poznań–Ławica Airport and Warsaw Chopin Airport experienced check-in disruptions. An emergency system was activated, and check-in processes were slower. Berlin Brandenburg Airport announced that since around 07:00 (UTC+2), operational processes were affected by "IT problems at an external provider", and that they planned to stop flights until 08:00 UTC. While passenger handling continued with some restrictions, there were delays and airlines cancelled some flights. Several airlines (Eurowings, Ryanair, Vueling, and Turkish Airlines) in Hamburg Airport had to issue tickets by hand. Croatian and Swedish air traffic control were also disrupted.

Swiss International Air Lines had 30% of flights grounded. Lufthansa in Germany experienced problems with the "profile and booking retrieval" features of their website. Ryanair's booking and check-in services were unavailable and the airline was "forced to cancel a small number of flights", advising passengers to arrive at airports at least three hours before departure. Wizz Air said the outage put its online services offline. Dutch airline KLM suspended most operations, announcing that flight handling was impossible with the issue, and Transavia Airlines experienced problems. Finnair reported that they were having trouble sending emails and SMS messages to customers. In Greece, citizens and tourists saw delays at major airports, notably at Athens International Airport and at Heraklion International Airport. This disruption, occurring at the peak of the tourist season, resulted in chaotic scenes as passengers were forced to wait for hours for their flights. Contributing factors included severe staff shortages and new schedules. In Heraklion, eight flights were problematic. The airport's chief, George Pliakas, indicated that flights were being manually arranged to manage the disruption, but the influx of arriving flights strained the system.

Several UK airports had problems, including Edinburgh Airport, whose departure boards froze, and Gatwick Airport, where automatic barcode scanning stopped working and had to be checked manually. Amadeus, which manages UK baggage at Heathrow, said they were affected by the IT outage. Disruption to flights was anticipated in the Isle of Man, particularly to and from the UK, but ultimately minimal.

Middle East and North Africa

Tunisia experienced temporary airport disruptions. Turkish Airlines cancelled some of its flights due to the outage.

North America

See also: 2024 Delta Air Lines meltdown

A blue screen at the Boise Airport in Idaho during the incident.

In the mid-morning of Friday 19 July, a ground stop was issued by United, Delta, and American Airlines, halting takeoffs but allowing aircraft aloft to reach their destinations. Allegiant Air was also grounded by the outage. Around 10:30 a.m. Eastern time, AP reported that about 1,500 flights had already been cancelled in the United States due to the outage. American Airlines, United, and Allegiant recovered relatively quickly after Friday.

But Delta, by far the hardest hit of the US major airlines, experienced an operational meltdown that continued through the weekend. The airline cancelled more than 1,200 flights on Friday. Thousands of stranded travelers were forced to spend the night at Hartsfield–Jackson Atlanta International Airport, Delta's largest hub and the busiest airport in the world by passenger traffic. Metro Atlanta hotels and rental car companies were overwhelmed by the crisis, leaving travelers no option but to stay in the airport. One traveler attempting to return home to Tampa (after giving up on reaching California) reported that Amtrak was charging $1,000 for a one-way train ticket from Atlanta to Tampa. Visibly distraught passengers with nowhere to go were seen trying to sleep in the airport on hard linoleum floors without blankets or food. The airport's custodial staff were also overwhelmed, with restrooms and trash reportedly "out of control". Without warning, Delta banned unaccompanied minors on its flights through the end of 23 July. This imposed hardship on parents who had been counting on that service to enable their children to fly without the expense of an accompanying adult.

Delta cancelled more than 1,400 flights on 20 July, and more than 1,300 flights on 21 July. With so many passengers still stuck in Hartsfield–Jackson after two consecutive nights, the airport implemented a "concessions crisis plan" and a plan to reunite passengers with their checked baggage. However, passengers in Atlanta continued to report "jam-packed" conditions and "heartbreaking" scenes in the terminals.

On 21 July, Delta CEO Ed Bastian apologised to customers in a statement and revealed that the outage had left one of Delta's crew-tracking software programs "unable to effectively process the unprecedented number of changes triggered by the system shutdown". Delta CIO Rahul Samant said the program had been brought back online around 11 a.m. on 19 July, but was overwhelmed by the backlog of updates awaiting processing and had been trying to catch up ever since. After the ground stop left too many crew members in the wrong places, Delta struggled to assemble enough pilots and flight attendants at airport gates to operate scheduled flights. Many flights were repeatedly delayed and finally cancelled because the one or two crew members who made it to the gate for a particular flight kept hitting their legal flight time limit before the airline could finish fully staffing the flight, and this caused the crisis to snowball as those crew and their aircraft were now in the wrong place for the following day's flights. (A similar phenomenon occurred during the 2022 Southwest Airlines scheduling crisis.) That same day, US Secretary of Transportation Pete Buttigieg said on social media that the US Department of Transportation had received hundreds of complaints about Delta, and reminded the airline of its legal obligations to affected passengers.

On 22 July, Delta cancelled more than 1,200 flights. On 23 July, the Department of Transportation announced the launch of a formal investigation into Delta's treatment of passengers. Delta officials promised to cooperate but said the airline was focused on its recovery. Senator Maria Cantwell, in her capacity as chair of the Senate Committee on Commerce, Science, and Transportation, wrote to Bastian to express her concern for Delta passengers. On 23 July, Secretary Buttigieg estimated that over 500,000 passengers had been affected by Delta flight cancellations. He told a press conference, "There's a lot of things I'm very concerned about, including people being on hold for hours and hours, trying to get a new flight, people having to sleep on airport floors, even accounts of unaccompanied minors being stranded in airports, unable to get on a flight". He told CBS News: "Stories about people in lines of more than a hundred people with just one customer service agent serving them at an airport, that's completely unacceptable." By then, numerous passengers had ended up in different airports than their baggage because of Delta's flight cancellations, resulting in large piles of unclaimed suitcases and other checked baggage at Delta's airport terminals around the world.

On 25 July, Delta returned to normal flight operations, having cancelled more than 7,000 flights; passengers had filed more than 5,000 complaints about Delta with the Department of Transportation. On 26 July, The Washington Post reported that the department was investigating allegedly misleading communications from Delta that offered only credit towards future Delta flights as compensation for cancelled flights and failed to clearly notify passengers of their legal right to a cash refund.

On 31 July, Delta CEO Ed Bastian said the disruption had cost the airline $500 million, and he said that Delta would sue CrowdStrike to recoup some of its losses. On 8 August, Delta confirmed in a filing with the US Securities and Exchange Commission that over 7,000 flights had been cancelled over five days, and estimated its losses at $380 million in lost revenue and $170 million in expenses (adding up to about $550 million). Delta also estimated that around 1.3 million passengers had been affected by the flight cancellations.

United Airlines' smaller number of cancellations had a significant impact on its hubs. For example, San Mateo County hotels around San Francisco International Airport rapidly filled up with travelers on 19 July. Guests reported difficulty with checking into the local Marriott hotel because Marriott International was also recovering from the outage.

Southwest Airlines (the third largest US major airline by domestic passengers) was entirely unaffected. A Southwest spokesperson confirmed that the airline had seen no impact from the CrowdStrike outage but refused to confirm speculation among aviation industry analysts that it had been shielded by its notoriously outdated software.

The flight delays meant that many people who had traveled to the 2024 Republican National Convention—which concluded the day the outages started—were stuck in the convention's host city of Milwaukee, Wisconsin.

Montréal–Trudeau International Airport and Toronto Pearson International Airport were affected in Canada, and Porter Airlines cancelled all flights. Vancouver International Airport was also reportedly affected in Canada, although it was unclear whether this was directly related to the global outages.

Finance

A payment terminal affected by service disruption from the incident in India

Microsoft and CrowdStrike stocks fell as a result of the outage. CrowdStrike's stock fell more than 11% on 19 July, although Microsoft stock was down less than 1%.

Banks that were affected included Chase, Bank of America, Wells Fargo, U.S. Bank, Capital One and Charles Schwab in the US; RBC, Scotiabank, and TD Bank in Canada; Capitec Bank and other South African banks; several Israeli banks; and several banks in the Philippines, including RCBC, Metrobank, LandBank, BDO, UnionBank, BPI, and PNB. E-wallets such as Maya and GCash also experienced problems in the Philippines. The website and mobile banking application of DenizBank in Turkey could not be accessed. Visa was affected. Numerous Singaporean companies, including Singapore Exchange (SGX) and DBS Bank, reported various levels of service difficulties throughout 19 July.

In India, the Reserve Bank of India said that only 10 banks and NBFCs were affected by the outage; few banks use CrowdStrike tools and many banks' critical systems do not run on the cloud. NSE, BSE, and India's largest bank, State Bank of India, said they were unaffected.

In Brazil, Bradesco Bank confirmed it was affected. During the morning customers were able to login, but at 12:00 UTC the bank disabled the login button.

New Zealand banks ASB and Kiwibank were affected, while Australian banks Westpac and ANZ also had problems. Apps of Australian banks NAB, Westpac, ANZ, Commonwealth Bank, Bendigo Bank, and Suncorp were affected.

The London Stock Exchange, while operating normally, was unable to push news updates to its website. English gambling company Ladbrokes Coral and English supermarket chain Morrisons also reported problems. Polish banks, including Santander Bank Polska, ING Bank Śląski and mBank, encountered issues related to the outage. Santander BP's helpline, video, and chat services were affected. PKO Bank Polski clarified that its iPKO and IKO services were stable, but other banks faced difficulties. In Finland, OP Financial Group reported minor disruptions on investment partner and stock savings accounts. Sense Bank in Ukraine experienced outages due to the update.

Paraguayan banks Ueno and Banco Continental were affected; their customers were unable to log in.

Government

The United States Department of Homeland Security, NASA, Federal Trade Commission, National Nuclear Security Administration, Department of Justice, and Department of Education were affected, and the Department of the Treasury and Department of State reported minor disruptions. The Department of Veterans Affairs and Department of Energy experienced disruptions, but it is not currently known if they are related to the incident. DMV agencies for the states of Georgia, Kansas, Missouri, North Carolina, Tennessee, and the District of Columbia were affected. Ted Wheeler, the mayor of Portland, Oregon, declared the outages to be a city emergency. Election and voting registration databases in Arizona, South Dakota, Texas and the state of Washington were affected. The website for the city of Sioux Falls, South Dakota, went down.

In the United States, there were outages in 911 service or disruptions in 911 call centres' operation in some parts of Alaska, Arizona, Florida, Iowa, Indiana, Kansas, Michigan, Minnesota, New York, Ohio, Oregon, Pennsylvania, and Virginia. 911 was down for all of New Hampshire. In addition, Alaska was experiencing issues with non-emergency call centres. Many call centres switched to working backup systems.

The CM/ECF and PACER computer systems used by the US federal courts were unaffected. However, several state courts reported problems with their computer systems, including courts in Alaska, California, Delaware, Idaho, Kansas, Maryland, Massachusetts, Michigan, Nevada, New York, and Pennsylvania. In New York City, courts and correctional facilities were disrupted, delaying a hearing in the trial of Harvey Weinstein for sex offenses.

Government websites in the Philippines, such as the website of the House of Representatives of the Philippines, were down due to the outage.

In Canada, services in Toronto were affected, and Canada Child Benefit payments were delayed. New Zealand Parliament had problems. Sunshine Coast Council was one of several councils affected in Australia. The National Security Authority spokesman confirmed several institutions in Slovakia were affected.

The fire department in Copenhagen, Denmark, was unable to receive automatic fire alerts from buildings.

Ground transport

Traffic disruptions were reported at the US–Canada border, including long delays at the Ambassador Bridge and Detroit–Windsor tunnel between Ontario and Michigan. The Canada Border Services Agency blamed a partial outage of its telephone reporting system which was later resolved. There were long delays and police advised motorists to avoid the area. The Washington Metro Area Transit Authority suffered minor service delays in the early morning in America; their website/live tracking was unavailable until around 9:30 am on 19 July. The Massachusetts Bay Transportation Authority in Boston, as well as the Metropolitan Transportation Authority in New York, lost vehicle tracking and arrival notices for passengers. Most of North American freight and passenger train operators went largely unaffected aside from some technical issues within Union Pacific and Canadian Pacific Kansas City, Amtrak was mostly unaffected aside from issues with credit card processing during the morning.

Malaysia's railway operator, Keretapi Tanah Melayu, confirmed that its KITS ticketing system was experiencing technical issues. Transport for Ireland said its apps were down due to the outage. Ireland's Road Safety Authority said it was experiencing "significant disruption" to its National Car Test (NCT) centres. In Singapore, the entrance and exit gantries of over 185 car parks managed by the Housing and Development Board (HDB) were affected.

Fuel stations have also been affected in Australia, with people stuck at fuel pumps unable to pay for petrol because payment systems were not working. Auckland Transport's HOP card in New Zealand had problems. Australian freight train operator Aurizon was affected. Regional trains in New South Wales, Australia on the Hunter Line and the Southern Highlands Line were cancelled or delayed with the Regional Bus and Train network in Victoria operated by V/Line having all lines suspended.

UK rail companies were also affected. Cab riders in London could not pay with credit or with debit cards and thus required cash. In Sweden and Belgium, tickets for public transport could not be sold, and Keolis Nederland experienced issues.

Healthcare

Many hospitals across North America paused non-urgent surgeries and visits. Some affected hospitals, while remaining open, had limited, if any, access to patient records. In the United States, Memorial Sloan Kettering Cancer Center postponed all procedures that required anaesthesia, the Mass General Brigham hospital system cancelled all non-emergency procedures and medical visits, and the Cincinnati Children's Hospital Medical Center was also affected. University Health Network experienced technical issues in Canada, saying hospitals' clinical activity would continue but warning that appointments may be delayed. A number of other Canadian hospitals faced difficulties, with Newfoundland and Labrador Health Services activating contingency plans as patient record systems were affected. LabCorp and Quest Diagnostics were affected by the outage. Kaiser San Jose Medical Center lost access to patient records, as well as systems that monitored newborn baby's vital signs, and the security systems to keep babies from being taken. Other hospitals lost surveillance cameras and the ability for employee badges to unlock secure areas. England's National Health Service (NHS) said that the issues are "causing disruption in the majority of [English] GP practices", with some of its services, such as GP surgeries, which rely on a software product called EMIS Web, unable to view and manage medical records, issue and manage prescriptions, or make appointments. Manx Radio reported that GP surgeries were affected in the Isle of Man. The London Ambulance Service experienced an unprecedented surge in 999 and 111 calls following the outage, responding to 4,500 emergency calls by 17:00 (BST).

Two-thirds of Northern Ireland's general practices (GPs) were affected. At hospitals radiation therapy, bookings for operating theatres, and staff rosters are also affected.

In Belgium, FPS Public Health said the outage disrupted new-patient admissions in two hospitals, which activated their emergency IT plans. Two hospitals in Lübeck and Kiel, Germany, cancelled non-emergency operations. The Spanish regional governments of Aragon, Basque Country, Castilla-La Mancha, Catalonia, and Galicia reported problems with their healthcare services. Hospital Fernando Fonseca in Portugal reported problems, while the Catholic Health system in New York experienced outages that caused delays in services.

In the Netherlands, the outages affected two hospitals—the Scheperziekenhuis in Emmen and the Slingeland Ziekenhuis in de Achterhoek—and numerous emergency aid stations were also affected, including those in Emmen, Hoogeveen, and Stadskanaal.

Systems in Wesley Hospital and St Andrews Hospital in Brisbane, Australia, were affected.

The Central Health information system in Croatia was affected, although it was clarified that it was due to a concurrent issue tied with moving their servers to a new location.

In Israel, Magen David Adom and its emergency service hotline was affected. Hospitals including Sheba Medical Center, Rambam Hospital and Laniado Hospital were experiencing problems that led to longer waiting times and delayed surgeries.

The pharmaceutical company Krka in Slovenia suffered a full production outage and sent its workforce home.

Media and communications

Numerous American TV stations were unable to broadcast because of the global outage. KSHB-TV, one of the affected stations, had to resort to airing national news via Scripps News. ESPN was unable to air the morning editions of SportsCenter on the morning of the outage in America, instead airing ESPN Radio's Unsportsmanlike, simulcasting with ESPN2. ESPN and ESPN2 later simulcasted Get Up! and First Take in place of SportsCenter, albeit without on-air graphics or b-roll. Various Paramount channels were also affected including Nicktoons (with its West Coast feed switching to an old emergency feed), TeenNick, BET Her, NickMusic and most channels on the Pluto TV service. MeTV Toons was sent off the air for five and a half hours. Mercedes AMG PETRONAS F1 Team (which is sponsored by CrowdStrike) also suffered issues on the Friday of the Hungarian Grand Prix, with a Mercedes spokesperson confirming that the team had to manually address the problem on every computer it used. The issue also affected their engine customers, McLaren, Aston Martin and Williams. Many video screens in New York City's Times Square turned off.

When some companies let their employees go home early as a result of the incident, the topic "Thank you Microsoft for an early vacation" momentarily became Weibo's most popular term. Universal Studios Japan announced that they would not be selling tickets via ticket booths over the weekend due to the outage; however, tickets would still be sold online or via designated ticket sales sites.

Vodafone experienced outages. The issue affected the office laptops of DPG Media Belgium – which impacts JOE and QMusic Radio, banks, post services, and government agencies. Telephone communication with the urban services in Antwerp were also affected. The Centre for Cybersecurity Belgium stated that the impact in Belgium was limited. Sky News was unable to broadcast live in the UK, as was the BBC's CBBC, a free-to-air children's television channel. Irish national broadcaster RTÉ said its newsroom was hit by "intermittent internet outages" with minimal impact to output. The Canadian Broadcasting Corporation was also affected.

Several French TV channels affected by the issues include TF1, TFX, LCI and Canal+ Group networks. Phone and internet service provider Bouygues Telecom has also announced the unavailability of its customer service as a result of the outage. The operations of the 2024 Summer Olympics, scheduled to start the following week in Paris, France, were also affected. The outage occurred a day after the Olympic Village opened and organisers were processing the arrivals of athletes and delegates. The organising committee said that a contingency plan was activated and that only the delivery of uniforms and accreditations were affected. The incident slowed down the operations, with the accreditation desk at the press centre closed and security checks done manually using a list of names.

IT workers and the BPO industry were affected in the Philippines. Numerous Singaporean companies, including SPH Media, Singtel, and M1, reported various levels of service difficulties throughout the day on 19 July.

Australian media firms affected by the issues include the ABC, SBS, Seven Network and Nine Network. Ticketing at Docklands Stadium for Friday night's Australian Football League match between the Essendon Bombers and the Adelaide Crows was affected.

Israel Post was affected and Ukrainian Nova Poshta experienced outages. In the US, UPS and FedEx were affected.

Sim racing service iRacing was also affected by the outage in America. Various Korean online games, like Black Desert Online, Ragnarok Online, and Ragnarok Origin shut down.

Amazon Web Services, eBay, Google Cloud, Instagram, and Plenty of Fish were also affected.

Retail

The Vanden Borre retail website in Belgium experienced downtime due to the outage.

Notice in a Woolworths supermarket in New Zealand

Supermarkets in the Philippines were affected due to crashed POS systems. German supermarket chain Tegut closed some of its stores. Customers experienced payment problems at Foodstuffs and Woolworths supermarkets in New Zealand. Coles was also affected. British grocery chain Waitrose could only accept cash from customers. Self-checkout and online order systems at some Australian retailers and fast food chains were out of service.

The mobile application of the Starbucks coffee chain was limited to basic viewing of accounts made before the update; ordering was limited to in-store purchases only; some outlets saw cash register software crashes.

In the United States, sporting goods retailer Dick's Sporting Goods closed some of its stores and saw temporary outages to its website.

Convenience store chain 7-Eleven experienced problems at Speedway locations that still used BlueCube and Radiant Site Manager dating from the days Speedway was owned by Marathon Petroleum Corporation, with some stores unable to accept credit or debit transactions while others were closed outright.

In Norway, the pharmacy chain Apotek1 and the insurance company Tryg suspended services; the Vitusapotek and Boots pharmacy chains were also affected. Norway experienced little further impact because of CrowdStrike's limited market share in the country.

Amazon saw disruption to its warehouse operations and internal software. An app used in Amazon warehouses to manage schedules and time-off requests called 'A to Z' was taken down by the outage and an internal service called 'Anytime Pay' became unavailable to employees. Operations were briefly halted at some sites, while Amazon's trucking operations were disrupted, with drivers saying a platform they use called 'Relay' suffered issues meaning they were briefly unable to pick up loads at warehouses.

Other sectors

The outage affected terminal operations at DCT Gdańsk, a major container hub in the Baltic port of Gdańsk in Poland. Shipping ports in the US were unaffected for the most part, although the Port of Houston (which handles the most foreign tonnage) closed briefly.

In Sweden, the Malmberget mine was evacuated as a precaution. Tickets for soccer games could not be sold.

In the United States, security provider ADT was affected.

In Germany, Tesla halted production at its Gigafactory Berlin-Brandenburg for about four hours.

Response

In a live interview on NBC's Today, CrowdStrike CEO George Kurtz apologised to the public. He said company leaders were "deeply sorry for the impact that we've caused to customers, to travelers, to anyone affected by this, including our companies". CrowdStrike warned that malicious actors might try to pose as its staff or independent researchers claiming to help fix the problem.

CrowdStrike offered $10 UberEats vouchers to some employees at companies that sell and support its software as thanks for helping Crowdstrike customers recover, prompting ridicule given the costs associated with the outage. Uber flagged the code as suspicious as it was used so frequently, so it did not work for some users.

CrowdStrike won the 2024 Pwnie Awards for the Most Epic Fail, which CrowdStrike president Michael Sentonas accepted in person at DEF CON's annual Pwnie Awards show. A parody website named ClownStrike was created in the aftermath of the incident; CrowdStrike later sent a Digital Millennium Copyright Act (DMCA) takedown notice to the owner of the site.

Political

The Australian government held a national emergency meeting to address the outage. The National Coordination Mechanism was activated; Prime Minister Anthony Albanese said, "I understand Australians are concerned about the outage that is unfolding globally and affecting a wide range of services. My Government is working closely with the National Cyber Security Coordinator". He later said, "There is no impact to critical infrastructure, government services, or Triple-0 services at this stage". Victorians were advised to call Triple-0 if a fire alarm sounds or smoke is detected, as some automatic alarms in buildings may not automatically call fire services due to the outage.

United States president Joe Biden's administration was in touch with CrowdStrike to offer assistance, and on 23 July, Kurz was invited to appear before Congress to explain how the outage occurred and what CrowdStrike is doing to prevent future incidents.

The UK government's COBR committee met to discuss the incident.

India's Minister of Information and Technology Ashwini Vaishnav said that the government was in touch with Microsoft. The government's cybersecurity agency CERT-IN classified the incident as "critical".

In Russia, the government noted that the sanctions and boycotts placed on Russia as a result of its invasion of Ukraine in 2022 had unintentionally shielded it from the outage. Russia's Digital Communications Ministry said, "At the moment, the ministry has not received reports of system failures at Russian airports," and "The situation with Microsoft once again shows the importance of import substitution of foreign software, primarily at critical information infrastructure facilities." The Russian Federal Air Transport Agency confirmed that no domestic airlines were affected. The Kremlin stated that its systems were working as normal.

An editorial in the Chinese state-run Global Times suggested a need for decreased dependency on Western firms, stating that reliance on "top companies to lead network security efforts" can introduce security risks and noting the perceived irony of the United States leading global security efforts while major companies monopolise the industry.

Industry

Cybersecurity consultant Troy Hunt called the incident the "largest IT outage in history", adding: "This is basically what we were all worried about with Y2K, except it's actually happened this time". Slate described it as "Y2K Lite".

Elon Musk—CEO of Tesla, X Corp, Neuralink, and SpaceX—posted on X that CrowdStrike has been "deleted from all our systems".

AirAsia CEO Tony Fernandes demanded answers and compensation for millions of dollars in revenue he said the company had lost in the incident.

Chinese cybersecurity companies such as 360 Security, QAX and Tencent took advantage of the CrowdStrike incident to promote their own software.

News reporters have used the term "digital pandemic" to describe the outage.

Criminal

Governments worldwide and cybersecurity agencies warned of digital phishing scams after the incident. Cyber criminals started sending phishing emails purporting to be CrowdStrike support and impersonating CrowdStrike staff in phone calls shortly afterward.

Analysis

Cause

The 19 July update was an instance of a template that was tested and released in March 2024 as part of an update to the Falcon Sensor software. This new instance, Channel File 291, passed validation due to a bug in CrowdStrike's content verification software. The Falcon Sensor itself parses the file differently in a way that led to a software crash in kernel mode.

Centralisation and homogeneity

The outage raised questions about oligopoly and centralisation in the information technology sector. The majority of the world's computers use Microsoft Windows, creating a monoculture that reduces resiliency. Ciaran Martin, a cybersecurity expert, said, "This is a very, very uncomfortable illustration of the fragility of the world's core internet infrastructure". Critical infrastructure expert Gregory Falco said, "Cybersecurity providers are part of this homogenous backbone of modern systems and are so core to how we operate that a glitch in their operations will have similar impacts to failures in systems that are household names". Security experts suggested more redundancy to avoid single points of failure, wider use of decentralised and heterogeneous federated systems, and public anger at the failure of political leaders to regulate for diversity and competition. Conversely, cybersecurity expert Andrew Plato argued that monocultures are a net positive, “from a security perspective, there’s actually a lot of benefits to running a smaller, standardized set of software, because it allows you to spot a problem quicker and easier."

IT practices

Experts speculate that the update was not put through routine patch management procedures (testing the update in a sandbox) to verify there were no problems.

Mandating disclosure of breaches and vulnerabilities has also been suggested. In an interview with Wired, cybersecurity consultant Jake Williams said that this outage has "shown why pushing updates without IT intervention is unsustainable," and that "people may now demand changes in this operating model."

Operating system design and antitrust enforcement

Microsoft blamed a 2009 antitrust agreement with the European Union that they said forced them to sustain low-level kernel access to third-party developers. The document does not explicitly state that Microsoft has to provide kernel-level access, but says Microsoft must provide access to the same APIs used by its own security products. The EU rejected the allegations. The European Commission spokesperson told Euronews that "Microsoft is free to decide on its business model. It is for Microsoft to adapt its security infrastructure to respond to threats in line with EU competition law. Additionally, consumers are free to benefit from competition and choose between different cybersecurity providers."

The spokesperson also said that "the incident was not limited to the European Union and that Microsoft has never raised any concerns about security with the Commission either before or after the incident."

In Linux, it is possible to use eBPF instead of kernel modules to program this type of software.

Since macOS Catalina (2019), this type of software can use the Endpoint Security Framework instead of a kernel extension, and this approach has been gradually enforced.

See also

• Internet outage
  • 2000 outages: Y2K problem
  • 2010 outage: McAfee DAT 5958 update
  • Google services outages (several different years)
  • 2021 Facebook outage
  • 2022 Rogers Communications outage
  • 2023 Optus outage
  • Anticipated outages
    • Year 2038 problem
• General mitigations
  • IT risk management
  • Patch management
  • Security management