2020 United States federal government data breach

2020 United States federal government data breach

U.S. federal institutions reportedly breached. From top, clockwise: Defense, Energy, State, National Institutes of Health, Commerce, Homeland Security, Treasury, Agriculture
Date	Before October 2019 (start of supply chain compromise) March 2020 (possible federal breach start date) December 13, 2020 (breach acknowledged)
Duration	At least 8 months or 9 months
Location	United States, United Kingdom, Spain, Israel, United Arab Emirates, Canada, Mexico, others
Type	Cyberattack, data breach
Theme	Malware, backdoor, advanced persistent threat, espionage
Cause	Outlook Web App software bug Sunburst trojan VMware software bug Zerologon software bug
Target	U.S. federal government, possibly others
First reporter	FireEye (responsible disclosure) NSA (responsible disclosure) Reuters (public disclosure)
Suspects	Berserk Bear (Russia) Cozy Bear (Russia) FSB (Russia) SVR (Russia)

In 2020, a major cyberattack by a group backed by a foreign government penetrated multiple parts of United States federal government, leading to a data breach. The hacking group Cozy Bear (APT29), backed by the Russian intelligence agency SVR, was identified as the cyberattackers. The cyberattack and data breach were reported to be among the worst cyber-espionage ever suffered by the U.S., due to the sensitivity and high profile of the targets and the long duration (six to nine months) in which the hackers had access. Within days of its discovery, at least 200 organizations around the world had been reported to be affected by the attack, and some of these may also have suffered data breaches.

The attack, which had gone undetected for months, was first publicly reported on December 13, 2020, and was initially only known to have affected the U.S. Treasury Department and the National Telecommunications and Information Administration (NTIA), part of the U.S. Department of Commerce. In the following days, more departments and private organizations reported breaches.

The cyberattack that led to the federal breaches began no later than March 2020. The attackers exploited software from at least three U.S. firms: Microsoft, SolarWinds, and VMware. A supply chain attack on SolarWinds's Orion software, widely used in government and industry, provided the initial entry point. Flaws in Microsoft and VMWare products allowed the attackers to access emails and other documents, and to perform federated authentication across victim resources via single sign-on infrastructure.

In addition to the theft of data, the attack caused costly inconvenience to tens of thousands of SolarWinds customers, who had to check whether they had been breached, and had to take systems offline and begin months-long decontamination procedures as a precaution. U.S. Senator Richard J. Durbin described the cyberattack as tantamount to a declaration of war. President Donald Trump was silent for days after the attack, before spuriously suggesting that China, not Russia, might have been responsible for it, and that "everything is well under control".

Background

In June 2019, the New York Times reported that American hackers from the United States Cyber Command planted malware potentially capable of disrupting the Russian electrical grid. According to Wired senior writer Andy Greenberg, "The Kremlin warned that the intrusions could escalate into a cyberwar between the two countries."

The federal data breach occurred over the course of at least 8 or 9 months during the final year of the presidency of Donald Trump. Throughout this time the White House lacked a cybersecurity coordinator, Trump having eliminated the post itself in 2018. When the breach was discovered, the U.S. also lacked a Senate-confirmed Director of CISA, the nation's top cybersecurity official, responsible for coordinating incident response. (The incumbent, Chris Krebs, had been fired by Trump on November 18, 2020.) Also at that time, the DHS, which manages CISA, lacked a Senate-confirmed Secretary, Deputy Secretary, General Counsel, Undersecretary for Intelligence and Analysis, and Undersecretary for Management; and Trump had recently forced out the Deputy Director of CISA.

SolarWinds, a Texas-based provider of network monitoring software to the U.S. federal government, had shown several security shortcomings prior to the attack. SolarWinds did not employ a chief information security officer or senior director of cybersecurity. Cybercriminals had been selling access to SolarWinds's infrastructure since at least as early as 2017. SolarWinds had been advising customers to disable antivirus tools before installing SolarWinds software. In November 2019, a security researcher had warned SolarWinds that their FTP server was not secure, warning that "any hacker could upload malicious [files]" that would then be distributed to SolarWinds customers. And SolarWinds's Microsoft Office 365 account had been compromised, with the attackers able to access emails and possibly other documents.

On December 7, 2020, a few days before trojaned SolarWinds software was publicly confirmed to have been used to attack other organizations, longstanding SolarWinds CEO Kevin Thompson retired. That same day, two private equity firms with ties to SolarWinds's board sold substantial amounts of stock in SolarWinds. The firms denied insider trading.

Methodology

Multiple attack vectors were used. On December 17, 2020, additional attack vectors were reported to have been found.

SolarWinds exploit

This is classic espionage. It's done in a highly sophisticated way ... But this is a stealthy operation.

— Thomas Rid, The Washington Post

The attackers used a supply chain attack. The attackers accessed the build system belonging to the software company SolarWinds, possibly via SolarWinds's Microsoft Office 365 account, which had also been compromised at some point.

The attackers established a foothold in SolarWinds's software publishing infrastructure no later than October 2019. In the build system, the attackers surreptitiously modified software updates provided by SolarWinds to users of its network monitoring software Orion. The first known modification, in October 2019, was merely a proof of concept. Once the proof had been established, the attackers spent December 2019 to February 2020 setting up a command-and-control infrastructure.

In March 2020, the attackers began to plant remote access tool malware into Orion updates, thereby trojaning them. These users included U.S. government customers in the executive branch, the military, and the intelligence services (see Impact section, below). If a user installed the update, this would execute the malware payload, which would stay dormant for 12–14 days before attempting to communicate with one or more of several command-and-control servers. The communications were designed to mimic legitimate SolarWinds traffic. If able to contact one of those servers, this would alert the attackers of a successful malware deployment and offer the attackers a back door that the attackers could choose to utilise if they wished to exploit the system further. The malware started to contact command-and-control servers in April 2020, initially from North America and Europe and subsequently from other continents too.

The attackers appear to have utilized only a small fraction of the successful malware deployments: ones located within computer networks belonging to high-value targets. Once inside the target networks, the attackers pivoted, installing exploitation tools such as Cobalt strike components, and seeking additional access. Because Orion was connected to customers' Office 365 accounts as a trusted 3rd-party application, the attackers were able to access emails and other confidential documents. This access apparently helped them to hunt for certificates that would let them sign SAML tokens, allowing them to masquerade as legitimate users to additional on-premises services and to cloud services like Microsoft Azure Active Directory. Once these additional footholds had been obtained, disabling the compromised Orion software would no longer be sufficient to sever the attackers' access to the target network. Having accessed data of interest, they encrypted and exfiltrated it.

The attackers hosted their command-and-control servers on commercial cloud services from Amazon, Microsoft, GoDaddy and others. By using command-and-control IP addresses based in the U.S., and because much of the malware involved was new, the attackers were able to evade detection by Einstein, a national cybersecurity system operated by the Department of Homeland Security (DHS).

Microsoft exploits

If you think about data that is only available to the CEO, or data that is only available to IT services, [the attacker would get] all of this data.

— Sami Ruohonen, F-Secure

"Zerologon", a vulnerability in the Microsoft authentication protocol NetLogon, allowed attackers to access all valid usernames and passwords in each Microsoft network that they breached. This allowed them to access additional credentials necessary to assume the privileges of any legitimate user of the network, which in turn allowed them to compromise Microsoft Office 365 email accounts.

Additionally, a flaw in Microsoft's Outlook Web App may have allowed attackers to bypass multi-factor authentication.

Attackers were found to have broken into Microsoft Office 365 in a way that allowed them to monitor NTIA and Treasury staff emails for several months. This attack apparently used counterfeit identity tokens of some kind, allowing the attackers to trick Microsoft's authentication systems. The presence of single sign-on infrastructure increased the viability of the attack.

VMware exploits

Vulnerabilities in VMware Access and VMware Identity Manager, allowing existing network intruders to pivot and gain persistence, were utilized in 2020 by Russian state-sponsored attackers. As of December 18, 2020, while it was definitively known that the Sunburst trojan would have provided suitable access to exploit the VMware bugs, it was not yet definitively known whether attackers had in fact chained those two exploits in the wild.

Discovery

SolarWinds exploit

On December 8, 2020, the cybersecurity firm FireEye announced that red team tools had been stolen from it by what it believed to be a state-sponsored attacker. FireEye was believed to be a target of the SVR, Russia's Foreign Intelligence Service. FireEye says that it discovered the SolarWinds supply chain attack in the course of investigating FireEye's own breach and tool theft.

After discovering that attack, FireEye reported it to the U.S. National Security Agency (NSA), a federal agency responsible for helping to defend the U.S. from cyberattacks. The NSA is not known to have been aware of the attack before being notified by FireEye. The NSA uses SolarWinds software itself.

Some days later, on December 13, when breaches at the Treasury and Department of Commerce breaches were publicly confirmed to exist, sources said that the FireEye breach was related. On December 15, FireEye confirmed that the vector used to attack the Treasury and other government departments was the same one that had been used to attack FireEye: a trojaned software update for SolarWinds Orion.

The security community shifted its attention to Orion. The infected versions were found to be 2019.4 through 2020.2.1 HF1, released between March 2020 and June 2020. FireEye named the malware SUNBURST. Microsoft called it Solorigate.

Subsequent analysis of the SolarWinds compromise using DNS data and reverse engineering of Orion binaries, by DomainTools and ReversingLabs respectively, revealed additional details about the attacker's timeline.

Microsoft exploits

During 2019 and 2020, cybersecurity firm Volexity discovered an attacker making suspicious usage of Microsoft products within the network of a think tank whose identity has not publicly been revealed. The attacker exploited a vulnerability in the organization's Microsoft Exchange Control Panel, and used a novel method to bypass multi-factor authentication. Later, in June and July 2020, Volexity observed the attacker utilising the SolarWinds Orion trojan; i.e. the attacker used Microsoft vulnerabilities (initially) and SolarWinds supply chain attacks (later on) to achieve their goals. Volexity said it was not able to identify the attacker.

Separately, in or shortly before October 2020, Microsoft Threat Intelligence Center reported that an apparently state-sponsored attacker had been observed exploiting zerologon, a vulnerability in Microsoft's NetLogon protocol. This was reported to CISA, who issued an alert on October 22, 2020, specifically warning state, local, territorial and tribal governments to search for indicators of compromise, and instructing them to rebuild their networks from scratch if compromised. Using VirusTotal, The Intercept discovered continued indicators of compromise in December 2020, suggesting that the attacker might still be active in the network of the city government of Austin, Texas.

VMware exploits

Some time before December 3, 2020, the NSA discovered and notified VMware of vulnerabilities in VMware Access and VMware Identity Manager. VMware released patches on December 3, 2020. On December 7, 2020, the NSA published an advisory warning customers to apply the patches because the vulnerabilities were being actively exploited by Russian state-sponsored attackers.

Suspects

On October 22, 2020, CISA and the FBI identified the Microsoft zerologon attacker as Berserk Bear, a state-sponsored group believed to be part of Russia's FSB.

SolarWinds said it believed the malware insertion into Orion was performed by a foreign nation. Russian-sponsored hackers were suspected to be responsible. According to The Washington Post, the specific groups responsible were probably the SVR or Cozy Bear (aka APT29). FireEye gave the suspects the placeholder name "UNC2452"; incident response firm Volexity called them "Dark Halo". On December 23, 2020, the CEO of FireEye said Russia was the most likely culprit and the attacks were "very consistent" with the SVR.

Statements by U.S. government

On December 15, 2020, Democratic U.S. Senator Richard Blumenthal confirmed that the U.S. government believed Russia was responsible for the attack. The executive branch had yet to publicly confirm this.

On December 18, U.S. Secretary of State Mike Pompeo said Russia was "pretty clearly" responsible for the massive attack.

On December 19, Trump publicly addressed the attacks for the first time, suggesting without evidence that China, rather than Russia, might be responsible. The same day, Republican senator Marco Rubio, acting chair of the Senate Intelligence Committee, said it was "increasingly clear that Russian intelligence conducted the gravest cyber intrusion in our history."

On December 20, Democratic senator Mark Warner, briefed on the incident by intelligence officials, said "all indications point to Russia."

Russia reiterated its denial of involvement.

On December 21, 2020, Attorney General William Barr stated that he believes that the SolarWinds hack appears to have been perpetrated by Russia, contradicting statements from President Donald Trump.

Impact

This is a much bigger story than one single agency. This is a huge cyber espionage campaign targeting the U.S. government and its interests.

— U.S. government source

Discovery of the breaches at the Treasury and the Department of Commerce immediately raised concerns that the attackers would attempt to breach other departments, or had already done so. Further investigation proved these concerns to be well-founded. Within days, additional federal departments were found to have been breached.

SolarWinds said that of its 300,000 customers, 33,000 use Orion. Of these, around 18,000 government and private users downloaded compromised versions.

Compromised versions were known to have been downloaded by the Centers for Disease Control and Prevention, the Justice Department, and some utility companies. Other prominent U.S. organisations known to use SolarWinds products, though not necessarily Orion, were the Los Alamos National Laboratory, Boeing, and most Fortune 500 companies. Outside the U.S., reported SolarWinds clients included parts of the British government, including the Home Office, National Health Service, and signals intelligence agencies; the North Atlantic Treaty Organization (NATO); the European Parliament; and likely AstraZeneca. FireEye said that additional government, consulting, technology, telecom and extractive entities in North America, Europe, Asia and the Middle East may also have been affected.

Simply downloading a compromised version of Orion was not necessarily sufficient to result in a data breach; further investigation was required in each case to establish whether a breach resulted. These investigations were complicated by: the fact that the attackers had in some cases removed evidence; the need to maintain separate secure networks as organizations' main networks were assumed to be compromised; and the fact that Orion was itself a network monitoring tool, without which users had less visibility of their networks. As of mid-December 2020, those investigations were ongoing.

As of mid-December 2020, U.S. officials were still investigating what was stolen in the cases where breaches had occurred, and trying to determine how it could be used. Commentators said that the information stolen in the attack would increase the perpetrator's influence for years to come. Possible future uses could include attacks on hard targets like the CIA and NSA, or using blackmail to recruit spies. Cyberconflict professor Thomas Rid said the stolen data would have myriad uses. He added that the amount of data taken was likely to be many times greater than during Moonlight Maze, and if printed would form a stack far taller than the Washington Monument.

Even where data was not exfiltrated, the impact was significant. The Cybersecurity and Infrastructure Security Agency (CISA) advised that affected devices be rebuilt from trusted sources, and that all credentials exposed to SolarWinds software should be considered compromised and should therefore be reset. Anti-malware companies additionally advised searching log files for specific indicators of compromise.

However, it appeared that the attackers had deleted or altered records, and may have modified network or system settings in ways that could require manual review. Former Homeland Security Advisor Thomas P. Bossert warned that it could take years to evict the attackers from US networks, leaving them able to continue to monitor, destroy or tamper with data in the meantime. Harvard's Bruce Schneier, and NYU's Pano Yannakogeorgos, founding dean of the Air Force Cyber College, said that affected networks may need to be replaced completely.

List of confirmed connected data breaches

U.S. federal departments

Department	Affected part(s) include	Sources
Department of Agriculture
Department of Commerce	National Telecommunications and Information Administration
Department of Defense	Parts of The Pentagon
Department of Energy	National Nuclear Security Administration
Department of Health and Human Services	National Institutes of Health
Department of Homeland Security	Cybersecurity and Infrastructure Security Agency
Department of State
Department of the Treasury

U.S. state and local governments

Department	Affected part(s) include	Sources
Arizona	Pima County
Texas	Austin City

Private sector

Organization	Sources
Cisco Systems
Cox Communications
Equifax
FireEye
Microsoft
SolarWinds
A think tank (unnamed as of December 15, 2020)

Responses

Technology companies and business

On December 8, 2020, before other organizations were known to have been breached, FireEye published countermeasures against the red team tools that had been stolen from FireEye.

On December 15, 2020, Microsoft announced that Sunburst, which only affects Windows platforms, had been added to Microsoft's malware database and would, from December 16 onwards, be detected and quarantined by Microsoft Defender.

GoDaddy handed ownership to Microsoft of a command-and-control domain used in the attack, allowing Microsoft to activate a killswitch in the Sunburst malware, and to discover which SolarWinds customers were infected.

On December 14, 2020, the CEOs of several American utility companies convened to discuss the risks posed to the power grid by the attacks. On December 22, 2020, the North American Electric Reliability Corporation asked electricity companies to report their level of exposure to Solarwinds software.

SolarWinds unpublished its featured customer list after the hack, although as of December 15, the founder of the cybersecurity firm GreyNoise Intelligence said that SolarWinds had not removed the infected software updates from its distribution server.

U.S. government

On December 18, 2020, U.S. Secretary of State Mike Pompeo said that some details of the event would likely be classified so as not to become public.

Investigations and response by security agencies

On December 12, 2020, a National Security Council (NSC) meeting was held at the White House to discuss the breach of federal organizations. On December 13, 2020, CISA issued an emergency directive asking federal agencies to disable the SolarWinds software, to reduce the risk of additional intrusions, even though doing so would reduce those agencies' ability to monitor their computer networks. Russia denied involvement in the attacks.

On December 14, 2020, the Department of Commerce confirmed that it had asked the CISA and the FBI to investigate. The NSC activated Presidential Policy Directive 41, an Obama-era emergency plan, and convened its Cyber Response Group. The U.S. Cyber Command threatened swift retaliation against the attackers, pending the outcome of investigations.

The Federal Energy Regulatory Commission (FERC) helped to compensate for a staffing shortfall at CISA. The FBI, CISA, and the Office of the Director of National Intelligence (ODNI) formed a Cyber Unified Coordination Group (UCG) to coordinate their efforts.

On December 24, 2020, CISA said state and local government networks, in addition to federal ones, and other organizations, had been impacted by the attack, but did not provide further details.

Investigations and response by Congress

The Senate Armed Services Committee's cybersecurity subcommittee was briefed by Defense Department officials. The House Committee on Homeland Security and House Committee on Oversight and Reform announced an investigation. Marco Rubio, acting chair of the Senate Intelligence Committee, said the U.S. must retaliate, but only once the perpetrator is certain. The committee's vice-chairman, Mark Warner, criticized President Trump for failing to acknowledge or react to the hack.

Senator Ron Wyden called for mandatory security reviews of software used by federal agencies.

On December 22, 2020, after U.S. Treasury Secretary Steven Mnuchin reporters that he was "completely on top of this", the Senate Finance Committee was briefed by Microsoft that dozens of Treasury email accounts had been breached, and the attackers had accessed systems of the Treasury's Departmental Offices division, home to top Treasury officials. Senatory Wyden said that the briefing showed that the Treasury "still does not know all of the actions taken by hackers, or precisely what information was stolen".

On December 23, 2020, Senator Bob Menendez asked the State Department to end its silence about the extent of its breach, and Senator Richard Blumenthal asked the same of the Veterans Administration.

Trump comments and response to Trump

President Donald Trump made no comment on the hack for days after it was reported, leading Senator Mitt Romney to decry his "silence and inaction". On December 19, Trump publicly addressed the attacks for the first time; he downplayed the hack, contended that the media had overblown the severity of the incident, said that "everything is well under control"; and proposed, without evidence, that China, rather than Russia, might be responsible for the attack. Trump then pivoted to insisting that he had won the 2020 presidential election. He speculated, without evidence, that the attack might also have involved a "hit" on voting machines, part of a long-running campaign by Trump to falsely assert that he won the 2020 election. Trump's claim was rebutted by former CISA director Chris Krebs, who pointed out that Trump's claim was not possible. Adam Schiff, chair of the House Intelligence Committee, described Trump's statements as dishonest, calling the comment a "scandalous betrayal of our national security" that "sounds like it could have been written in the Kremlin."

Former Homeland Security Advisor Thomas P. Bossert said, "President Trump is on the verge of leaving behind a federal government, and perhaps a large number of major industries, compromised by the Russian government," and noted that congressional action, including via the National Defense Authorization Act would be required to mitigate the damage caused by the attacks. Russell Brandom, policy editor for The Verge, called the U.S. ill-prepared for the hack, and criticized Trump for having consistently "treated the federal cybersecurity effort as one more partisan battleground, with attacks and vulnerabilities embraced or rejected largely on the basis of their value as a political cudgel"; Brandom wrote that "this is no way to run the world’s most powerful intelligence apparatus." Fred Kaplan, writing in Slate, criticized Trump for promoting fake claims of election fraud while "ignoring a real cybersecurity crisis," writing: "For all of Trump's wailing about fictitious hacks that stole the election, he has been otherwise notably uncurious about the nation's cybersecurity." Esquire commentator Charles P. Pierce criticized the Trump administration for being "asleep at the switch" and termed Trump a "crooked, incompetent agent of chaos."

President-elect Biden

President-elect Joe Biden said that, "A good defense isn't enough; we need to disrupt and deter our adversaries from undertaking significant cyberattacks in the first place. I will not stand idly by in the face of cyberassaults on our nation." Biden said he has instructed his transition team to study the breach, will make cybersecurity a priority at every level of government, and will identify and penalize the attackers. Biden's incoming chief of staff, Ron Klain, said the Biden administration's response to the hack would extend beyond sanctions.

On December 22, 2020, Biden said that, "I see no evidence that it's under control," and reported that his transition team was still being denied access to some briefings about the attack by Trump administration officials.

Rest of world

NATO said that it was "currently assessing the situation, with a view to identifying and mitigating any potential risks to our networks." On December 18, the United Kingdom National Cyber Security Centre said that it was still establishing the attacks' impact on the UK. The UK and Irish cybersecurity agencies published alerts targeting SolarWinds customers.

On December 23, 2020, the UK Information Commissioner's Office - a national privacy authority - told UK organizations to check immediately whether they were impacted.

On December 24, 2020, the Canadian Centre for Cyber Security asked SolarWinds Orion users in Canada to check for system compromises.

Classification of the hack

The attack prompted a debate on whether the hack should be treated as cyber-espionage, or as a cyberattack constituting an act of war. Most current and former U.S. officials considered the 2020 Russian hack to be a "stunning and distressing feat of espionage" but not a cyberattack because the Russians did not appear to destroy or manipulate data or cause physical damage (for example, to the electrical grid). Erica Borghard of the Atlantic Council and Columbia's Saltzman Institute and Jacquelyn Schneider of the Hoover Institution and Naval War College argued that the breach was an act of espionage that could be responded to with "arrests, diplomacy, or counterintelligence" and had not yet been shown to be a cyberattack, a classification that would legally allow the U.S. to respond with force. Law professor Jack Goldsmith wrote that the hack was a damaging act of cyber-espionage but "does not violate international law or norms" and wrote that "because of its own practices, the U.S. government has traditionally accepted the legitimacy of foreign governmental electronic spying in U.S. government networks." Law professor Michael Schmitt concurred, citing the Tallinn Manual.

By contrast, Microsoft president Brad Smith termed the hack a cyberattack, stating that it was "not 'espionage as usual,' even in the digital age" because it was "not just an attack on specific targets, but on the trust and reliability of the world's critical infrastructure." U.S. Senator Richard J. Durbin (D-IL) described the attack as tantamount to a declaration of war.

Debate on possible U.S. responses

Writing for Wired, Borghard and Schneider opined that the U.S. "should continue to build and rely on strategic deterrence to convince states not to weaponize the cyber intelligence they collect". They also stated that because deterrence may not effectively discourage cyber-espionage attempts by threat actors, the U.S. should also focus on making cyber-espionage less successful through methods such as enhanced cyber-defenses, better information-sharing, and "defending forward" (reducing Russian and Chinese offensive cyber-capabilities).

Writing for The Dispatch, Goldsmith wrote that the failure of defense and deterrence strategies against cyber-intrusion should prompt consideration of a "mutual restraint" strategy, "whereby the United States agrees to curb certain activities in foreign networks in exchange for forbearance by our adversaries in our networks."

Bruce Schneier advocated against retaliation or increases in offensive capabilities, proposing instead the adoption of a defense-dominant strategy and ratification of the Paris Call for Trust and Security in Cyberspace or the Global Commission on the Stability of Cyberspace.

In the New York Times, Paul Kolbe, former CIA agent and director of the Intelligence Project at Harvard's Belfer Center for Science and International Affairs, echoed Schneier's call for improvements in the U.S.'s cyberdefenses and international agreements.

In Slate, Fred Kaplan argued that the structural problems that enable computer network intrusions like this had been public knowledge since 1967 and that successive U.S. governments had failed to implement the structural defenses repeatedly requested by subject experts. He pointed out that an escalatory response to espionage would be counterproductive for U.S. interests, whereas finally strengthening the defenses and drawing clear red lines in the gray areas of cyber-conflict policy would be more fruitful strategies.

See also

• Cyberwarfare in the United States
• Cyberwarfare by Russia
• EternalBlue
• Global surveillance disclosures (2013–present)
• List of data breaches
• Moonlight Maze
• Office of Personnel Management data breach
• Security dilemma
• The Shadow Brokers