Globally, hundreds of thousand of organizations running Exchange email servers from Microsoft just got mass-hacked, including at least 30,000 victims in the United States. Each hacked server has been retrofitted with a “web shell” backdoor that gives the bad guys total, remote control, the ability to read all email, and easy access to the victim’s other computers. Researchers are now racing to identify, alert and help victims, and hopefully prevent further mayhem.

On Mar. 5, KrebsOnSecurity broke the news that at least 30,000 organizations and hundreds of thousands globally had been hacked. The same sources who shared those figures say the victim list has grown considerably since then, with many victims compromised by multiple cybercrime groups.

Security experts are now trying to alert and assist these victims before malicious hackers launch what many refer to with a mix of dread and anticipation as “Stage 2,” when the bad guys revisit all these hacked servers and seed them with ransomware or else additional hacking tools for crawling even deeper into victim networks.

But that rescue effort has been stymied by the sheer volume of attacks on these Exchange vulnerabilities, and by the number of apparently distinct hacking groups that are vying for control over vulnerable systems.

A security expert who has briefed federal and military advisors on the threat says many victims appear to have more than one type of backdoor installed. Some victims had three of these web shells installed. One was pelted with eight distinct backdoors. This initially caused a major overcount of potential victims, and required a great deal of de-duping various victim lists.

The source, who spoke on condition of anonymity, said many in the cybersecurity community recently saw a large spike in attacks on thousands of Exchange servers that was later linked to a profit-motivated cybercriminal group.

“What we thought was Stage 2 actually was one criminal group hijacking like 10,000 exchange servers,” said one source who’s briefed U.S. national security advisors on the outbreak.

On Mar. 2, when Microsoft released updates to plug the four Exchange flaws being attacked, it attributed the hacking activity to a previously unidentified Chinese cyber espionage group it called “Hafnium.” Microsoft said Hafnium had been using the Exchange flaws to conduct a series of low-and-slow attacks against specific strategic targets, such as non-governmental organization (NGOs) and think tanks.

But by Feb. 26, that relatively stealthy activity was morphing into the indiscriminate mass-exploitation of all vulnerable Exchange servers. That means even Exchange users that patched the same day Microsoft released security updates may have had servers seeded with backdoors.

Many experts who spoke to KrebsOnSecurity said they believe different cybercriminal groups somehow learned of Microsoft’s plans to ship fixes for the Exchange flaws a week earlier than they’d hoped (Microsoft originally targeted today, Patch Tuesday, as the release date).

The vulnerability scanning activity also ramped up markedly after Microsoft released its updates on Mar. 2. Security researchers love to tear apart patches for clues about the underlying security holes, and one major concern is that various cybercriminal groups may have already worked out how to exploit the flaws independently.

AVERTING MASS-RANSOMWARE

Security experts now are desperately trying to reach tens of thousands of victim organizations with a single message: Whether you have patched yet or have been hacked, backup any data stored on those servers immediately.

Every source I’ve spoken with about this incident says they fully expect profit-motivated cybercriminals to pounce on victims by mass-deploying ransomware. Given that so many groups now have backdoor web shells installed, it would be trivial to unleash ransomware on the lot of them in one go. Also, compromised Exchange servers can be a virtual doorway into the rest of the victim’s network.

“With the number of different threat actors dropping [web] shells on servers increasing, ransomware is inevitable,” said Allison Nixon, chief research officer at Unit221B, a New York City-based cyber investigations firm.

So far there are no signs of victims of this mass-hack being ransomed. But that may well change if the exploit code used to break into these vulnerable Exchange servers goes public. And nobody I’ve interviewed seems to think working exploit code is going to stay unpublished for much longer.

When that happens, the exploits will get folded into publicly available exploit testing kits, effectively making it simple for any attacker to find and compromise a decent number of victims who haven’t already patched.

CHECK MY OWA

Nixon is part of a group of security industry leaders who are contributing data and time to a new victim notification platform online called Check My OWA (Outlook Web Access, the Internet-facing Web component of Exchange Server machines).

Perhaps it’s better to call it a self-notification service that is operated from Unit221B’s own web site. It draws on tens of thousands of data points that various ISPs and hosting firms have tied to victims around the world who are likely compromised by the backdoor shells. The data comes from large networks watching the sources and targets of mass-scans for vulnerable Exchange servers.

“Our goal is to motivate people who we might otherwise have never been able to contact,” Nixon said. “My hope is if this site can get out there, then there’s a chance some victim companies are notified and take action or can get attention.”

Enter an email address at Check My OWA, and if that address matches a domain name for a victim organization, that email address will get a notice.

If the email’s domain name (anything to the right of the @ sign) is detected in their database, the site will send that user an email stating that is has observed the email domain in a list of targeted domains.

“Malicious actors were able to successfully compromise, and some of this information suggested they may have been able to install a webshell on an Exchange server associated with this domain,” reads one of the messages to victims. “We strongly recommend saving an offline backup of your Exchange server’s emails immediately, and refer back to the site for additional information on patching and remediation.”

Other Exchange users may see this message:

“We have observed your e-mail domain appears in our list of domains the malicious actors were able to successfully compromise, and some of this information suggested they may have been able to install a webshell on an Exchange server associated with this domain,” is another message the site may return. We strongly recommend saving an offline backup of your Exchange server’s emails immediately, and refer back to the site for additional information on patching and remediation.”

Nixon said Exchange users can save themselves a potentially nightmarish scenario if they just back up any affected systems now. And given the number of adversaries currently attacking still-unpatched Exchange systems, there is almost no way this won’t end in disaster for at least some victims.

“There are researchers running honeypots to [attract] attacks from different groups, and those honeypots are getting shelled left and right,” she said. “The sooner they can run a backup, the better. This can help save a lot of heartache.”

Oh, and one more important thing: You’ll want to keep any backups disconnected from everything. Ransomware has a tendency to infect everything it can, so make sure at least one backup is stored completely offline.

“Just disconnect them from a computer, put them in a safe place and pray you don’t need them,” Nixon said.

Sometimes when a complex story takes us by surprise or knocks us back on our heels, it pays to revisit the events in a somewhat linear fashion. Here’s a brief timeline of what we know leading up to last week’s mass-hack, when hundreds of thousands of Microsoft Exchange Server systems got compromised and seeded with a powerful backdoor Trojan horse program.

When did Microsoft find out about attacks on previously unknown vulnerabilities in Exchange?

Pressed for a date when it first became aware of the problem, Microsoft told KrebsOnSecurity it was initially notified “in early January.” So far the earliest known report came on Jan. 5, from a principal security researcher for security testing firm DEVCOR who goes by the handle “Orange Tsai.” DEVCOR is credited with reporting two of the four Exchange flaws that Microsoft patched on Mar. 2.

Reston, Va.-based Volexity first identified attacks on the flaws on Jan. 6, and officially informed Microsoft about it on Feb. 2. Volexity now says it can see attack traffic going back to Jan. 3. Microsoft credits Volexity with reporting the same two Exchange flaws as DEVCOR.

Danish security firm Dubex says it first saw clients hit on Jan. 18, and reported their incident response findings to Microsoft on Jan. 27.

In a blog post on their discovery, Please Leave an Exploit After the Beep, Dubex said the victims it investigated in January had a “web shell” backdoor installed via the “unifying messaging” module, a component of Exchange that allows an organization to store voicemail and faxes along with emails, calendars, and contacts in users’ mailboxes.

“A unified messaging server also allows users access to voicemail features via smartphones, Microsoft Outlook and Outlook Web App,” Dubex wrote. “Most users and IT departments manage their voicemail separately from their email, and voicemail and email exist as separate inboxes hosted on separate servers. Unified Messaging offers an integrated store for all messages and access to content through the computer and the telephone.”

Dubex says Microsoft “escalated” their issue on Feb. 8, but never confirmed the zero-day with Dubex prior to the emergency patch plea on Mar. 2. “We never got a ‘real’ confirmation of the zero-day before the patch was released,” said Dubex’s Chief Technology Officer Jacob Herbst.

How long have the vulnerabilities exploited here been around?

On Mar. 2, Microsoft patched four flaws in Exchange Server 2013 through 2019. Exchange Server 2010 is no longer supported, but the software giant made a “defense in depth” exception and gave Server 2010 users a freebie patch, too. That means the vulnerabilities the attackers exploited have been in the Microsoft Exchange Server code base for more than ten years.

The timeline also means Microsoft had almost two months to push out the patch it ultimately shipped Mar. 2, or else help hundreds of thousands of Exchange customers mitigate the threat from this flaw before attackers started exploiting it indiscriminately.

Here’s a rough timeline as we know it so far:

• Jan. 5: DEVCOR alerts Microsoft of its findings.
• Jan. 6: Volexity spots attacks that use unknown vulnerabilities in Exchange.
• Jan. 8: DEVCOR reports Microsoft had reproduced the problems and verified their findings.
• Jan. 11: DEVCOR snags proxylogon.com, a domain now used to explain its vulnerability discovery process.
• Jan. 27: Dubex alerts Microsoft about attacks on a new Exchange flaw.
• Jan. 29: Trend Micro publishes a blog post about “Chopper” web shells being dropped via Exchange flaws.
• Feb. 2: Volexity warns Microsoft about active attacks on previously unknown Exchange vulnerabilities.
• Feb. 8: Microsoft tells Dubex it has “escalated” its report internally.
• Feb. 18: Microsoft confirms with DEVCOR a target date of Mar. 9 (tomorrow) for publishing security updates for the Exchange flaws. That is the second Tuesday of the month — a.k.a. “Patch Tuesday,” when Microsoft releases monthly security updates (and yes that means check back here tomorrow for the always riveting Patch Tuesday roundup).
• Feb. 26-27: Targeted exploitation gradually turns into a global mass-scan; attackers start rapidly backdooring vulnerable servers.
• Mar. 2: A week earlier than previously planned, Microsoft releases updates to plug 4 zero-day flaws.
• Mar. 3: Tens of thousands of Exchange servers compromised worldwide, with thousands more servers getting freshly hacked each hour.
• Mar. 5: KrebsOnSecurity breaks the news that at least 30,000 organizations in the U.S. — and hundreds of thousands worldwide — now have backdoors installed.
• Mar. 5: Wired.com confirms the reported number of victims. White House makes statement expressing concern over the size of the attack. Former CISA head Chris Krebs tweets the numbers are bigger than what’s been reported.
• Mar. 6: CISA says it is aware of “widespread domestic and international exploitation of Microsoft Exchange Server flaws.”
• Mar. 7-Present: Security experts continue effort to notify victims, coordinate remediation, and remain vigilant for “Stage 2” of this attack (further exploitation of already-compromised servers).

At least 30,000 organizations across the United States — including a significant number of small businesses, towns, cities and local governments — have over the past few days been hacked by an unusually aggressive Chinese cyber espionage unit that’s focused on stealing email from victim organizations, multiple sources tell KrebsOnSecurity. The espionage group is exploiting four newly-discovered flaws in Microsoft Exchange Server email software, and has seeded hundreds of thousands of victim organizations worldwide with tools that give the attackers total, remote control over affected systems.

On March 2, Microsoft released emergency security updates to plug four security holes in Exchange Server versions 2013 through 2019 that hackers were actively using to siphon email communications from Internet-facing systems running Exchange.

In the three days since then, security experts say the same Chinese cyber espionage group has dramatically stepped up attacks on any vulnerable, unpatched Exchange servers worldwide.

In each incident, the intruders have left behind a “web shell,” an easy-to-use, password-protected hacking tool that can be accessed over the Internet from any browser. The web shell gives the attackers administrative access to the victim’s computer servers.

Speaking on condition of anonymity, two cybersecurity experts who’ve briefed U.S. national security advisors on the attack told KrebsOnSecurity the Chinese hacking group thought to be responsible has seized control over “hundreds of thousands” of Microsoft Exchange Servers worldwide — with each victim system representing approximately one organization that uses Exchange to process email.

Microsoft said the Exchange flaws are being targeted by a previously unidentified Chinese hacking crew it dubbed “Hafnium,” and said the group had been conducting targeted attacks on email systems used by a range of industry sectors, including infectious disease researchers, law firms, higher education institutions, defense contractors, policy think tanks, and NGOs.

Microsoft’s initial advisory about the Exchange flaws credited Reston, Va. based Volexity for reporting the vulnerabilities. Volexity President Steven Adair said the company first saw attackers quietly exploiting the Exchange bugs on Jan. 6, 2021, a day when most of the world was glued to television coverage of the riot at the U.S. Capitol.

But Adair said that over the past few days the hacking group has shifted into high gear, moving quickly to scan the Internet for Exchange servers that weren’t yet protected by those security updates.

“We’ve worked on dozens of cases so far where web shells were put on the victim system back on Feb. 28 [before Microsoft announced its patches], all the way up to today,” Adair said. “Even if you patched the same day Microsoft published its patches, there’s still a high chance there is a web shell on your server. The truth is, if you’re running Exchange and you haven’t patched this yet, there’s a very high chance that your organization is already compromised.”

Reached for comment, Microsoft said it is working closely with the U.S. Cybersecurity & Infrastructure Security Agency (CISA), other government agencies, and security companies, to ensure it is providing the best possible guidance and mitigation for its customers.

“The best protection is to apply updates as soon as possible across all impacted systems,” a Microsoft spokesperson said in a written statement. “We continue to help customers by providing additional investigation and mitigation guidance. Impacted customers should contact our support teams for additional help and resources.”

Adair said he’s fielded dozens of calls today from state and local government agencies that have identified the backdoors in their Exchange servers and are pleading for help. The trouble is, patching the flaws only blocks the four different ways the hackers are using to get in. But it does nothing to undo the damage that may already have been done.

By all accounts, rooting out these intruders is going to require an unprecedented and urgent nationwide clean-up effort. Adair and others say they’re worried that the longer it takes for victims to remove the backdoors, the more likely it is that the intruders will follow up by installing additional backdoors, and perhaps broadening the attack to include other portions of the victim’s network infrastructure.

Security researchers have published a tool on Microsoft’s Github code repository that lets anyone scan the Internet for Exchange servers that have been infected with the backdoor shell.

KrebsOnSecurity has seen portions of a victim list compiled by running this tool, and it is not a pretty picture. The backdoor web shell is verifiably present on the networks of thousands of U.S. organizations, including banks, credit unions, non-profits, telecommunications providers, public utilities and police, fire and rescue units.

“It’s police departments, hospitals, tons of city and state governments and credit unions,” said one source who’s working closely with federal officials on the matter. “Just about everyone who’s running self-hosted Outlook Web Access and wasn’t patched as of a few days ago got hit with a zero-day attack.”

Another government cybersecurity expert who participated in a recent call with multiple stakeholders impacted by this hacking spree worries the cleanup effort required is going to be Herculean.

“On the call, many questions were from school districts or local governments that all need help,” the source said, speaking on condition they were not identified by name. “If these numbers are in the tens of thousands, how does incident response get done? There are just not enough incident response teams out there to do that quickly.”

When it released patches for the four Exchange Server flaws on Tuesday, Microsoft emphasized that the vulnerability did not affect customers running its Exchange Online service (Microsoft’s cloud-hosted email for businesses). But sources say the vast majority of the organizations victimized so far are running some form of Internet-facing Microsoft Outlook Web Access (OWA) email systems in tandem with Exchange servers internally.

“It’s a question worth asking, what’s Microsoft’s recommendation going to be?,” the government cybersecurity expert said. “They’ll say ‘Patch, but it’s better to go to the cloud.’ But how are they securing their non-cloud products? Letting them wither on the vine.”

The government cybersecurity expert said this most recent round of attacks is uncharacteristic of the kinds of nation-state level hacking typically attributed to China, which tends to be fairly focused on compromising specific strategic targets.

“Its reckless,” the source said. “It seems out of character for Chinese state actors to be this indiscriminate.”

Microsoft has said the incursions by Hafnium on vulnerable Exchange servers are in no way connected to the separate SolarWinds-related attacks, in which a suspected Russian intelligence group installed backdoors in network management software used by more than 18,000 organizations.

“We continue to see no evidence that the actor behind SolarWinds discovered or exploited any vulnerability in Microsoft products and services,” the company said.

Nevertheless, the events of the past few days may well end up far eclipsing the damage done by the SolarWinds intruders.

This is a fast-moving story, and likely will be updated multiple times throughout the day. Stay tuned.

Over the past few weeks, three of the longest running and most venerated Russian-language online forums serving thousands of experienced cybercriminals have been hacked. In two of the intrusions, the attackers made off with the forums’ user databases, including email and Internet addresses and hashed passwords.

On Tuesday, someone dumped thousands of usernames, email addresses and obfuscated passwords on the dark web apparently pilfered from Mazafaka (a.k.a. “Maza,” “MFclub“), an exclusive crime forum that has for more than a decade played host to some of the most experienced and infamous Russian cyberthieves.

At the top of a 35-page PDF leaked online is a private encryption key allegedly used by Maza administrators. The database also includes ICQ numbers for many users. ICQ, also known as “I seek you,” was an instant message platform trusted by countless early denizens of these older crime forums before its use fell out of fashion in favor of more private networks, such as Jabber and Telegram.

This is notable because ICQ numbers tied to specific accounts often are a reliable data point that security researchers can use to connect multiple accounts to the same user across many forums and different nicknames over time.

Cyber intelligence firm Intel 471 assesses that the leaked Maza database is legitimate, and that the file includes more than 3,000 rows containing usernames, partially obfuscated password hashes, email addresses and other contact details.

“The file comprised more than 3,000 rows, containing usernames, partially obfuscated password hashes, email addresses and other contact details,” Intel 471 found, noting that Maza forum visitors are now redirected to a breach announcement page. “Initial analysis of the leaked data pointed to its probable authenticity, as at least a portion of the leaked user records correlated with our own data holdings.”

The attack on Maza comes just weeks after another major Russian crime forum got plundered. On Jan. 20, a longtime administrator of the Russian language forum Verified disclosed that the community’s domain registrar had been hacked, and that the site’s domain was redirected to an Internet server the attackers controlled.

“Our [bitcoin] wallet has been cracked. Luckily, we did not keep large amounts in it, but this is an unpleasant incident anyway. Once the circumstances became clear, the admin assumed that THEORETICALLY, all the forum’s accounts could have been compromised (the probability is low, but it is there). In our business, it’s better to play safe. So, we’ve decided to reset everyone’s codes. This is not a big deal. Simply write them down and use them from now on.”

A short time later, the administrator updated his post, saying:

“We are getting messages that the forum’s databases were filched after all when the forum was hacked. Everyone’s account passwords were forcibly reset. Pass this information to people you know. The forum was hacked through the domain registrar. The registrar was hacked first, then domain name servers were changed, and traffic was sniffed.”

On Feb. 15, the administrator posted a message purportedly sent on behalf of the intruders, who claimed they hacked Verified’s domain registrar between Jan. 16 and 20.

“It should be clear by now that the forum administration did not do an acceptable job with the security of this whole thing,” the attacker explained. “Most likely just out of laziness or incompetence, they gave up the whole thing. But the main surprise for us was that they saved all the user data, including cookies, referrers, ip addresses of the first registrations, login analytics, and everything else.”

The compromise of Maza and Verified — and possibly a third major forum — has many community members concerned that their real-life identities could be exposed. Exploit — perhaps the next-largest and most popular Russian forum after Verified, also experienced an apparent compromise this week.

According to Intel 471, on March 1, 2021, the administrator of the Exploit cybercrime forum claimed that a proxy server the forum used for protection from distributed denial-of-service (DDoS) attacks might have been compromised by an unknown party. The administrator stated that on Feb. 27, 2021, a monitoring system detected unauthorized secure shell access to the server and an attempt to dump network traffic.

Some forum lurkers have speculated that these recent compromises feel like the work of some government spy agency.

“Only intelligence services or people who know where the servers are located can pull off things like that,” mused one mainstay of Exploit. “Three forums in one month is just weird. I don’t think those were regular hackers. Someone is purposefully ruining forums.”

Others are wondering aloud which forum will fall next, and bemoaning the loss of trust among users that could be bad for business.

“Perhaps they work according to the following logic,” wrote one Exploit user. “There will be no forums, there will be no trust between everyone, less cooperation, more difficult to find partners – fewer attacks.”

Microsoft Corp. today released software updates to plug four security holes that attackers have been using to plunder email communications at companies that use its Exchange Server products. The company says all four flaws are being actively exploited as part of a complex attack chain deployed by a previously unidentified Chinese cyber espionage group.

The software giant typically releases security updates on the second Tuesday of each month, but it occasionally deviates from that schedule when addressing active attacks that target newly identified and serious vulnerabilities in its products.

The patches released today fix security problems in Microsoft Exchange Server 2013, 2016 and 2019. Microsoft said its Exchange Online service — basically hosted email for businesses — is not impacted by these flaws.

Microsoft credited researchers at Reston, Va. based Volexity for reporting the attacks. Volexity President Steven Adair told KrebsOnSecurity it first spotted the attacks on Jan. 6, 2021.

Adair said while the exploits used by the group may have taken great skills to develop, they require little technical know-how to use and can give an attacker easy access to all of an organization’s email if their vulnerable Exchange Servers are directly exposed to the Internet.

“These flaws are very easy to exploit,” Adair said. “You don’t need any special knowledge with these exploits. You just show up and say ‘I would like to break in and read all their email.’ That’s all there is to it.”

Microsoft says the flaws are being used by a previously unknown Chinese espionage group that’s been dubbed “Hafnium,” which is known to launch its attacks using hosting companies based in the United States.

“Hafnium primarily targets entities in the United States across a number of industry sectors, including infectious disease researchers, law firms, higher education institutions, defense contractors, policy think tanks, and NGOs,” Microsoft said. “HAFNIUM has previously compromised victims by exploiting vulnerabilities in internet-facing servers. Once they’ve gained access to a victim network, HAFNIUM typically exfiltrates data to file sharing sites like MEGA.”

According to Microsoft, Hafnium attackers have been observed combining all four zero-day flaws to target organizations running vulnerable Exchange Server products.

CVE-2021-26855 is a “server-side request forgery” (SSRF) flaw, in which a server (in this case, an on-premises Exchange Server) can be tricked into running commands that it should never have been permitted to run, such as authenticating as the Exchange server itself.

The attackers used CVE-2021-26857 to run code of their choice under the “system” account on a targeted Exchange server. The other two zero-day flaws — CVE-2021-26858 and CVE-2021-27065 — could allow an attacker to write a file to any part of the server.

After exploiting these vulnerabilities to gain initial access, Hafnium operators deployed web shells on the compromised server, Microsoft said. Web shells are essentially software backdoors that allow attackers to steal data and perform additional malicious actions that lead to further compromise.

Neither Microsoft nor Volexity is aware of publicly available code that would allow other cybercriminals to exploit these Exchange vulnerabilities. But given that these attacks are in the wild now, it may only be a matter of days before exploit code is publicly available online.

Microsoft stressed that the exploits detailed today were in no way connected to the separate SolarWinds-related attacks. “We continue to see no evidence that the actor behind SolarWinds discovered or exploited any vulnerability in Microsoft products and services,” the company said.

Further reading:

Microsoft’s writeup on new Hafnium nation state cyberattacks

Microsoft technical advisory on the four Exchange Server flaws