Sometimes when a complex story takes us by surprise or knocks us back on our heels, it pays to revisit the events in a somewhat linear fashion. Here’s a brief timeline of what we know leading up to last week’s mass-hack, when hundreds of thousands of Microsoft Exchange Server systems got compromised and seeded with a powerful backdoor Trojan horse program.

When did Microsoft find out about attacks on previously unknown vulnerabilities in Exchange?

Pressed for a date when it first became aware of the problem, Microsoft told KrebsOnSecurity it was initially notified “in early January.” So far the earliest known report came on Jan. 5, from a principal security researcher for security testing firm DEVCOR who goes by the handle “Orange Tsai.” DEVCOR is credited with reporting two of the four Exchange flaws that Microsoft patched on Mar. 2.

Reston, Va.-based Volexity first identified attacks on the flaws on Jan. 6, and officially informed Microsoft about it on Feb. 2. Volexity now says it can see attack traffic going back to Jan. 3. Microsoft credits Volexity with reporting the same two Exchange flaws as DEVCOR.

Danish security firm Dubex says it first saw clients hit on Jan. 18, and reported their incident response findings to Microsoft on Jan. 27.

In a blog post on their discovery, Please Leave an Exploit After the Beep, Dubex said the victims it investigated in January had a “web shell” backdoor installed via the “unifying messaging” module, a component of Exchange that allows an organization to store voicemail and faxes along with emails, calendars, and contacts in users’ mailboxes.

“A unified messaging server also allows users access to voicemail features via smartphones, Microsoft Outlook and Outlook Web App,” Dubex wrote. “Most users and IT departments manage their voicemail separately from their email, and voicemail and email exist as separate inboxes hosted on separate servers. Unified Messaging offers an integrated store for all messages and access to content through the computer and the telephone.”

Dubex says Microsoft “escalated” their issue on Feb. 8, but never confirmed the zero-day with Dubex prior to the emergency patch plea on Mar. 2. “We never got a ‘real’ confirmation of the zero-day before the patch was released,” said Dubex’s Chief Technology Officer Jacob Herbst.

How long have the vulnerabilities exploited here been around?

On Mar. 2, Microsoft patched four flaws in Exchange Server 2013 through 2019. Exchange Server 2010 is no longer supported, but the software giant made a “defense in depth” exception and gave Server 2010 users a freebie patch, too. That means the vulnerabilities the attackers exploited have been in the Microsoft Exchange Server code base for more than ten years.

The timeline also means Microsoft had almost two months to push out the patch it ultimately shipped Mar. 2, or else help hundreds of thousands of Exchange customers mitigate the threat from this flaw before attackers started exploiting it indiscriminately.

Here’s a rough timeline as we know it so far:

• Jan. 5: DEVCOR alerts Microsoft of its findings.
• Jan. 6: Volexity spots attacks that use unknown vulnerabilities in Exchange.
• Jan. 8: DEVCOR reports Microsoft had reproduced the problems and verified their findings.
• Jan. 11: DEVCOR snags proxylogon.com, a domain now used to explain its vulnerability discovery process.
• Jan. 27: Dubex alerts Microsoft about attacks on a new Exchange flaw.
• Jan. 29: Trend Micro publishes a blog post about “Chopper” web shells being dropped via Exchange flaws.
• Feb. 2: Volexity warns Microsoft about active attacks on previously unknown Exchange vulnerabilities.
• Feb. 8: Microsoft tells Dubex it has “escalated” its report internally.
• Feb. 18: Microsoft confirms with DEVCOR a target date of Mar. 9 (tomorrow) for publishing security updates for the Exchange flaws. That is the second Tuesday of the month — a.k.a. “Patch Tuesday,” when Microsoft releases monthly security updates (and yes that means check back here tomorrow for the always riveting Patch Tuesday roundup).
• Feb. 26-27: Targeted exploitation gradually turns into a global mass-scan; attackers start rapidly backdooring vulnerable servers.
• Mar. 2: A week earlier than previously planned, Microsoft releases updates to plug 4 zero-day flaws.
• Mar. 3: Tens of thousands of Exchange servers compromised worldwide, with thousands more servers getting freshly hacked each hour.
• Mar. 5: KrebsOnSecurity breaks the news that at least 30,000 organizations in the U.S. — and hundreds of thousands worldwide — now have backdoors installed.
• Mar. 5: Wired.com confirms the reported number of victims. White House makes statement expressing concern over the size of the attack. Former CISA head Chris Krebs tweets the numbers are bigger than what’s been reported.
• Mar. 6: CISA says it is aware of “widespread domestic and international exploitation of Microsoft Exchange Server flaws.”
• Mar. 7-Present: Security experts continue effort to notify victims, coordinate remediation, and remain vigilant for “Stage 2” of this attack (further exploitation of already-compromised servers).

At least 30,000 organizations across the United States — including a significant number of small businesses, towns, cities and local governments — have over the past few days been hacked by an unusually aggressive Chinese cyber espionage unit that’s focused on stealing email from victim organizations, multiple sources tell KrebsOnSecurity. The espionage group is exploiting four newly-discovered flaws in Microsoft Exchange Server email software, and has seeded hundreds of thousands of victim organizations worldwide with tools that give the attackers total, remote control over affected systems.

On March 2, Microsoft released emergency security updates to plug four security holes in Exchange Server versions 2013 through 2019 that hackers were actively using to siphon email communications from Internet-facing systems running Exchange.

In the three days since then, security experts say the same Chinese cyber espionage group has dramatically stepped up attacks on any vulnerable, unpatched Exchange servers worldwide.

In each incident, the intruders have left behind a “web shell,” an easy-to-use, password-protected hacking tool that can be accessed over the Internet from any browser. The web shell gives the attackers administrative access to the victim’s computer servers.

Speaking on condition of anonymity, two cybersecurity experts who’ve briefed U.S. national security advisors on the attack told KrebsOnSecurity the Chinese hacking group thought to be responsible has seized control over “hundreds of thousands” of Microsoft Exchange Servers worldwide — with each victim system representing approximately one organization that uses Exchange to process email.

Microsoft said the Exchange flaws are being targeted by a previously unidentified Chinese hacking crew it dubbed “Hafnium,” and said the group had been conducting targeted attacks on email systems used by a range of industry sectors, including infectious disease researchers, law firms, higher education institutions, defense contractors, policy think tanks, and NGOs.

Microsoft’s initial advisory about the Exchange flaws credited Reston, Va. based Volexity for reporting the vulnerabilities. Volexity President Steven Adair said the company first saw attackers quietly exploiting the Exchange bugs on Jan. 6, 2021, a day when most of the world was glued to television coverage of the riot at the U.S. Capitol.

But Adair said that over the past few days the hacking group has shifted into high gear, moving quickly to scan the Internet for Exchange servers that weren’t yet protected by those security updates.

“We’ve worked on dozens of cases so far where web shells were put on the victim system back on Feb. 28 [before Microsoft announced its patches], all the way up to today,” Adair said. “Even if you patched the same day Microsoft published its patches, there’s still a high chance there is a web shell on your server. The truth is, if you’re running Exchange and you haven’t patched this yet, there’s a very high chance that your organization is already compromised.”

Reached for comment, Microsoft said it is working closely with the U.S. Cybersecurity & Infrastructure Security Agency (CISA), other government agencies, and security companies, to ensure it is providing the best possible guidance and mitigation for its customers.

“The best protection is to apply updates as soon as possible across all impacted systems,” a Microsoft spokesperson said in a written statement. “We continue to help customers by providing additional investigation and mitigation guidance. Impacted customers should contact our support teams for additional help and resources.”

Adair said he’s fielded dozens of calls today from state and local government agencies that have identified the backdoors in their Exchange servers and are pleading for help. The trouble is, patching the flaws only blocks the four different ways the hackers are using to get in. But it does nothing to undo the damage that may already have been done.

By all accounts, rooting out these intruders is going to require an unprecedented and urgent nationwide clean-up effort. Adair and others say they’re worried that the longer it takes for victims to remove the backdoors, the more likely it is that the intruders will follow up by installing additional backdoors, and perhaps broadening the attack to include other portions of the victim’s network infrastructure.

Security researchers have published a tool on Microsoft’s Github code repository that lets anyone scan the Internet for Exchange servers that have been infected with the backdoor shell.

KrebsOnSecurity has seen portions of a victim list compiled by running this tool, and it is not a pretty picture. The backdoor web shell is verifiably present on the networks of thousands of U.S. organizations, including banks, credit unions, non-profits, telecommunications providers, public utilities and police, fire and rescue units.

“It’s police departments, hospitals, tons of city and state governments and credit unions,” said one source who’s working closely with federal officials on the matter. “Just about everyone who’s running self-hosted Outlook Web Access and wasn’t patched as of a few days ago got hit with a zero-day attack.”

Another government cybersecurity expert who participated in a recent call with multiple stakeholders impacted by this hacking spree worries the cleanup effort required is going to be Herculean.

“On the call, many questions were from school districts or local governments that all need help,” the source said, speaking on condition they were not identified by name. “If these numbers are in the tens of thousands, how does incident response get done? There are just not enough incident response teams out there to do that quickly.”

When it released patches for the four Exchange Server flaws on Tuesday, Microsoft emphasized that the vulnerability did not affect customers running its Exchange Online service (Microsoft’s cloud-hosted email for businesses). But sources say the vast majority of the organizations victimized so far are running some form of Internet-facing Microsoft Outlook Web Access (OWA) email systems in tandem with Exchange servers internally.

“It’s a question worth asking, what’s Microsoft’s recommendation going to be?,” the government cybersecurity expert said. “They’ll say ‘Patch, but it’s better to go to the cloud.’ But how are they securing their non-cloud products? Letting them wither on the vine.”

The government cybersecurity expert said this most recent round of attacks is uncharacteristic of the kinds of nation-state level hacking typically attributed to China, which tends to be fairly focused on compromising specific strategic targets.

“Its reckless,” the source said. “It seems out of character for Chinese state actors to be this indiscriminate.”

Microsoft has said the incursions by Hafnium on vulnerable Exchange servers are in no way connected to the separate SolarWinds-related attacks, in which a suspected Russian intelligence group installed backdoors in network management software used by more than 18,000 organizations.

“We continue to see no evidence that the actor behind SolarWinds discovered or exploited any vulnerability in Microsoft products and services,” the company said.

Nevertheless, the events of the past few days may well end up far eclipsing the damage done by the SolarWinds intruders.

This is a fast-moving story, and likely will be updated multiple times throughout the day. Stay tuned.

Over the past few weeks, three of the longest running and most venerated Russian-language online forums serving thousands of experienced cybercriminals have been hacked. In two of the intrusions, the attackers made off with the forums’ user databases, including email and Internet addresses and hashed passwords.

On Tuesday, someone dumped thousands of usernames, email addresses and obfuscated passwords on the dark web apparently pilfered from Mazafaka (a.k.a. “Maza,” “MFclub“), an exclusive crime forum that has for more than a decade played host to some of the most experienced and infamous Russian cyberthieves.

At the top of a 35-page PDF leaked online is a private encryption key allegedly used by Maza administrators. The database also includes ICQ numbers for many users. ICQ, also known as “I seek you,” was an instant message platform trusted by countless early denizens of these older crime forums before its use fell out of fashion in favor of more private networks, such as Jabber and Telegram.

This is notable because ICQ numbers tied to specific accounts often are a reliable data point that security researchers can use to connect multiple accounts to the same user across many forums and different nicknames over time.

Cyber intelligence firm Intel 471 assesses that the leaked Maza database is legitimate, and that the file includes more than 3,000 rows containing usernames, partially obfuscated password hashes, email addresses and other contact details.

“The file comprised more than 3,000 rows, containing usernames, partially obfuscated password hashes, email addresses and other contact details,” Intel 471 found, noting that Maza forum visitors are now redirected to a breach announcement page. “Initial analysis of the leaked data pointed to its probable authenticity, as at least a portion of the leaked user records correlated with our own data holdings.”

The attack on Maza comes just weeks after another major Russian crime forum got plundered. On Jan. 20, a longtime administrator of the Russian language forum Verified disclosed that the community’s domain registrar had been hacked, and that the site’s domain was redirected to an Internet server the attackers controlled.

“Our [bitcoin] wallet has been cracked. Luckily, we did not keep large amounts in it, but this is an unpleasant incident anyway. Once the circumstances became clear, the admin assumed that THEORETICALLY, all the forum’s accounts could have been compromised (the probability is low, but it is there). In our business, it’s better to play safe. So, we’ve decided to reset everyone’s codes. This is not a big deal. Simply write them down and use them from now on.”

A short time later, the administrator updated his post, saying:

“We are getting messages that the forum’s databases were filched after all when the forum was hacked. Everyone’s account passwords were forcibly reset. Pass this information to people you know. The forum was hacked through the domain registrar. The registrar was hacked first, then domain name servers were changed, and traffic was sniffed.”

On Feb. 15, the administrator posted a message purportedly sent on behalf of the intruders, who claimed they hacked Verified’s domain registrar between Jan. 16 and 20.

“It should be clear by now that the forum administration did not do an acceptable job with the security of this whole thing,” the attacker explained. “Most likely just out of laziness or incompetence, they gave up the whole thing. But the main surprise for us was that they saved all the user data, including cookies, referrers, ip addresses of the first registrations, login analytics, and everything else.”

The compromise of Maza and Verified — and possibly a third major forum — has many community members concerned that their real-life identities could be exposed. Exploit — perhaps the next-largest and most popular Russian forum after Verified, also experienced an apparent compromise this week.

According to Intel 471, on March 1, 2021, the administrator of the Exploit cybercrime forum claimed that a proxy server the forum used for protection from distributed denial-of-service (DDoS) attacks might have been compromised by an unknown party. The administrator stated that on Feb. 27, 2021, a monitoring system detected unauthorized secure shell access to the server and an attempt to dump network traffic.

Some forum lurkers have speculated that these recent compromises feel like the work of some government spy agency.

“Only intelligence services or people who know where the servers are located can pull off things like that,” mused one mainstay of Exploit. “Three forums in one month is just weird. I don’t think those were regular hackers. Someone is purposefully ruining forums.”

Others are wondering aloud which forum will fall next, and bemoaning the loss of trust among users that could be bad for business.

“Perhaps they work according to the following logic,” wrote one Exploit user. “There will be no forums, there will be no trust between everyone, less cooperation, more difficult to find partners – fewer attacks.”

Microsoft Corp. today released software updates to plug four security holes that attackers have been using to plunder email communications at companies that use its Exchange Server products. The company says all four flaws are being actively exploited as part of a complex attack chain deployed by a previously unidentified Chinese cyber espionage group.

The software giant typically releases security updates on the second Tuesday of each month, but it occasionally deviates from that schedule when addressing active attacks that target newly identified and serious vulnerabilities in its products.

The patches released today fix security problems in Microsoft Exchange Server 2013, 2016 and 2019. Microsoft said its Exchange Online service — basically hosted email for businesses — is not impacted by these flaws.

Microsoft credited researchers at Reston, Va. based Volexity for reporting the attacks. Volexity President Steven Adair told KrebsOnSecurity it first spotted the attacks on Jan. 6, 2021.

Adair said while the exploits used by the group may have taken great skills to develop, they require little technical know-how to use and can give an attacker easy access to all of an organization’s email if their vulnerable Exchange Servers are directly exposed to the Internet.

“These flaws are very easy to exploit,” Adair said. “You don’t need any special knowledge with these exploits. You just show up and say ‘I would like to break in and read all their email.’ That’s all there is to it.”

Microsoft says the flaws are being used by a previously unknown Chinese espionage group that’s been dubbed “Hafnium,” which is known to launch its attacks using hosting companies based in the United States.

“Hafnium primarily targets entities in the United States across a number of industry sectors, including infectious disease researchers, law firms, higher education institutions, defense contractors, policy think tanks, and NGOs,” Microsoft said. “HAFNIUM has previously compromised victims by exploiting vulnerabilities in internet-facing servers. Once they’ve gained access to a victim network, HAFNIUM typically exfiltrates data to file sharing sites like MEGA.”

According to Microsoft, Hafnium attackers have been observed combining all four zero-day flaws to target organizations running vulnerable Exchange Server products.

CVE-2021-26855 is a “server-side request forgery” (SSRF) flaw, in which a server (in this case, an on-premises Exchange Server) can be tricked into running commands that it should never have been permitted to run, such as authenticating as the Exchange server itself.

The attackers used CVE-2021-26857 to run code of their choice under the “system” account on a targeted Exchange server. The other two zero-day flaws — CVE-2021-26858 and CVE-2021-27065 — could allow an attacker to write a file to any part of the server.

After exploiting these vulnerabilities to gain initial access, Hafnium operators deployed web shells on the compromised server, Microsoft said. Web shells are essentially software backdoors that allow attackers to steal data and perform additional malicious actions that lead to further compromise.

Neither Microsoft nor Volexity is aware of publicly available code that would allow other cybercriminals to exploit these Exchange vulnerabilities. But given that these attacks are in the wild now, it may only be a matter of days before exploit code is publicly available online.

Microsoft stressed that the exploits detailed today were in no way connected to the separate SolarWinds-related attacks. “We continue to see no evidence that the actor behind SolarWinds discovered or exploited any vulnerability in Microsoft products and services,” the company said.

Further reading:

Microsoft’s writeup on new Hafnium nation state cyberattacks

Microsoft technical advisory on the four Exchange Server flaws

PrismHR, a company that sells technology used by other firms to help more than 80,000 small businesses manage payroll, benefits, and human resources, has suffered what appears to be an ongoing ransomware attack that is disrupting many of its services.

Hopkinton, Mass.-based PrismHR handles everything from payroll processing and human resources to health insurance and tax forms for hundreds of “professional employer organizations” (PEOs) that serve more than two million employees. The company processes more than $80 billion payroll payments annually on behalf of PEOs and their clients.

Countless small businesses turn to PEOs in part because they simplify compliance with various state payroll taxes, and because PEOs are the easiest way for small businesses to pool their resources and obtain more favorable health insurance rates for their employees.

PrismHR has not yet responded to requests for comment. But in a notice sent to its PEO partners, PrismHR said it detected suspicious activity within its networks on Feb. 28, and that it disabled access to its platform for all users in an effort to contain the security incident.

The company said the disruption has affected 200 PEO clients across the country, and that the most immediate concern is helping PEOs ensure their customers can process payrolls this week.

“The outage may extend throughout today and possibly later, with potential impact on payroll processing,” Prism explained in a template email it suggested PEO partners share with their customers. “We are committed to ensuring everyone receives their pay as timely and as accurately as possible. For this payroll period, we will use estimates from the last available payroll period. Once the software platform is back online, we will perform a reconciliation and correct any discrepancies as soon as possible.”

Jacob Cloran is co-founder of Decimal, a company that does accounting for small businesses, many of whom rely on PEOs affected by the PrismHR outage. Decimal itself uses a PEO that relies on PrismHR.

“We don’t have a good option to run our payroll this week, and the message we’ve received from our PEO doesn’t give me a lot of confidence we’ll be able to do that,” Cloran said.

Cloran said while there are other cloud-based companies that work with multiple PEOs, PrismHR is by far the largest.

“Prism is the only real option on the PEO software market,” he said. “Everyone I know who has tried any of the others ends up back at Prism. It’s the best of all bad available options.”

PrismHR did not specify what was responsible for the suspicious network activity, but their actions so far are straight out of the textbook recommendations for responding to a ransomware outbreak. A notice from the PEO working with some of Cloran’s clients stated that PrismHR was in the process of rebuilding its entire system from data backups in a new environment.

Also, the crooks behind ransomware attacks typically wait until the weekend to unleash their malware within victim organizations, knowing that most targets will be short-staffed or out of the office at this time. PrismHR said it detected the activity on Sunday.

Ransomware victims perhaps in the toughest spot include those providing cloud data hosting and software-as-service offerings, as these businesses are often unable to serve their customers while a ransomware infestation is active.

Ransomware renders any files it touches unreadable unless and until a victim pays for a digital key needed to unlock the encryption on them. Worse, it has become almost a best practice among ransomware criminal groups to steal as much data as possible from the victim organization prior to unleashing the ransom malware within a target environment.

Some of that data is often then published on dark web victim shaming sites in a bid to force the victim company into paying up. Some companies victimized by ransomware even face dual ransom demands: One for a digital key needed to unlock access to files, and a second payment in exchange for a promise not to publish all of the stolen data. Those that refuse to be extorted are told to expect that huge amounts of sensitive company data will be published online or sold on the dark web (or both).

PrismHR said in a statement to its PEO customers that while its investigation and response to the incident is ongoing, the company “is not aware of any sensitive data being breached or compromised.”

Given the volume and sensitive nature of the data PrismHR managed on behalf of PEO clients, it’s no doubt those clients and their customers are hoping that statement is accurate as well.