The consulting firm PricewaterhouseCoopers recently published lessons learned from the disruptive and costly ransomware attack in May 2021 on Ireland’s public health system. The unusually candid post-mortem found that nearly two months elapsed between the initial intrusion and the launching of the ransomware. It also found affected hospitals had tens of thousands of outdated Windows 7 systems, and that the health system’s IT administrators failed to respond to multiple warning signs that a massive attack was imminent.

Ireland’s Health Service Executive (HSE), which operates the country’s public health system, got hit with Conti ransomware on May 14, 2021. A timeline in the report (above) says the initial infection of the “patient zero” workstation happened on Mar. 18, 2021, when an employee on a Windows computer opened a booby-trapped Microsoft Excel document in a phishing email that had been sent two days earlier.

Less than a week later, the attacker had established a reliable backdoor connection to the employee’s infected workstation. After infecting the system, “the attacker continued to operate in the environment over an eight week period until the detonation of the Conti ransomware on May 14, 2021,” the report states.

According to PWC’s report (PDF), there were multiple warning signs about a serious network intrusion, but those red flags were either misidentified or not acted on quickly enough:

• On Mar. 31, 2021, the HSE’s antivirus software detected the execution of two software tools commonly used by ransomware groups — Cobalt Strike and Mimikatz — on the Patient Zero Workstation. But the antivirus software was set to monitor mode, so it did not block the malicious commands.”
• On May 7, the attacker compromised the HSE’s servers for the first time, and over the next five days the intruder would compromise six HSE hospitals. On May 10, one of the hospitals detected malicious activity on its Microsoft Windows Domain Controller, a critical “keys to the kingdom” component of any Windows enterprise network that manages user authentication and network access.
• On 10 May 2021, security auditors first identified evidence of the attacker compromising systems within Hospital C and Hospital L. Hospital C’s antivirus software detected Cobalt Strike on two systems but failed to quarantine the malicious files.
• On May 13, the HSE’s antivirus security provider emailed the HSE’s security operations team, highlighting unhandled threat events dating back to May 7 on at least 16 systems. The HSE Security Operations team requested that the Server team restart servers.

By then it was too late. At just after midnight Ireland time on May 14, the attacker executed the Conti ransomware within the HSE. The attack disrupted services at several Irish hospitals and resulted in the near complete shutdown of the HSE’s national and local networks, forcing the cancellation of many outpatient clinics and healthcare services. The number of appointments in some areas dropped by up to 80 percent.”

Conti initially demanded USD $20 million worth of virtual currency in exchange for a digital key to unlock HSE servers compromised by the group. But perhaps in response to the public outcry over the HSE disruption, Conti reversed course and gave the HSE the decryption keys without requiring payment.

Still, the work to restore infected systems would take months. The HSE ultimately enlisted members of the Irish military to bring in laptops and PCs to help restore computer systems by hand. It wasn’t until September 21, 2021 that the HSE declared 100 percent of its servers were decrypted.

As bad as the HSE ransomware attack was, the PWC report emphasizes that it could have been far worse. For example, it is unclear how much data would have been unrecoverable if a decryption key had not become available as the HSE’s backup infrastructure was only periodically backed up to offline tape.

The attack also could have been worse, the report found:

• if there had been intent by the Attacker to target specific devices within the HSE environment (e.g. medical devices);
• if the ransomware took actions to destroy data at scale;
• if the ransomware had auto-propagation and persistence capabilities, for example by using an exploit to propagate across domains and trust-boundaries to medical devices (e.g. the EternalBlue exploit used by the WannaCry and NotPetya15 attacks);
• if cloud systems had also been encrypted such as the COVID-19 vaccination system

The PWC report contains numerous recommendations, most of which center around hiring new personnel to lead the organization’s redoubled security efforts. But it is clear that the HSE has an enormous amount of work ahead to grow in security maturity. For example, the report notes the HSE’s hospital network had over 30,000 Windows 7 workstations that were deemed end of life by the vendor.

“The HSE assessed its cybersecurity maturity rating as low,” PWC wrote. “For example, they do not have a CISO or a Security Operations Center established.”

PWC also estimates that efforts to build up the HSE’s cybersecurity program to the point where it can rapidly detect and respond to intrusions are likely to cost “a multiple of the HSE’s current capital and operation expenditure in these areas over several years.”

In June 2021, the HSE’s director general said the recovery costs for the May ransomware attack were likely to exceed USD $600 million.

What’s remarkable about this incident is that the HSE is publicly funded by the Irish government, and so in theory it has the money to spend (or raise) to pay for all these ambitious recommendations for increasing their security maturity.

That stands in stark contrast to the healthcare system here in the United States, where the single biggest impediment to doing security well continues to be lack of making it a real budget priority. Also, most healthcare organizations in the United States are private companies that operate on razor-thin profit margins.

I know this because in 2018 I was asked to give the keynote at an annual gathering of the Healthcare Information Sharing and Analysis Group (H-ISAC), an industry group centered on sharing information about cybersecurity threats. I almost didn’t accept the invitation: I’d written very little about healthcare security, which seemed to be dominated by coverage of whether healthcare organizations complied with the letter of the law in the United States. That compliance centered on the Health Insurance Portability and Accountability Act (HIPPA), which prioritizes protecting the integrity and privacy of patient data.

To get up to speed, I interviewed over a dozen of the healthcare security industry’s best and brightest minds. A common refrain I heard from those interviewed was that if it was security-related but didn’t have to do with compliance, there probably wasn’t much chance it would get any budget.

Those sources unanimously said that however well-intentioned, it’s not clear that the “protect the data” regulatory approach of HIPPA was working from an overall threat perspective. According to HealthcareIT News, more than 40 million patient records have been compromised in incidents reported to the federal government in 2021 so far alone.

During my 2018 talk, I tried to emphasize the primary importance of being able to respond quickly to intrusions. Here’s a snippet of what I told that H-ISAC audience:

“The term ‘Security Maturity’ refers to the street smarts of an individual or organization, and this maturity generally comes from making plenty of mistakes, getting hacked a lot, and hopefully learning from each incident, measuring response times, and improving.

Let me say up front that all organizations get hacked. Even ones that are doing everything right from a security perspective get hacked probably every day if they’re big enough. By hacked I mean someone within the organization falls for a phishing scam, or clicks a malicious link and downloads malware. Because let’s face it, it only takes one screw up for the hackers to get a foothold in the network.

Now this is in itself isn’t bad. Unless you don’t have the capability to detect it and respond quickly. And if you can’t do that, you run the serious risk of having a small incident metastasize into a much larger problem.

Think of it like the medical concept of the ‘Golden Hour:’ That short window of time directly following a traumatic injury like a stroke or heart attack in which life-saving medicine and attention is likely to be most effective. The same concept holds true in cybersecurity, and it’s exactly why so many organizations these days are placing more of their resources into incident response, instead of just prevention.”

The United States’ somewhat decentralized healthcare system means that many ransomware outbreaks tend to be limited to regional or local healthcare facilities. But a well-placed ransomware attack or series of attacks could inflict serious damage on the sector: A December 2020 report from Deloitte says the top 10 health systems now control 24 market share and their revenue grew at twice the rate of the rest of the market.

In October 2020, KrebsOnSecurity broke the story that the FBI and U.S. Department of Homeland Security had obtained chatter from a top ransomware group which warned of an “imminent cybercrime threat to U.S. hospitals and healthcare providers.” Members associated with the Russian-speaking ransomware group known as Ryuk had discussed plans to deploy ransomware at more than 400 healthcare facilities in the United States.

Hours after that piece ran, I head from a respected H-ISAC security professional who questioned whether it was worth getting the public so riled up. The story had been updated multiple times throughout the day, and there were at least five healthcare organizations hit with ransomware within the span of 24 hours.

“I guess it would help if I understood what the baseline is, like how many healthcare organizations get hit with ransomware on average in one week?” I asked the source.

“It’s more like one a day,” the source confided.

In all likelihood, the HSE will get the money it needs to implement the programs recommended by PWC, however long that takes. I wonder how many U.S.-based healthcare organizations could say the same.

A 31-year-old Canadian man has been arrested and charged with fraud in connection with numerous ransomware attacks against businesses, government agencies and private citizens throughout Canada and the United States. Canadian authorities describe him as “the most prolific cybercriminal we’ve identified in Canada,” but so far they’ve released few other details about the investigation or the defendant. Helpfully, an email address and nickname apparently connected to the accused offer some additional clues.

Matthew Philbert of Ottawa, Ontario was charged with fraud and conspiracy in a joint law enforcement action by Canadian and U.S. authorities dubbed “Project CODA.” The Ontario Provincial Police (OPP) on Tuesday said the investigation began in January 2020 when the U.S. Federal Bureau of Investigation (FBI) contacted them regarding ransomware attacks that were based in Canada.

“During the course of this investigation, OPP investigators determined an individual was responsible for numerous ransomware attacks affecting businesses, government agencies and private individuals throughout Canada as well as cyber-related offenses in the United States,” reads an OPP statement.

“A quantity of evidentiary materials was seized and held for investigation, including desktop and laptop computers, a tablet, several hard drives, cellphones, a Bitcoin seed phrase and a quantity of blank cards with magnetic stripes,” the statement continues.

The U.S. indictment of Philbert (PDF) is unusually sparse, but it does charge him with conspiracy, suggesting the defendant was part of a group. In an interview with KrebsOnSecurity, OPP Detective Inspector Matt Watson declined to say whether other defendants were being sought in connection with the investigation, but said the inquiry is ongoing.

“I will say this, Philbert is the most prolific cybercriminal we’ve identified to date in Canada,” Watson said. “We’ve identified in excess of a thousand of his victims. And a lot of these were small businesses that were just holding on by their fingernails during COVID.”

A DARK CLOUD

There is a now-dormant Myspace account for a Matthew Philbert from Orleans, a suburb of Ottawa, Ontario. The information tied to the Myspace account matches the age and town of the defendant. The Myspace account was registered under the nickname “Darkcloudowner,” and to the email address dark_cl0ud6@hotmail.com.

A search in DomainTools on that email address reveals multiple domains registered to a Matthew Philbert and to the Ottawa phone number 6138999251 [DomainTools is a frequent advertiser on this site]. That same phone number is tied to a Facebook account for a 31-year-old Matthew Philbert from Orleans, who describes himself as a self-employed “broke bitcoin baron.”

Mr. Filbert did not respond to multiple requests for comment.

According to cyber intelligence firm Intel 471, that dark_cl0ud6@hotmail.com address has been used in conjunction with the handle “DCReavers2” to register user accounts on a half-dozen English-language cybercrime forums since 2008, including Hackforums, Blackhatworld, and Ghostmarket.

Perhaps the earliest and most important cybercrime forum DCReavers2 frequented was Darkode, where he was among the first two-dozen members. Darkode was taken down in 2015 as part of an FBI investigation sting operation, but screenshots of the community saved by this author show that DCReavers2 was already well known to the Darkode founders when his membership to the forum was accepted in May 2009.

Most of DCReavers’s posts on Darkode appear to have been removed by forum administrators early on (likely at DCReavers’ request), but the handful of posts that survived the purge show that more than a decade ago DCReavers2 was involved in running botnets, or large collections of hacked computers.

“My exploit pack is hosted there with 0 problems,” DCReaver2 says of a shady online provider that another member asked about in May 2010.

Searching the Web on “DCreavers2” brings up a fascinating chat conversation allegedly between DCReavers2 and an individual in Australia who was selling access to an “exploit kit,” commercial crimeware designed to be stitched into hacked or malicious sites and exploit a variety of Web-browser vulnerabilities for the purposes of installing malware of the customer’s choosing.

In that 2009 chat, indexed by the researchers behind the website exposedbotnets.com, DCReavers2 uses the Dark_Cl0ud6 email address and actually shares his real name as Matthew Philbert. DCReavers2 also says his partner uses the nickname “The Rogue,” which corresponds to a former Darkode administrator who was the second user ever registered on the forum (see screenshot above).

In that same conversation, DCReavers2 discusses managing a botnet built on ButterFly Bot. Also known as “Mariposa,” ButterFly was a plug-and-play malware strain that allowed even the most novice of would-be cybercriminals to set up a global operation capable of harvesting data from thousands of infected PCs, and using the enslaved systems for crippling attacks on Web sites. The ButterFly Bot kit sold for prices ranging from $500 to $2,000.

The author of ButterFly Bot — Slovenian hacker Matjaz “Iserdo” Skorjanc — was Darkode’s original founder back in 2008. Arrested in 2010, Skorjanc was sentenced to nearly five years in prison for selling and supporting Mariposa, which was used to compromise millions of Microsoft Windows computers.

Upon release from prison, Skorjanc became chief technology officer for NiceHash, a cryptocurrency mining service. In December 2017, $52 million worth of Bitcoin mysteriously disappeared from NiceHash coffers. In October 2019, Skorjanc was arrested in Germany in response to a U.S.-issued international arrest warrant for his extradition.

The indictment (PDF) tied to Skorjanc’s 2019 arrest also names several other alleged founding members of Darkode, including Thomas “Fubar” McCormick, a Massachusetts man who was allegedly one of the last administrators of Darkode. Prosecutors say McCormick also was a reseller of the Mariposa botnet, the ZeuS banking trojan, and a bot malware he allegedly helped create called “Ngrbot.” The U.S. federal prosecution against Skorjanc and McCormick is ongoing.

At the time the FBI dismantled Darkode in 2015, the Justice Department said that out of 800 or so crime forums worldwide, Darkode was the most sophisticated English-language forum, and that it represented “one of the gravest threats to the integrity of data on computers in the United States and around the world.”

Some of Darkode’s core members were either customers or sellers of various “locker” kits, which were basically web-based exploits that would lock the victim’s screen into a webpage spoofing the FBI or Justice Department and warning that victims had been caught accessing child sexual abuse material. Victims who agreed to pay a “fine” of several hundred dollars worth of GreenDot prepaid cards could then be rid of the PC locker program.

In many ways, lockers were the precursors to the modern cybercrime scourge we now know as ransomware. The main reason lockers never took off as an existential threat to organizations worldwide was that there is only so much money locker users could reasonably demand via GreenDot cards.

But with the ascendance and broader acceptance of virtual currencies like Bitcoin, suddenly criminal hackers could start demanding millions of dollars from victims. And it stands to reason that a great many Darkode members who were never caught have since transitioned from lockers, exploit kits and GreenDot cards to doing what every other self-respecting cybercrook seems to be involved with these days: Locking entire companies and industries for ransomware payments.

One final observation about the Philbert indictment: It’s good to see the Canadian authorities working closely with the FBI on important cybercrime cases. Indeed, this investigation is remarkable for that fact alone. For years I’ve been wondering aloud why more American cybercriminals don’t just move to Canada, because historically there has been almost no probability that they will ever get caught — let alone prosecuted there. With any luck, this case will be the start of something new.

Rarely do cybercriminal gangs that deploy ransomware gain the initial access to the target themselves. More commonly, that access is purchased from a cybercriminal broker who specializes in acquiring remote access credentials — such as usernames and passwords needed to remotely connect to the target’s network. In this post we’ll look at the clues left behind by “Babam,” the handle chosen by a cybercriminal who has sold such access to ransomware groups on many occasions over the past few years.

Since the beginning of 2020, Babam has set up numerous auctions on the Russian-language cybercrime forum Exploit, mainly selling virtual private networking (VPN) credentials stolen from various companies. Babam has authored more than 270 posts since joining Exploit in 2015, including dozens of sales threads. However, none of Babam’s posts on Exploit include any personal information or clues about his identity.

But in February 2016, Babam joined Verified, another Russian-language crime forum. Verified was hacked at least twice in the past five years, and its user database posted online. That information shows that Babam joined Verified using the email address “operns@gmail.com.” The latest Verified leak also exposed private messages exchanged by forum members, including more than 800 private messages that Babam sent or received on the forum over the years.

In early 2017, Babam confided to another Verified user via private message that he is from Lithuania. In virtually all of his forum posts and private messages, Babam can be seen communicating in transliterated Russian rather than by using the Cyrillic alphabet. This is common among cybercriminal actors for whom Russian is not their native tongue.

Cyber intelligence platform Constella Intelligence told KrebsOnSecurity that the operns@gmail.com address was used in 2016 to register an account at filmai.in, which is a movie streaming service catering to Lithuanian speakers. The username associated with that account was “bo3dom.”

A reverse WHOIS search via DomainTools.com says operns@gmail.com was used to register two domain names: bonnjoeder[.]com back in 2011, and sanjulianhotels[.]com (2017). It’s unclear whether these domains ever were online, but the street address on both records was “24 Brondeg St.” in the United Kingdom. [Full disclosure: DomainTools is a frequent advertiser on this website.]

A reverse search at DomainTools on “24 Brondeg St.” reveals one other domain: wwwecardone[.]com. The use of domains that begin with “www” is fairly common among phishers, and by passive “typosquatting” sites that seek to siphon credentials from legitimate websites when people mistype a domain, such as accidentally omitting the “.” after typing “www”.

Searching DomainTools for the phone number in the WHOIS records for wwwecardone[.]com  — +44.0774829141 — leads to a handful of similar typosquatting domains, including wwwebuygold[.]com and wwwpexpay[.]com. A different UK phone number in a more recent record for the wwwebuygold[.]com domain — 44.0472882112 — is tied to two more domains – howtounlockiphonefree[.]com, and portalsagepay[.]com. All of these domains date back to between 2012 and 2013.

The original registration records for the iPhone, Sagepay and Gold domains share an email address: devrian26@gmail.com. A search on the username “bo3dom” using Constella’s service reveals an account at ipmart-forum.com, a now-defunct forum concerned with IT products, such as mobile devices, computers and online gaming. That search shows the user bo3dom registered at ipmart-forum.com with the email address devrian27@gmail.com, and from an Internet address in Vilnius, Lithuania.

Devrian27@gmail.com was used to register multiple domains, including wwwsuperchange.ru back in 2008 (notice again the suspect “www” as part of the domain name). Gmail’s password recovery function says the backup email address for devrian27@gmail.com is bo3*******@gmail.com. Gmail accepts the address bo3domster@gmail.com as the recovery email for that devrian27 account.

According to Constella, the bo3domster@gmail.com address was exposed in multiple data breaches over the years, and in each case it used one of two passwords: “lebeda1” and “a123456“.

Searching in Constella for accounts using those passwords reveals a slew of additional “bo3dom” email addresses, including bo3dom@gmail.com.  Pivoting on that address in Constella reveals that someone with the name Vytautas Mockus used it to register an account at mindjolt.com, a site featuring dozens of simple puzzle games that visitors can play online.

At some point, mindjolt.com apparently also was hacked, because a copy of its database at Constella says the bo3dom@gmail.com used two passwords at that site: lebeda1 and a123456.

A reverse WHOIS search on “Vytautas Mockus” at DomainTools shows the email address devrian25@gmail.com was used in 2010 to register the domain name perfectmoney[.]co. This is one character off of perfectmoney[.]com, which is an early virtual currency that was quite popular with cybercriminals at the time. The phone number tied to that domain registration was “86.7273687“.

A Google search for “Vytautas Mockus” says there’s a person by that name who runs a mobile food service company in Lithuania called “Palvisa.” A report on Palvisa (PDF) purchased from Rekvizitai.vz — an official online directory of Lithuanian companies — says Palvisa was established in 2011 by a Vytautaus Mockus, using the phone number 86.7273687, and the email address bo3dom@gmail.com. The report states that Palvisa is active, but has had no employees other than its founder.

Reached via the bo3dom@gmail.com address, the 36-year-old Mr. Mockus expressed mystification as to how his personal information wound up in so many records. “I am not involved in any crime,” Mockus wrote in reply.

The domains apparently registered by Babam over nearly 10 years suggest he started off mainly stealing from other cybercrooks. By 2015, Babam was heavily into “carding,” the sale and use of stolen payment card data. By 2020, he’d shifted his focus almost entirely to selling access to companies.

A profile produced by threat intelligence firm Flashpoint says Babam has received at least four positive feedback reviews on the Exploit cybercrime forum from crooks associated with the LockBit ransomware gang.

According to Flashpoint, in April 2021 Babam advertised the sale of Citrix credentials for an international company that is active in the field of laboratory testing, inspection and certification, and that has more than $5 billion in annual revenues and more than 78,000 employees.

Flashpoint says Babam initially announced he’d sold the access, but later reopened the auction because the prospective buyer backed out of the deal. Several days later, Babam reposted the auction, adding more information about the depth of the illicit access and lowering his asking price. The access sold less than 24 hours later.

“Based on the provided statistics and sensitive source reporting, Flashpoint analysts assess with high confidence that the compromised organization was likely Bureau Veritas, an organization headquartered in France that operates in a variety of sectors,” the company concluded.

In November, Bureau Veritas acknowledged that it shut down its network in response to a cyber attack. The company hasn’t said whether the incident involved ransomware and if so what strain of ransomware, but its response to the incident is straight out of the playbook for responding to ransomware attacks. Bureau Veritas has not yet responded to requests for comment; its latest public statement on Dec. 2 provides no additional details about the cause of the incident.

Flashpoint notes that Babam’s use of transliterated Russian persists on both Exploit and Verified until around March 2020, when he switches over to using mostly Cyrillc in his forum comments and sales threads. Flashpoint said this could be an indication that a different person started using the Babam account since then, or more likely that Babam had only a tenuous grasp of Russian to begin with and that his language skills and confidence improved over time.

Lending credence to the latter theory is that Babam still makes linguistic errors in his postings that suggest Russian is not his original language, Flashpoint found.

“The use of double “n” in such words as “проданно” (correct – продано) and “сделанны” (correct – сделаны) by the threat actor proves that this style of writing is not possible when using machine translation since this would not be the correct spelling of the word,” Flashpoint analysts wrote.

“These types of grammatical errors are often found among people who did not receive sufficient education at school or if Russian is their second language,” the analysis continues. “In such cases, when someone tries to spell a word correctly, then by accident or unknowingly, they overdo the spelling and make these types of mistakes. At the same time, colloquial speech can be fluent or even native. This is often typical for a person who comes from the former Soviet Union states.”

In January 2021, technology vendor Ubiquiti Inc. [NYSE:UI] disclosed that a breach at a third party cloud provider had exposed customer account credentials. In March, a Ubiquiti employee warned that the company had drastically understated the scope of the incident, and that the third-party cloud provider claim was a fabrication. On Wednesday, a former Ubiquiti developer was arrested and charged with stealing data and trying to extort his employer while pretending to be a whistleblower.

Federal prosecutors say Nickolas Sharp, a senior developer at Ubiquiti, actually caused the “breach” that forced Ubiquiti to disclose a cybersecurity incident in January. They allege that in late December 2020, Sharp applied for a job at another technology company, and then abused his privileged access to Ubiquiti’s systems at Amazon’s AWS cloud service and the company’s GitHub accounts to download large amounts of proprietary data.

Sharp’s indictment doesn’t specify how much data he allegedly downloaded, but it says some of the downloads took hours, and that he cloned approximately 155 Ubiquiti data repositories via multiple downloads over nearly two weeks.

On Dec. 28, other Ubiquiti employees spotted the unusual downloads, which had leveraged internal company credentials and a Surfshark VPN connection to hide the downloader’s true Internet address. Assuming an external attacker had breached its security, Ubiquiti quickly launched an investigation.

But Sharp was a member of the team doing the forensic investigation, the indictment alleges.

“At the time the defendant was part of a team working to assess the scope and damage caused by the incident and remediate its effects, all while concealing his role in committing the incident,” wrote prosecutors with the Southern District of New York.

According to the indictment, on January 7 a senior Ubiquiti employee received a ransom email. The message was sent through an IP address associated with the same Surfshark VPN. The ransom message warned that internal Ubiquiti data had been stolen, and that the information would not be used or published online as long as Ubiquiti agreed to pay 25 Bitcoin.

The ransom email also offered to identify a purportedly still unblocked “backdoor” used by the attacker for the sum of another 25 Bitcoin (the total amount requested was equivalent to approximately $1.9 million at the time). Ubiquiti did not pay the ransom demands.

Investigators say they were able to tie the downloads to Sharp and his work-issued laptop because his Internet connection briefly failed on several occasions while he was downloading the Ubiquiti data. Those outages were enough to prevent Sharp’s Surfshark VPN connection from functioning properly — thus exposing his Internet address as the source of the downloads.

When FBI agents raided Sharp’s residence on Mar. 24, he reportedly maintained his innocence and told agents someone else must have used his Paypal account to purchase the Surfshark VPN subscription.

Several days after the FBI executed its search warrant, Sharp “caused false or misleading news stories to be published about the incident,” prosecutors say. Among the claims made in those news stories was that Ubiquiti had neglected to keep access logs that would allow the company to understand the full scope of the intrusion. In reality, the indictment alleges, Sharp had shortened to one day the amount of time Ubiquiti’s systems kept certain logs of user activity in AWS.

“Following the publication of these articles, between Tuesday, March 30, 2021 and Wednesday March 31, [Ubiquiti’s] stock price fell approximately 20 percent, losing over four billion dollars in market capitalization,” the indictment states.

Sharp faces four criminal counts, including wire fraud, intentionally damaging protected computers, transmission of interstate communications with intent to extort, and making false statements to the FBI.

News of Sharp’s arrest was first reported by BleepingComputer, which wrote that while the Justice Department didn’t name Sharp’s employer in its press release or indictment, all of the details align with previous reporting on the Ubiquiti incident and information presented in Sharp’s LinkedIn account. A link to the indictment is here (PDF).

Imagine being able to disconnect or redirect Internet traffic destined for some of the world’s biggest companies — just by spoofing an email. This is the nature of a threat vector recently removed by a Fortune 500 firm that operates one of the largest Internet backbones.

Based in Monroe, La., Lumen Technologies Inc. [NYSE: LUMN] (formerly CenturyLink) is one of more than two dozen entities that operate what’s known as an Internet Routing Registry (IRR). These IRRs maintain routing databases used by network operators to register their assigned network resources — i.e., the Internet addresses that have been allocated to their organization.

The data maintained by the IRRs help keep track of which organizations have the right to access what Internet address space in the global routing system. Collectively, the information voluntarily submitted to the IRRs forms a distributed database of Internet routing instructions that helps connect a vast array of individual networks.

There are about 70,000 distinct networks on the Internet today, ranging from huge broadband providers like AT&T, Comcast and Verizon to many thousands of enterprises that connect to the edge of the Internet for access. Each of these so-called “Autonomous Systems” (ASes) make their own decisions about how and with whom they will connect to the larger Internet.

Regardless of how they get online, each AS uses the same language to specify which Internet IP address ranges they control: It’s called the Border Gateway Protocol, or BGP. Using BGP, an AS tells its directly connected neighbor AS(es) the addresses that it can reach. That neighbor in turn passes the information on to its neighbors, and so on, until the information has propagated everywhere [1].

A key function of the BGP data maintained by IRRs is preventing rogue network operators from claiming another network’s addresses and hijacking their traffic. In essence, an organization can use IRRs to declare to the rest of the Internet, “These specific Internet address ranges are ours, should only originate from our network, and you should ignore any other networks trying to lay claim to these address ranges.”

In the early days of the Internet, when organizations wanted to update their records with an IRR, the changes usually involved some amount of human interaction — often someone manually editing the new coordinates into an Internet backbone router. But over the years the various IRRs made it easier to automate this process via email.

For a long time, any changes to an organization’s routing information with an IRR could be processed via email as long as one of the following authentication methods was successfully used:

-CRYPT-PW: A password is added to the text of an email to the IRR containing the record they wish to add, change or delete (the IRR then compares that password to a hash of the password);

-PGPKEY: The requestor signs the email containing the update with an encryption key the IRR recognizes;

-MAIL-FROM: The requestor sends the record changes in an email to the IRR, and the authentication is based solely on the “From:” header of the email.

Of these, MAIL-FROM has long been considered insecure, for the simple reason that it’s not difficult to spoof the return address of an email. And virtually all IRRs have disallowed its use since at least 2012, said Adam Korab, a network engineer and security researcher based in Houston.

All except Level3, a major Internet backbone provider acquired by Lumen/CenturyLink.

“LEVEL3 is the last IRR operator which allows the use of this method, although they have discouraged its use since at least 2012,” Korab told KrebsOnSecurity. “Other IRR operators have fully deprecated MAIL-FROM.”

Importantly, the name and email address of each Autonomous System’s official contact for making updates with the IRRs is public information.

Korab filed a vulnerability report with Lumen demonstrating how a simple spoofed email could be used to disrupt Internet service for banks, telecommunications firms and even government entities.

“If such an attack were successful, it would result in customer IP address blocks being filtered and dropped, making them unreachable from some or all of the global Internet,” Korab said, noting that he found more than 2,000 Lumen customers were potentially affected. “This would effectively cut off Internet access for the impacted IP address blocks.”

The recent outage that took Facebook, Instagram and WhatsApp offline for the better part of a day was caused by an erroneous BGP update submitted by Facebook. That update took away the map telling the world’s computers how to find its various online properties.

Now consider the mayhem that would ensue if someone spoofed IRR updates to remove or alter routing entries for multiple e-commerce providers, banks and telecommunications companies at the same time.

“Depending on the scope of an attack, this could impact individual customers, geographic market areas, or potentially the [Lumen] backbone,” Korab continued. “This attack is trivial to exploit, and has a difficult recovery. Our conjecture is that any impacted Lumen or customer IP address blocks would be offline for 24-48 hours. In the worst-case scenario, this could extend much longer.”

Lumen told KrebsOnSecurity that it continued offering MAIL-FROM: authentication because many of its customers still relied on it due to legacy systems. Nevertheless, after receiving Korab’s report the company decided the wisest course of action was to disable MAIL-FROM: authentication altogether.

“We recently received notice of a known insecure configuration with our Route Registry,” reads a statement Lumen shared with KrebsOnSecurity. “We already had mitigating controls in place and to date we have not identified any additional issues. As part of our normal cybersecurity protocol, we carefully considered this notice and took steps to further mitigate any potential risks the vulnerability may have created for our customers or systems.”

KC Claffy is the founder and director of the Center for Applied Internet Data Analysis (CAIDA), and a resident research scientist of the San Diego Supercomputer Center at the University of California, San Diego. Claffy said there is scant public evidence of a threat actor using the weakness now fixed by Lumen to hijack Internet routes.

“People often don’t notice, and a malicious actor certainly works to achieve this,” Claffy said in an email to KrebsOnSecurity. “But also, if a victim does notice, they generally aren’t going to release details that they’ve been hijacked. This is why we need mandatory reporting of such breaches, as Dan Geer has been saying for years.”

But there are plenty of examples of cybercriminals hijacking IP address blocks after a domain name associated with an email address in an IRR record has expired. In those cases, the thieves simply register the expired domain and then send email from it to an IRR specifying any route changes.

While it’s nice that Lumen is no longer the weakest link in the IRR chain, the remaining authentication mechanisms aren’t great. Claffy said after years of debate over approaches to improving routing security, the operator community deployed an alternative known as the Resource Public Key Infrastructure (RPKI).

“The RPKI includes cryptographic attestation of records, including expiration dates, with each Regional Internet Registry (RIR) operating as a ‘root’ of trust,” wrote Claffy and two other UC San Diego researchers in a paper that is still undergoing peer review. “Similar to the IRR, operators can use the RPKI to discard routing messages that do not pass origin validation checks.”

However, the additional integrity RPKI brings also comes with a fair amount of added complexity and cost, the researchers found.

“Operational and legal implications of potential malfunctions have limited registration in and use of the RPKI,” the study observed (link added). “In response, some networks have redoubled their efforts to improve the accuracy of IRR registration data. These two technologies are now operating in parallel, along with the option of doing nothing at all to validate routes.”

[1]: I borrowed some descriptive text in the 5th and 6th paragraphs from a CAIDA/UCSD draft paper — IRR Hygiene in the RPKI Era (PDF).

Further reading:

Trust Zones: A Path to a More Secure Internet Infrastructure (PDF).

Reviewing a historical Internet vulnerability: Why isn’t BGP more secure and what can we do about it? (PDF)