Microsoft this week released software updates to fix roughly 50 security problems with various versions of its Windows operating system and related software, including one flaw that is already being exploited and another for which exploit code is publicly available.

The zero-day bug — CVE-2018-8453 — affects Windows versions 7, 8.1, 10 and Server 2008, 2012, 2016 and 2019. According to security firm Ivanti, an attacker first needs to log into the operating system, but then can exploit this vulnerability to gain administrator privileges.

Another vulnerability patched on Tuesday — CVE-2018-8423 — was publicly disclosed last month along with sample exploit code. This flaw involves a component shipped on all Windows machines and used by a number of programs, and could be exploited by getting a user to open a specially-crafted file — such as a booby-trapped Microsoft Office document.

KrebsOnSecurity has frequently suggested that Windows users wait a day or two after Microsoft releases monthly security updates before installing the fixes, with the rational that occasionally buggy patches can cause serious headaches for users who install them before all the kinks are worked out.

This month, Microsoft briefly paused updates for Windows 10 users after many users reported losing all of the files in their “My Documents” folder. The worst part? Rolling back to previous saved versions of Windows prior to the update did not restore the files.

Microsoft appears to have since fixed the issue, but these kinds of incidents illustrate the value of not only waiting a day or two to install updates but also manually backing up your data prior to installing patches (i.e., not just simply counting on Microsoft’s System Restore feature to save the day should things go haywire).

Mercifully, Adobe has spared us an update this month for its Flash Player software, although it has shipped a non-security update for Flash.

For more on this month’s Patch Tuesday batch, check out posts from Ivanti and Qualys.

As always, if you experience any issues installing any of these patches this month, please feel free to leave a comment about it below; there’s a good chance other readers have experienced the same and may even chime in here with some helpful tips. My apologies for the tardiness of this post; I have been traveling in Australia this past week with only sporadic access to the Internet.

What do we do with a company that regularly pumps metric tons of virtual toxic sludge onto the Internet and yet refuses to clean up their act? If ever there were a technology giant that deserved to be named and shamed for polluting the Web, it is Xiongmai — a Chinese maker of electronic parts that power a huge percentage of cheap digital video recorders (DVRs) and Internet-connected security cameras.

In late 2016, the world witnessed the sheer disruptive power of Mirai, a powerful botnet strain fueled by Internet of Things (IoT) devices like DVRs and IP cameras that were put online with factory-default passwords and other poor security settings.

Security experts soon discovered that a majority of Mirai-infected devices were chiefly composed of components made by Xiongmai (a.k.a. Hangzhou Xiongmai Technology Co., Ltd.) and a handful of other Chinese tech firms that seemed to have a history of placing product market share and price above security.

Since then, two of those firms — Huawei and Dahua — have taken steps to increase the security of their IoT products out-of-the-box. But Xiongmai — despite repeated warnings from researchers about deep-seated vulnerabilities in its hardware — has continued to ignore such warnings and to ship massively insecure hardware and software for use in products that are white-labeled and sold by more than 100 third-party vendors.

On Tuesday, Austrian security firm SEC Consult released the results of extensive research into multiple, lingering and serious security holes in Xiongmai’s hardware.

SEC Consult said it began the process of working with Xiongmai on these problems back in March 2018, but that it finally published its research after it became clear that Xiongmai wasn’t going to address any of the problems.

“Although Xiongmai had seven months notice, they have not fixed any of the issues,” the researchers wrote in a blog post published today. “The conversation with them over the past months has shown that security is just not a priority to them at all.”

Xiongmai did not respond to requests for comment.

PROBLEM TO PROBLEM

A core part of the problem is the peer-to-peer (P2P) communications component called “XMEye” that ships with all Xiongmai devices and automatically connects them to a cloud network run by Xiongmai. The P2P feature is designed so that consumers can access their DVRs or security cameras remotely anywhere in the world and without having to configure anything.

To access a Xiongmai device via the P2P network, one must know the Unique ID (UID) assigned to each device. The UID is essentially derived in an easily reproducible way using the device’s built-in MAC address (a 16-character string of numbers and letters, such as 68ab8124db83c8db).

Electronics firms are assigned ranges of MAC address that they may use, but SEC Consult discovered that Xiongmai for some reason actually uses MAC address ranges assigned to a number of other companies, including tech giant Cisco Systems, German printing press maker Koenig & Bauer AG, and Swiss chemical analysis firm Metrohm AG.

SEC Consult learned that it was trivial to find Xiongmai devices simply by computing all possible ranges of UIDs for each range of MAC addresses, and then scanning Xiongmai’s public cloud for XMEye-enabled devices. Based on scanning just two percent of the available ranges, SEC Consult conservatively estimates there are around 9 million Xiongmai P2P devices online.

[For the record, KrebsOnSecurity has long advised buyers of IoT devices to avoid those advertise P2P capabilities for just this reason. The Xiongmai debacle is yet another example of why this remains solid advice].

BLANK TO BANK

While one still needs to provide a username and password to remotely access XMEye devices via this method, SEC Consult notes that the default password of the all-powerful administrative user (username “admin”) is blank (i.e, no password).

The admin account can be used to do anything to the device, such as changing its settings or uploading software — including malware like Mirai. And because users are not required to set a secure password in the initial setup phase, it is likely that a large number of devices are accessible via these default credentials.

Even if a customer has changed the default admin password, SEC Consult discovered there is an undocumented user with the name “default,” whose password is “tluafed” (default in reverse). While this user account can’t change system settings, it is still able to view any video streams.

Normally, hardware devices are secured against unauthorized software updates by requiring that any new software pushed to the devices be digitally signed with a secret cryptographic key that is held only by the hardware or software maker. However, XMEye-enabled devices have no such protections.

In fact, the researchers found it was trivial to set up a system that mimics the XMEye cloud and push malicious firmware updates to any device. Worse still, unlike with the Mirai malware — which gets permanently wiped from memory when an infected device powers off or is rebooted — the update method devised by SEC Consult makes it so that any software uploaded survives a reboot.

CAN XIONGMAI REALLY BE THAT BAD?

In the wake of the Mirai botnet’s emergence in 2016 and the subsequent record denial-of-service attacks that brought down chunks of the Internet at a time (including this Web site and my DDoS protection provider at times), multiple security firms said Xiongmai’s insecure products were a huge contributor to the problem.

Among the company’s strongest critics was New York City-based security firm Flashpoint, which pointed out that even basic security features built into Xiongmai’s hardware had completely failed at basic tasks.

For example, Flashpoint’s analysts discovered that the login page for a camera or DVR running Xiongmai hardware and software could be bypassed just by navigating to a page called “DVR.htm” prior to login.

Flashpoint’s researchers also found that any changes to passwords for various user accounts accessible via the Web administration page for Xiongmai products did nothing to change passwords for accounts that were hard-coded into these devices and accessible only via more obscure, command-line communications interfaces like Telnet and SSH.

Not long after Xiongmai was publicly shamed for failing to fix obvious security weaknesses that helped contribute to the spread of Mirai and related IoT botnets, Xiongmai lashed out at multiple security firms and journalists, promising to sue its critics for defamation (it never followed through on that threat, as far as I can tell).

At the same time, Xiongmai promised that it would be issuing a product recall on millions of devices to ensure they were not deployed with insecure settings and software. But according to Flashpoint’s Zach Wikholm, Xiongmai never followed through with the recall, either. Rather, it was all a way for the company to save face publicly and with its business partners.

“This company said they were going to do a product recall, but it looks like they never got around to it,” Wikholm said. “They were just trying to cover up and keep moving.”

Wikholm said Flashpoint discovered a number of additional glaring vulnerabilities in Xiongmai’s hardware and software that left them wide open to takeover by malicious hackers, and that several of those weaknesses still exist in the company’s core product line.

“We could have kept releasing our findings, but it just got really difficult to keep doing that because Xiongmai wouldn’t fix them and it would only make it easier for people to compromise these devices,” Wikholm said.

The Flashpoint analyst said he believes SEC Consult’s estimates of the number of vulnerable Xiongmai devices to be extremely conservative.

“Nine million devices sounds quite low because these guys hold 25 percent of the world’s DVR market,” to say nothing of the company’s share in the market for cheapo IP cameras, Wikholm said.

What’s more, he said, Xiongmai has turned a deaf ear to reports about dangerous security holes across its product lines principally because it doesn’t answer directly to customers who purchase the gear.

“The only reason they’ve maintained this level of [not caring] is they’ve been in this market for a long time and established very strong regional sales channels to dozens of third-party companies,” that ultimately rebrand Xiongmai’s products as their own, he said.

Also, the typical consumer of cheap electronics powered by Xiongmai’s kit don’t really care how easily these devices can be commandeered by cybercriminals, Wikholm observed.

“They just want a security system around their house or business that doesn’t cost an arm and leg, and Xiongmai is by far the biggest player in that space,” he said. “Most companies at least have some sort of incentive to make things better when faced with public pressure. But they don’t seem to have that drive.”

A PHANTOM MENACE

SEC Consult concluded its technical advisory about the security flaws by saying Xiongmai “does not provide any mitigations and hence it is recommended not to use any products associated with the XMeye P2P Cloud until all of the identified security issues have been fixed and a thorough security analysis has been performed by professionals.”

While this may sound easy enough, acting on that advice is difficult in practice because very few devices made with Xiongmai’s deeply flawed hardware and software advertise that fact on the label or product name. Rather, the components that Xiongmai makes are sold downstream to vendors who then use it in their own products and slap on a label with their own brand name.

How many vendors? It’s difficult to say for sure, but a search on the term XMEye via the e-commerce sites where Xiongmai’s white-labeled products typically are sold (Amazon, Aliexpress.com, Homedepot.com and Walmart) reveals more than 100 companies that you’ve probably never heard of which brand Xionmai’s hardware and software as their own.  That list is available here (PDF) and is also pasted at the conclusion of this post for the benefit of search engines.

SEC Consult’s technical advisory about their findings lists a number of indicators that system and network administrators can use to quickly determine whether any of these vulnerable P2P Xiongmai devices happen to be on your network.

For end users concerned about this, one way of fingerprinting Xiongmai devices is to search Amazon.com, aliexpress.com, walmart.com and other online merchants for the brand on the side of your device and the term “XMEye.” If you get a hit, chances are excellent you’ve got a device built on Xionmai’s technology.

Another option: open a browser and navigate to the local Internet address of your device. If you have one of these devices on your local network, the login page should look like the one below:

Another giveaway on virtually all Xiongmai devices is pasting “http://IP/err.htm” into a browser address bar should display the following error message (where IP= the local IP address of the device):

According to SEC Consult, Xiongmai’s electronics and hardware make up the guts of IP cameras and DVRs marketed and sold under the company names below.

What’s most remarkable about many of the companies listed below is that about half of them don’t even have their own Web sites, and instead simply rely on direct-to-consumer product listings at Amazon.com or other e-commerce outlets. Among those that do sell Xiongmai’s products directly via the Web, very few of them seem to even offer secure (https://) Web sites.

SEC Consult’s blog post about their findings has more technical details, as does the security advisory they released today.

Here’s the current list of companies that white label Xiongmai’s insecure products, according to SEC Consult:

9Trading
Abowone
AHWVSE
ANRAN
ASECAM
Autoeye
AZISHN
A-ZONE
BESDER/BESDERSEC
BESSKY
Bestmo
BFMore
BOAVISION
BULWARK
CANAVIS
CWH
DAGRO
datocctv
DEFEWAY
digoo
DiySecurityCameraWorld
DONPHIA
ENKLOV
ESAMACT
ESCAM
EVTEVISION
Fayele
FLOUREON
Funi
GADINAN
GARUNK
HAMROL
HAMROLTE
Highfly
Hiseeu
HISVISION
HMQC
IHOMEGUARD
ISSEUSEE
iTooner
JENNOV
Jooan
Jshida
JUESENWDM
JUFENG
JZTEK
KERUI
KKMOON
KONLEN
Kopda
Lenyes
LESHP
LEVCOECAM
LINGSEE
LOOSAFE
MIEBUL
MISECU
Nextrend
OEM
OLOEY
OUERTECH
QNTSQ
SACAM
SANNCE
SANSCO
SecTec
Shell film
Sifvision/sifsecurityvision
smar
SMTSEC
SSICON
SUNBA
Sunivision
Susikum
TECBOX
Techage
Techege
TianAnXun
TMEZON
TVPSii
Unique Vision
unitoptek
USAFEQLO
VOLDRELI
Westmile
Westshine
Wistino
Witrue
WNK Security Technology
WOFEA
WOSHIJIA
WUSONLUSAN
XIAO MA
XinAnX
xloongx
YiiSPO
YUCHENG
YUNSYE
zclever
zilnk
ZJUXIN
zmodo
ZRHUNTER

From time to time, there emerge cybersecurity stories of such potential impact that they have the effect of making all other security concerns seem minuscule and trifling by comparison. Yesterday was one of those times. Bloomberg Businessweek on Thursday published a bombshell investigation alleging that Chinese cyber spies had used a U.S.-based tech firm to secretly embed tiny computer chips into electronic devices purchased and used by almost 30 different companies. There aren’t any corroborating accounts of this scoop so far, but it is both fascinating and terrifying to look at why threats to the global technology supply chain can be so difficult to detect, verify and counter.

In the context of computer and Internet security, supply chain security refers to the challenge of validating that a given piece of electronics — and by extension the software that powers those computing parts — does not include any extraneous or fraudulent components beyond what was specified by the company that paid for the production of said item.

In a nutshell, the Bloomberg story claims that San Jose, Calif. based tech giant Supermicro was somehow caught up in a plan to quietly insert a rice-sized computer chip on the circuit boards that get put into a variety of servers and electronic components purchased by major vendors, allegedly including Amazon and Apple. The chips were alleged to have spied on users of the devices and sent unspecified data back to the Chinese military.

It’s critical to note up top that Amazon, Apple and Supermicro have categorically denied most of the claims in the Bloomberg piece. That is, their positions refuting core components of the story would appear to leave little wiggle room for future backtracking on those statements. Amazon also penned a blog post that more emphatically stated their objections to the Bloomberg piece.

Nevertheless, Bloomberg reporters write that “the companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation.”

The story continues:

Today, Supermicro sells more server motherboards than almost anyone else. It also dominates the $1 billion market for boards used in special-purpose computers, from MRI machines to weapons systems. Its motherboards can be found in made-to-order server setups at banks, hedge funds, cloud computing providers, and web-hosting services, among other places. Supermicro has assembly facilities in California, the Netherlands, and Taiwan, but its motherboards—its core product—are nearly all manufactured by contractors in China.

Many readers have asked for my take on this piece. I heard similar allegations earlier this year about Supermicro and tried mightily to verify them but could not. That in itself should be zero gauge of the story’s potential merit. After all, I am just one guy, whereas this is the type of scoop that usually takes entire portions of a newsroom to research, report and vet. By Bloomberg’s own account, the story took more than a year to report and write, and cites 17 anonymous sources as confirming the activity.

Most of what I have to share here is based on conversations with some clueful people over the years who would probably find themselves confined to a tiny, windowless room for an extended period if their names or quotes ever showed up in a story like this, so I will tread carefully around this subject.

The U.S. Government isn’t eager to admit it, but there has long been an unofficial inventory of tech components and vendors that are forbidden to buy from if you’re in charge of procuring products or services on behalf of the U.S. Government. Call it the “brown list, “black list,” “entity list” or what have you, but it’s basically an indelible index of companies that are on the permanent Shit List of Uncle Sam for having been caught pulling some kind of supply chain shenanigans.

More than a decade ago when I was a reporter with The Washington Post, I heard from an extremely well-placed source that one Chinese tech company had made it onto Uncle Sam’s entity list because they sold a custom hardware component for many Internet-enabled printers that secretly made a copy of every document or image sent to the printer and forwarded that to a server allegedly controlled by hackers aligned with the Chinese government.

That example gives a whole new meaning to the term “supply chain,” doesn’t it? If Bloomberg’s reporting is accurate, that’s more or less what we’re dealing with here in Supermicro as well.

But here’s the thing: Even if you identify which technology vendors are guilty of supply-chain hacks, it can be difficult to enforce their banishment from the procurement chain. One reason is that it is often tough to tell from the brand name of a given gizmo who actually makes all the multifarious components that go into any one electronic device sold today.

Take, for instance, the problem right now with insecure Internet of Things (IoT) devices — cheapo security cameras, Internet routers and digital video recorders — sold at places like Amazon and Walmart. Many of these IoT devices have become a major security problem because they are massively insecure by default and difficult if not also impractical to secure after they are sold and put into use.

For every company in China that produces these IoT devices, there are dozens of “white label” firms that market and/or sell the core electronic components as their own. So while security researchers might identify a set of security holes in IoT products made by one company whose products are white labeled by others, actually informing consumers about which third-party products include those vulnerabilities can be extremely challenging. In some cases, a technology vendor responsible for some part of this mess may simply go out of business or close its doors and re-emerge under different names and managers.

Mind you, there is no indication anyone is purposefully engineering so many of these IoT products to be insecure; a more likely explanation is that building in more security tends to make devices considerably more expensive and slower to market. In many cases, their insecurity stems from a combination of factors: They ship with every imaginable feature turned on by default; they bundle outdated software and firmware components; and their default settings are difficult or impossible for users to change.

We don’t often hear about intentional efforts to subvert the security of the technology supply chain simply because these incidents tend to get quickly classified by the military when they are discovered. But the U.S. Congress has held multiple hearings about supply chain security challenges, and the U.S. government has taken steps on several occasions to block Chinese tech companies from doing business with the federal government and/or U.S.-based firms.

Most recently, the Pentagon banned the sale of Chinese-made ZTE and Huawei phones on military bases, according to a Defense Department directive that cites security risks posed by the devices. The U.S. Department of Commerce also has instituted a seven-year export restriction for ZTE, resulting in a ban on U.S. component makers selling to ZTE.

Still, the issue here isn’t that we can’t trust technology products made in China. Indeed there are numerous examples of other countries — including the United States and its allies — slipping their own “backdoors” into hardware and software products.

Like it or not, the vast majority of electronics are made in China, and this is unlikely to change anytime soon. The central issue is that we don’t have any other choice right now. The reason is that by nearly all accounts it would be punishingly expensive to replicate that manufacturing process here in the United States.

Even if the U.S. government and Silicon Valley somehow mustered the funding and political will to do that, insisting that products sold to U.S. consumers or the U.S. government be made only with components made here in the U.S.A. would massively drive up the cost of all forms of technology. Consumers would almost certainly balk at buying these way more expensive devices. Years of experience has shown that consumers aren’t interested in paying a huge premium for security when a comparable product with the features they want is available much more cheaply.

Indeed, noted security expert Bruce Schneier calls supply-chain security “an insurmountably hard problem.”

“Our IT industry is inexorably international, and anyone involved in the process can subvert the security of the end product,” Schneier wrote in an opinion piece published earlier this year in The Washington Post. “No one wants to even think about a US-only anything; prices would multiply many times over. We cannot trust anyone, yet we have no choice but to trust everyone. No one is ready for the costs that solving this would entail.”

The Bloomberg piece also addresses this elephant in the room:

“The problem under discussion wasn’t just technological. It spoke to decisions made decades ago to send advanced production work to Southeast Asia. In the intervening years, low-cost Chinese manufacturing had come to underpin the business models of many of America’s largest technology companies. Early on, Apple, for instance, made many of its most sophisticated electronics domestically. Then in 1992, it closed a state-of-the-art plant for motherboard and computer assembly in Fremont, Calif., and sent much of that work overseas.

Over the decades, the security of the supply chain became an article of faith despite repeated warnings by Western officials. A belief formed that China was unlikely to jeopardize its position as workshop to the world by letting its spies meddle in its factories. That left the decision about where to build commercial systems resting largely on where capacity was greatest and cheapest. “You end up with a classic Satan’s bargain,” one former U.S. official says. “You can have less supply than you want and guarantee it’s secure, or you can have the supply you need, but there will be risk. Every organization has accepted the second proposition.”

Another huge challenge of securing the technology supply chain is that it’s quite time consuming and expensive to detect when products may have been intentionally compromised during some part of the manufacturing process. Your typical motherboard of the kind produced by a company like Supermicro can include hundreds of chips, but it only takes one hinky chip to subvert the security of the entire product.

Also, most of the U.S. government’s efforts to police the global technology supply chain seem to be focused on preventing counterfeits — not finding secretly added spying components.

Finally, it’s not clear that private industry is up to the job, either. At least not yet.

“In the three years since the briefing in McLean, no commercially viable way to detect attacks like the one on Supermicro’s motherboards has emerged—or has looked likely to emerge,” the Bloomberg story concludes. “Few companies have the resources of Apple and Amazon, and it took some luck even for them to spot the problem. ‘This stuff is at the cutting edge of the cutting edge, and there is no easy technological solution,’ one of the people present in McLean says. ‘You have to invest in things that the world wants. You cannot invest in things that the world is not ready to accept yet.'”

For my part, I try not to spin my wheels worrying about things I can’t change, and the supply chain challenges definitely fit into that category. I’ll have some more thoughts on the supply chain problem and what we can do about it in an interview to be published next week.

But for the time being, there are some things worth thinking about that can help mitigate the threat from stealthy supply chain hacks. Writing for this week’s newsletter put out by the SANS Institute, a security training company based in Bethesda, Md., editorial board member William Hugh Murray has a few provocative thoughts:

1. Abandon the password for all but trivial applications. Steve Jobs and the ubiquitous mobile computer have lowered the cost and improved the convenience of strong authentication enough to overcome all arguments against it.
2. Abandon the flat network. Secure and trusted communication now trump ease of any-to-any communication.
3. Move traffic monitoring from encouraged to essential.
4. Establish and maintain end-to-end encryption for all applications. Think TLS, VPNs, VLANs and physically segmented networks. Software Defined Networks put this within the budget of most enterprises.
5. Abandon the convenient but dangerously permissive default access control rule of “read/write/execute” in favor of restrictive “read/execute-only” or even better, “Least privilege.” Least privilege is expensive to administer but it is effective. Our current strategy of “ship low-quality early/patch late” is proving to be ineffective and more expensive in maintenance and breaches than we could ever have imagined.

A ridiculous number of companies are exposing some or all of their proprietary and customer data by putting it in the cloud without any kind of authentication needed to read, alter or destroy it. When cybercriminals are the first to discover these missteps, usually the outcome is a demand for money in return for the stolen data. But when these screw-ups are unearthed by security professionals seeking to make a name for themselves, the resulting publicity often can leave the breached organization wishing they’d instead been quietly extorted by anonymous crooks.

Last week, I was on a train from New York to Washington, D.C. when I received a phone call from Vinny Troia, a security researcher who runs a startup in Missouri called NightLion Security. Troia had discovered that All American Entertainment, a speaker bureau which represents a number of celebrities who also can be hired to do public speaking, had exposed thousands of speaking contracts via an unsecured Amazon cloud instance.

The contracts laid out how much each speaker makes per event, details about their travel arrangements, and any requirements or obligations stated in advance by both parties to the contract. No secret access or password was needed to view the documents.

It was a juicy find to be sure: I can now tell you how much Oprah makes per event (it’s a lot). Ditto for Gwyneth Paltrow, Olivia Newton John, Michael J. Fox and a host of others. But I’m not going to do that.

Firstly, it’s nobody’s business what they make. More to the point, All American also is my speaker bureau, and included in the cache of documents the company exposed in the cloud were some of my speaking contracts. In fact, when Troia called about his find, I was on my way home from one such engagement.

I quickly informed my contact at All American and asked them to let me know the moment they confirmed the data was removed from the Internet. While awaiting that confirmation, my pent-up frustration seeped into a tweet that seemed to touch a raw nerve among others in the security industry.

The same day I alerted them, All American took down its bucket of unsecured speaker contract data, and apologized profusely for the oversight (although I have yet to hear a good explanation as to why this data needed to be stored in the cloud to begin with).

This was hardly the first time Troia had alerted me about a huge cache of important or sensitive data that companies have left exposed online. On Monday, TechCrunch broke the story about a “breach” at Apollo, a sales engagement startup boasting a database of more than 200 million contact records. Calling it a breach seems a bit of a stretch; it probably would be more accurate to describe the incident as a data leak.

Just like my speaker bureau, Apollo had simply put all this data up on an Amazon server that anyone on the Internet could access without providing a password. And Troia was again the one who figured out that the data had been leaked by Apollo — the result of an intensive, months-long process that took some extremely interesting twists and turns.

That journey — which I will endeavor to describe here — offered some uncomfortable insights into how organizations frequently learn about data leaks these days, and indeed whether they derive any lasting security lessons from the experience at all. It also gave me a new appreciation for how difficult it can be for organizations that screw up this way to tell the difference between a security researcher and a bad guy.

THE DARK OVERLORD

I began hearing from Troia almost daily beginning in mid-2017. At the time, he was on something of a personal mission to discover the real-life identity behind The Dark Overlord (TDO), the pseudonym used by an individual or group of criminals who have been extorting dozens of companies — particularly healthcare providers — after hacking into their systems and stealing sensitive data.

The Dark Overlord’s method was roughly the same in each attack. Gain access to sensitive data (often by purchasing access through crimeware-as-a-service offerings), and send a long, rambling ransom note to the victim organization demanding tens of thousands of dollars in Bitcoin for the safe return of said data.

Victims were typically told that if they refused to pay, the stolen data would be sold to cybercriminals lurking on Dark Web forums. Worse yet, TDO also promised to make sure the news media knew that victim organizations were more interested in keeping the breach private than in securing the privacy of their customers or patients.

In fact, the apparent ringleader of TDO reached out to KrebsOnSecurity in May 2016 with a remarkable offer. Using the nickname “Arnie,” the public voice of TDO said he was offering exclusive access to news about their latest extortion targets.

Arnie claimed he was an administrator or key member on several top Dark Web forums, and provided a handful of convincing clues to back up his claim. He told me he had real-time access to dozens of healthcare organizations they’d hacked into, and that each one which refused to give in to TDO’s extortion demands could turn into a juicy scoop for KrebsOnSecurity.

Arnie said he was coming to me first with the offer, but that he was planning to approach other journalists and news outlets if I declined. I balked after discovering that Arnie wasn’t offering this access for free: He wanted 10 bitcoin in exchange for exclusivity (at the time, his asking price was roughly equivalent to USD $5,000).

Perhaps other news outlets are accustomed to paying for scoops, but that is not something I would ever consider. And in any case the whole thing was starting to smell like a shakedown or scam. I declined the offer. It’s possible other news outlets or journalists did not; I will not speculate on this matter further, other than to say readers can draw their own conclusions based on the timeline and the public record.

WHO IS SOUNDCARD?

Fast-forward to September 2017, and Troia was contacting me almost daily to share tidbits of research into email addresses, phone numbers and other bits of data apparently tied to TDO’s communications with victims and their various identities on Dark Web forums.

His research was exhaustive and occasionally impressive, and for a while I caught the TDO bug and became engaged in a concurrent effort to learn the identities of the TDO members. For better or worse, the results of that research will have to wait for another story and another time.

At one point, Troia told me he’d gained acceptance on the Dark Web forum Kickass, using the hacker nickname “Soundcard“. He said he believed a presence on all of the forums TDO was active on was necessary for figuring out once and for all who was behind this brazen and very busy extortion group.

Here is a screen shot Troia shared with me of Soundcard’s posting there, which concerned a July 2018 forum discussion thread about a data leak of 340 million records from Florida-based marketing firm Exactis. As detailed by Wired.com in June 2018, Troia had discovered this huge cache of data unprotected and sitting wide open on a cloud server, and ultimately traced it back to Exactis.

After several weeks of comparing notes about TDO with Troia, I learned that he was telling random people that we were “working together,” and that he was throwing my name around to various security industry sources and friends as a way of gaining access to new sources of data.

I respectfully told Troia that this was not okay — that I never told people about our private conversations (or indeed that we spoke at all) — and I asked him to stop doing that. He apologized, said he didn’t understand he’d overstepped certain boundaries, and that it would never happen again.

But it would. Multiple times. Here’s one time that really stood out for me. Earlier this summer, Troia sent me a link to a database of truly staggering size — nearly 10 terabytes of data — that someone had left open to anyone via a cloud instance. Again, no authentication or password was needed to access the information.

At first glance, it appeared to be LinkedIn profile data. Working off that assumption, I began a hard target search of the database for specific LinkedIn profiles of important people. I first used the Web to locate the public LinkedIn profile pages for nearly all of the CEOs of the world’s top 20 largest companies, and then searched those profile names in the database that Troia had discovered.

Suddenly, I had the cell phone numbers, addresses, email addresses and other contact data for some of the most powerful people in the world. Immediately, I reached out to contacts at LinkedIn and Microsoft (which bought LinkedIn in 2016) and arranged a call to discuss the findings.

LinkedIn’s security team told me the data I was looking at was in fact an amalgamation of information scraped from LinkedIn and dozens of public sources, and being sold by the same firm that was doing the scraping and profile collating. LinkedIn declined to name that company, and it has not yet responded to follow-up questions about whether the company it was referring to was Apollo.

Sure enough, a closer inspection of the database revealed the presence of other public data sources, including startup web site AngelList, Facebook, Salesforce, Twitter, and Yelp, among others.

Several other trusted sources I approached with samples of data spliced from the nearly 10 TB trove of data Troia found in the cloud said they believed LinkedIn’s explanation, and that the data appeared to have been scraped off the public Internet from a variety of sources and combined into a single database.

I told Troia it didn’t look like the data came exclusively from LinkedIn, or at least wasn’t stolen from them, and that all indications suggested it was a collection of data scraped from public profiles. He seemed unconvinced.

Several days after my second call with LinkedIn’s security team — around Aug. 15 — I was made aware of a sales posting on the Kickass crime forum by someone selling what they claimed was “all of the LinkedIN user-base.” The ad, a blurry, partial screenshot of which can be seen below, was posted by the Kickass user Soundcard. The text of the sales thread was as follows:

“KA users –

I present you with exclusive opportunity to purchase all (yes ALL) of the LinkedIN user-base for the low low price of 2 BTC.

I found a database server with all LinkedIN users. All of user’s personal information is included in this database (including private email and phone number NOT listed on public profile). No passwords, sorry.

Size: 2.1TB.

user count: 212 million

Why so large for 212 million users? See the sample data per record. There is lot of marketing and CRM data as well. I sell original data only. no editz.

Here is index of server. The LinkedIN users spread across people and contacts indexes. Sale includes both of those indexes.

Questions, comments, purchase? DM me, or message me – soundcard@exploit[.]im

The “sample data” included in the sales thread was from my records in this huge database, although Soundcard said he had sanitized certain data elements from this snippet. He explained his reasoning for that in a short Q&A from his sales thread:

Question 1: Why you sanitize Brian Krebs’ information in sample?

Answer 1: Because nothing in life free. This only to show i have data.

I soon confronted Troia not only for offering to sell leaked data on the Dark Web, but also for once again throwing my name around in his various activities — despite past assurances that he would not. Also, his actions had boxed me into a corner: Any plans I had to credit him in a story for eventually helping to determine the source of the leaked data (which we now know to be Apollo) became more complicated without also explaining his Dark Web alter ego as Soundcard, and I am not in the habit of omitting such important details from stories.

Troia assured me that he never had any intention of selling the data, and that the whole thing had been a ruse to help smoke out some of the suspected TDO members.

For its part, LinkedIn’s security team was not amused, and published a short post to its media page denying that the company had suffered a security breach.

“We want our members to know that a recent claim of a LinkedIn data breach is not accurate,” the company wrote. “Our investigation into this claim found that a third-party sales intelligence company that is not associated with LinkedIn was compromised and exposed a large set of data aggregated from a number of social networks, websites, and the company’s own customers. It also included a limited set of publicly available data about LinkedIn members, such as profile URL, industry and number of connections. This was not a breach of LinkedIn.”

It is quite a fine line to walk when self-styled security researchers mimic cyber criminals in the name of making things more secure. On the one hand, reaching out to companies that are inadvertently exposing sensitive data and getting them to secure it or pull it offline altogether is a worthwhile and often thankless effort, and clearly many organizations still need a lot of help in this regard.

On the other hand, most organizations that fit this description simply lack the security maturity to tell the difference between someone trying to make the Internet a safer place and someone trying to sell them a product or service.

As a result, victim organizations tend to react with deep suspicion or even hostility to legitimate researchers and security journalists who alert them about a data breach or leak. And stunts like the ones described above tend to have the effect of deepening that suspicion, and sowing fear, uncertainty and doubt about the security industry as a whole.

Most of us have been trained to be wary of clicking on links and attachments that arrive in emails unexpected, but it’s easy to forget scam artists are constantly dreaming up innovations that put a new shine on old-fashioned telephone-based phishing scams. Think you’re too smart to fall for one? Think again: Even technology experts are getting taken in by some of the more recent schemes (or very nearly).

Matt Haughey is the creator of the community Weblog MetaFilter and a writer at Slack. Haughey banks at a small Portland credit union, and last week he got a call on his mobile phone from an 800-number that matched the number his credit union uses.

Actually, he got three calls from the same number in rapid succession. He ignored the first two, letting them both go to voicemail. But he picked up on the third call, thinking it must be something urgent and important. After all, his credit union had rarely ever called him.

Haughey said he was greeted by a female voice who explained that the credit union had blocked two phony-looking charges in Ohio made to his debit/ATM card. She proceeded to then read him the last four digits of the card that was currently in his wallet. It checked out.

Haughey told the lady that he would need a replacement card immediately because he was about to travel out of state to California. Without missing a beat, the caller said he could keep his card and that the credit union would simply block any future charges that weren’t made in either Oregon or California.

This struck Haughey as a bit off. Why would the bank say they were freezing his card but then say they could keep it open for his upcoming trip? It was the first time the voice inside his head spoke up and said, “Something isn’t right, Matt.” But, he figured, the customer service person at the credit union was trying to be helpful: She was doing him a favor, he reasoned.

The caller then read his entire home address to double check it was the correct destination to send a new card at the conclusion of his trip. Then the caller said she needed to verify his mother’s maiden name. The voice in his head spoke out in protest again, but then banks had asked for this in the past. He provided it.

Next she asked him to verify the three digit security code printed on the back of his card. Once more, the voice of caution in his brain was silenced: He’d given this code out previously in the few times he’d used his card to pay for something over the phone.

Then she asked him for his current card PIN, just so she could apply that same PIN to the new card being mailed out, she assured him. Ding, ding, ding went the alarm bells in his head. Haughey hesitated, then asked the lady to repeat the question. When she did, he gave her the PIN, and she assured him she’d make sure his existing PIN also served as the PIN for his new card.

Haughey said after hanging up he felt fairly certain the entire transaction was legitimate, although the part about her requesting the PIN kept nagging at him.

“I balked at challenging her because everything lined up,” he said in an interview with KrebsOnSecurity. “But when I hung up the phone and told a friend about it, he was like, ‘Oh man, you just got scammed, there’s no way that’s real.'”

Now more concerned, Haughey visited his credit union to make sure his travel arrangements were set. When he began telling the bank employee what had transpired, he could tell by the look on her face that his friend was right.

A review of his account showed that there were indeed two fraudulent charges on his account from earlier that day totaling $3,400, but neither charge was from Ohio. Rather, someone used a counterfeit copy of his debit card to spend more than $2,900 at a Kroger near Atlanta, and to withdraw almost $500 from an ATM in the same area. After the unauthorized charges, he had just $300 remaining in his account.

“People I’ve talked to about this say there’s no way they’d fall for that, but when someone from a trustworthy number calls, says they’re from your small town bank, and sounds incredibly professional, you’d fall for it, too,” Haughey said.

Fraudsters can use a variety of open-source and free tools to fake or “spoof” the number displayed as the caller ID, lending legitimacy to phone phishing schemes. Often, just sprinkling in a little foreknowledge of the target’s personal details — SSNs, dates of birth, addresses and other information that can be purchased for a nominal fee from any one of several underground sites that sell such data — adds enough detail to the call to make it seem legitimate.

A CLOSE CALL

Cabel Sasser is founder of a Mac and iOS software company called Panic Inc. Sasser said he almost got scammed recently after receiving a call that appeared to be the same number as the one displayed on the back of his Wells Fargo ATM card.

“I answered, and a Fraud Department agent said my ATM card has just been used at a Target in Minnesota, was I on vacation?” Sasser recalled in a tweet about the experience.

What Sasser didn’t mention in his tweet was that his corporate debit card had just been hit with two instances of fraud: Someone had charged $10,000 worth of metal air ducts to his card. When he disputed the charge, his bank sent a replacement card.

“I used the new card at maybe four places and immediately another fraud charge popped up for like $20,000 in custom bathtubs,” Sasser recalled in an interview with KrebsOnSecurity. “The morning this scam call came in I was spending time trying to figure out who might have lost our card data and was already in that frame of mind when I got the call about fraud on my card.”

And so the card-replacement dance began.

“Is the card in your possession?,” the caller asked. It was. The agent then asked him to read the three-digit CVV code printed on the back of his card.

After verifying the CVV, the agent offered to expedite a replacement, Sasser said. “First he had to read some disclosures. Then he asked me to key in a new PIN. I picked a random PIN and entered it. Verified it again. Then he asked me to key in my current PIN.”

That made Sasser pause. Wouldn’t an actual representative from Wells Fargo’s fraud division already have access to his current PIN?

“It’s just to confirm the change,” the caller told him. “I can’t see what you enter.”

“But…you’re the bank,” he countered. “You have my PIN, and you can see what I enter…”

The caller had a snappy reply for this retort as well.

“Only the IVR [interactive voice response] system can see it,” the caller assured him. “Hey, if it helps, I have all of your account info up…to confirm, the last four digits of your Social Security number are XXXX, right?”

Sure enough, that was correct. But something still seemed off. At this point, Sasser said he told the agent he would call back by dialing the number printed on his ATM card — the same number his mobile phone was already displaying as the source of the call. After doing just that, the representative who answered said there had been no such fraud detected on his account.

“I was just four key presses away from having all my cash drained by someone at an ATM,” Sasser recalled. A visit to the local Wells Fargo branch before his trip confirmed that he’d dodged a bullet.

“The Wells person was super surprised that I bailed out when I did, and said most people are 100 percent taken by this scam,” Sasser said.

HUMAN, ROBOT OR HYBRID?

In Sasser’s case, the scammer was a live person, but some equally convincing voice phishing schemes — sometimes called “vishing” — use a combination of humans and automation. Consider the following vishing attempt, reported to KrebsOnSecurity in August by “Curt,” a longtime reader from Canada.

“I’m both a TD customer and Rogers phone subscriber and just experienced what I consider a very convincing and/or elaborate social engineering/vishing attempt,” Curt wrote. “At 7:46pm I received a call from (647-475-1636) purporting to be from Credit Alert (alertservice.ca) on behalf of TD Canada Trust offering me a free 30-day trial for a credit monitoring service.”

The caller said her name was Jen Hansen, and began the call with what Curt described as “over-the-top courtesy.”

“It sounded like a very well-scripted Customer Service call, where they seem to be trying so hard to please that it seems disingenuous,” Curt recalled. “But honestly it still sounded very much like a real person, not like a text to speech voice which sounds robotic. This sounded VERY natural.”

Ms. Hansen proceeded to tell Curt that TD Bank was offering a credit monitoring service free for one month, and that he could cancel at any time. To enroll, he only needed to confirm his home mailing address.

“I’m mega paranoid (I read krebsonsecurity.com daily) and asked her to tell me what address I had on their file, knowing full well my home address can be found in a variety of ways,” Curt wrote in an email to this author. “She said, ‘One moment while I access that information.'”

After a short pause, a new voice came on the line.

“And here’s where I realized I was finally talking to a real human — a female with a slight French accent — who read me my correct address,” Curt recalled.

After another pause, Ms. Hansen’s voice came back on the line. While she was explaining that part of the package included free antivirus and anti-keylogging software, Curt asked her if he could opt-in to receive his credit reports while opting-out of installing the software.

“I’m sorry, can you repeat that?” the voice identifying itself as Ms. Hansen replied. Curt repeated himself. After another, “I’m sorry, can you repeat that,” Curt asked Ms. Hansen where she was from.

The voice confirmed what was indicated by the number displayed on his caller ID: That she was calling from Barrie, Ontario. Trying to throw the robot voice further off-script, Curt asked what the weather was like in Barrie, Ontario. Another Long pause. The voice continued describing the offered service.

“I asked again about the weather, and she said, ‘I’m sorry, I don’t have that information. Would you like me to transfer you to someone that does?’ I said yes and again the real person with a French accent started speaking, ignoring my question about the weather and saying that if I’d like to continue with the offer I needed to provide my date of birth. This is when I hung up and immediately called TD Bank.” No one from TD had called him, they assured him.

FULLY AUTOMATED PHONE PHISHING

And then there are the fully-automated voice phishing scams, which can be be equally convincing. Last week I heard from “Jon,” a cybersecurity professional with more than 30 years of experience under his belt (Jon asked to leave his last name out of this story).

Answering a call on his mobile device from a phone number in Missouri, Jon was greeted with the familiar four-note AT&T jingle, followed by a recorded voice saying AT&T was calling to prevent his phone service from being suspended for non-payment.

“It then prompted me to enter my security PIN to be connected to a billing department representative,” Jon said. “My number was originally an AT&T number (it reports as Cingular Wireless) but I have been on T-Mobile for several years, so clearly a scam if I had any doubt. However, I suspect that the average Joe would fall for it.”

WHAT CAN YOU DO?

Just as you would never give out personal information if asked to do so via email, never give out any information about yourself in response to an unsolicited phone call.

Phone phishing, like email scams, usually invokes an element of urgency in a bid to get people to let their guard down. If call has you worried that there might be something wrong and you wish to call them back, don’t call the number offered to you by the caller. If you want to reach your bank, call the number on the back of your card. If it’s another company you do business with, go to the company’s site and look up their main customer support number.

Unfortunately, this may take a little work. It’s not just banks and phone companies that are being impersonated by fraudsters. Reports on social media suggest many consumers also are receiving voice phishing scams that spoof customer support numbers at Apple, Amazon and other big-name tech companies. In many cases, the scammers are polluting top search engine results with phony 800-numbers for customer support lines that lead directly to fraudsters.

These days, scam calls happen on my mobile so often that I almost never answer my phone unless it appears to come from someone in my contacts list. The Federal Trade Commission’s do-not-call list does not appear to have done anything to block scam callers, and the major wireless carriers seem to be pretty useless in blocking incessant robocalls, even when the scammers are impersonating the carriers themselves, as in Jon’s case above.

I suspect people my age (mid-40s) and younger also generally let most unrecognized calls go to voicemail. It seems to be a very different reality for folks from an older generation, many of whom still primarily call friends and family using land lines, and who will always answer a ringing phone whenever it is humanly possible to do so.

It’s a good idea to advise your loved ones to ignore calls unless they appear to come from a friend or family member, and to just hang up the moment the caller starts asking for personal information.