> Nested on the servers’ motherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design.
> During the ensuing top-secret probe, which remains open more than three years later, investigators determined that the chips allowed the attackers to create a stealth doorway into any network that included the altered machines. Multiple people familiar with the matter say investigators found that the chips had been inserted at factories run by manufacturing subcontractors in China.
Is there an article that describes a bit more in detail what the chips actually did (or were capable of doing)? They only say "the microchip altered the operating system’s core so it could accept modifications.", which I might interpret as circumventing signature checks to allow installing modified firmware on the systems? But how does the chip connect to the network and how does it receive commands?
That said, it's pretty scary that you can hide so much malicious functionality in such a small device, makes me wonder what might be hidden in my Lenovo. In any case it speaks highly of the auditing firm that they were able to locate this. I wonder if they performed an x-ray analysis of the board, as given the size of these chips it should be possible to embed such devices in one of the internal layers of the board as well, making them essentially invisible to optical inspection.
SuperMicro hardware has very extensive IPMI integration into the mothebroard, which amongst other things can take over and inject frames into the network interface, emulate a VGA device, talk to the CPUs serial lines directly, flash firmware, control the state of a number of physical devices- and this is what it supports just from the web interface it presents by default with the password "ADMIN:ADMIN". My money, based on experience attempting to harden their devices, is that any modification were injected into the IPMI hardware where most of this was already supported.
This stuff ends up being extremely difficult to disable. The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC invisibly to the OS and DHCPs on its own to open up an administration port, web interface, and some assorted call homes. You have to explicitly tell it to use the non-connected port, change credentials, and modify it so that it is not accessible within operating system as well. Hopefully while the machine is offline to prevented any automated scanning finding it within your network.
The number of times I'd end up nmaping our local networks and being able to remotely access production hardware with an interface that allowed me to reach this interface was maddening. The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely nu-authenticated avenues exposed. The course of action was always just to write off the hardware and bin it, because god knows what impact you could actually have using that interface.
> The organizations behind the new project each have already made substantial contributions to creating open source baseboard management controller (BMC) firmware. Now, working together, they will define the vision for a standard stack that can be used across systems and computing environments.
LinuxBoot and Open Compute OSF are working on open-source server firmware that can be measured on every boot and validated against hardware root of trust keys controlled by the server owner instead of the server OEM, https://www.platformsecuritysummit.com/2018/speaker/hudson/
Wouldn't help. The BMC hardware has direct serial access to CPUs and other hardware in the machine. Communication is unencrypted. A hardware modification attack wouldn't touch the firmware at all and could still compromise IPMI functionality.
Even now, supply chain hardware modification attacks remind people of fiction. However, the number of people known to be affected by buggy BMC firmware is orders of magnitude larger, as described two comments upthread:
> The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely un-authenticated avenues exposed.
SuperMicro hardware in particular always struck me as such. Asking users to pay a license fee to be able to update the BIOS on their devices (after paying tens of thousands for the hardware itself) is a kick in the teeth.
To be clear, this is only for the ability to update the BIOS over the BMC interface, not for BIOS updates in general. (unlike some other vendors, where you need a support contract to be even able to download the updates)
At least on some boards you can boot the USB drive image containing the BIOS updater through the BMC and do a remote update that way.
For the models I was looking at a number of years ago, the options were through IPMI or via a USB Floppy Drive. Perhaps my memory is failing me here, but I seem to remember being quite enraged at the prospect and would have done a lot not to have to pay the license fee on principle.
Supermicro doesn't publish changelogs for their BIOSes. Their disclaimer on their BIOS download page says, "don't update the BIOS unless you have to". It's a ridiculous stance for a server company. You buy Supermicro over Dell for the price. That's pretty much it.
This is only really valid for protocols or products designed before the Morris worm of 1988. Anything designed beyond 2000 has no excuse for not thinking about internet security.
Well, IPMI isn't supposed to be exposed to the internet. Best practices have you running your BMC's on a completely separate, highly locked down administrative network.
Well yes, typically you'd 1) configure IPMI to use the dedicated port, 2) put those ports into a VLAN with no outgoing internet access. But since this is BMC, what's stopping it from just using your management or production port to fire off its secret phone-homes and whatnot?
Best practice would be for the BMC to not have access to the regular network ports when it has a dedicated network jack. All the ones I've looked at don't have any kind of physical interlock or switch, it's a software interlock.
Now even a hardware interlock could be subverted, but that's harder than sticking code in the bmc firmware, which does tend to get updated during the life of a server.
I did a security audit on a company that had a setup similar to this.
After popping an internet facing web server, I was able to compromise the IPMI system and use the management network to bounce around to any server in the enterprise completely bypassing all the firewalls and segmentation on the production network.
No. SuperMicro make some models which appear to be for government application which lack the ASIC completely, they were available on NewEgg with no further description of purpose at one point.
No, the controller runs it's own totally separate OS, has connections to most/all the buses and is able to emulate devices whether you have drivers installed or not.
No, it boots prior/separate to the board itself. It's basically a mini-PC embedded in the board that has it's own CPU/Memory and tentacles attached to everything in the mainboard.
If people are interested in digging into how IPMI works every SuperMicro board I've ever seen uses some variety of ASPEED Baseboard Management Controller (BMC). Facebook uses the same chip in their open hardware projects.
The problem with an IPMI BMC is that if you have malware that roots the OS from the BMC that in turn roots the BMC to reinstall itself from the OS, then you can never actually get rid of it. I actually opened a bug with a major clustering vendor in about 2012 because of this. Their response was a docfix.
> The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC
cool, thanks for that info.
> just to write off the hardware
maybe you could just standup the mgmt network but blackhole route it at each switch port. The mgmt NIC thinks it's working properly but it can't talk to anyone nor can anyone talk to it.
5. When a server was installed and
switched on, the microchip altered
the operating system’s core so it
could accept modifications. The chip
could also contact computers controlled
by the attackers in search of further
instructions and code.
So, in typical vulnerability/payload/exploit fashion, the board's bus is vulnerable by default, because the chip pierces all the usual lines of defense protecting against network and operator I/O. It carries a payload intended to target very common features used everywhere commodity servers are used, one that likely listens for DMA traffic on the bus, and alters the signal stream, by escaping upon the occurrence of a magic sequence, and inserting its own signal, before resuming the authentic stream in flight.
The payload could be pretty small, since the server boards are likely using OS packages that match the chipset. This limits the software to a small set of well known targets, Linux, Windows, Apple. Target their kernels, and you only have to snip out a small chunk of bytes, and splice your own pre-defined package in. Splice in a miniature runtime, that operates a turing complete set of operations, and open up a listener that waits for network access, and now, your payload can enable arbitrary code execution, irrespective of permissions.
Now, to exploit, the payload needs to time the opportunity to splice itself onto the disk correctly. If certain well-known chunks of code will always exist in each given operating system, then with every disk access event one just needs to wait for the inevitable moment those magic system-specific bytes travel over the bus, in order to replace the known bytes with the poisoned modification. Events might target when the bytes are originally installed with the OS, or every time the OS reads those known bytes back into live memory, from any source.
The total payload package could probably fit inside a couple of megabytes, pack on a few more for the "listen & splice" part of the attack to round out the entire mass, and all we know how much data an SD card can fit into say... five grains of rice?
It's not particularly magical, there's consumer chips around which are not a whole lot bigger (though obviously in a more standard package). You don't get a lot of resources, but you don't really need it if all the other frameworks are in place in other software. If this sort of thing is something you can buy on Mouser for a few cents, the espionage grade material is probably an order or magnitude more higher quality.
For example, looking for ELF or Portable Executable headers, as a crude estimate to determine attack opportunities. In this case, the magic numbers would probably be more selective and sophisticated, but still have an aspect of hard-coded values, since we're talking custom silicon.
This sounds like speculation. I'm quite capable of coming up with my own unfounded speculation, but there is a real report out there with the actual details in that really needs to be made public, legally or otherwise. There ought to be a CVE about this. Where is it?
That would make a lot of sense and would give the attacker a way to interface with all of the other hardware (network, disk etc.). Do you have a source for this information?
I looked up supermicro blade motherboards, and saw that the chip was right near the IPMI chip's line to spi flash.
And prior to that, there were already persistent rumors in the Chinese interney of certain Chinese mobos sending "weird garbage on ICMP," and "BMCs that somehow boot and work with their flash memory soldered off"
Remembering that, I might even suggest that this is not a modchip that does something with signal on the go, but just a very tiny flash chip that has the modded firmware.
Going further from that, to pack, say, 16 mB on a sandgrain sized chip, the densities need to be like that of best flash chips out there, which also means that they have access to last gen flash fab.
There really, really needs to be. If this story is real and things have been known compromised since 2015, it’s unacceptable that this information isn’t shared more widely. The “we’re going to follow them” idea doesn’t hold water over 3 years, countless small vendors using Supermicro and not being well-connected enough to know (like Apple or Amazon) about it.
The recovery overrides the primary if detected by default.
The place they put their "filter cap" is right on top the empty TSOP8 pad for the recovery flash. And they probably ordered the factory to sneak the traces just a little bit more, or put hidden vias under it, or simply had somebody very dexterous to solder it to pads with hair thin wires.
Well, it means that provision for the second flash was already there, and PLA simply exploited that fact that Aspeed chips are virtually omnipresent in higher end servers.
It also means that the extend of intervention into board design was minimal, and that a trivial automatic xray would not have picked it up. And as implied in the article, later they buried the bug to beat the AOI, if it was done higher upstream.
So, they would've been screwed even if they were doing board testing outside of China.
That's a clever trick.
But the sole fact that the chip has "to phone home" makes detection trivial, and puts the usefulness of the method to nil - anybody sees the router blink when it shouldn't and your bug's cover is blown.
This is the most plausible theory I've read in this thread. Assuming the image in the article is a stock image (there isn't yet a clear image of a definitely compromised board), then the added part could simply be another TSOP8 Flash part. This implies the firmware to the AST2400 is unsigned (which it appears to be, as there's coreboot options for it).
That makes the whole thing gloriously simple.
A part "stuck on" afterwards is obvious. A part fitted into a no-fit footprint after optical inspection is not, it looks exactly as if it was meant to be there.
Hm, but DMA messages get distributed over a parallel bus and this chip seems to employ a serial interface, so I would assume that it's not directly connected to anything that requires high throughput (i.e. memory, disk and peripheral access).
But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
To me, that makes it sound like they could download from a remote host and inject code and do literally anything.
If this is indeed the case, I'm surprised someone didn't catch something earlier when the device was calling "home" over their network. I'm wondering if China stole BlackRidge First Packet Authentication tech [1] to keep things dark. BlackRidge is... "involved" in IC and defense projects.
I take a different view of this. Some 30 companies got this hardware, two are named as having the resources to find the offending hardware. By this logic you’re still safer using a company such as Amazon.
My point is that you should care about privacy, even if you're using, for example, apple services. Thinking that nobody has the resources to hack a company that big isn't an argument and has just been disproven.
In other words, if you have plans to ever be a politician that china may not want in any position of power, don't store your nudes on icloud/dropbox/google drive, or they may suddenly get leaked when you least expect it and ruin your career in favor of a more... Shall we say "convenient"? ...alternative.
I’m not saying nobody has the resources to hack a company. I’m saying few companies have the resources to detect such a sophisticated hack. Therefore my infra is probably safer with, say, Amazon vs my local colo.
You are correct that individuals seeking personal infosec against state actors must be eternally vigilant.
The denials by Amazon, Apple, Supermicro and the Chinese Ministry of Foreign Affairs [1] are relatively pro forma, both directed by respective nation states involved in this matter. One of the reporters interviewed on Bloomberg noted Amazon and Apple could be directed by US national security interests to deny to protect the ongoing US investigation. Supermicro could similarly be directed by Chinese national security interests to protect plausible deniability.
There was a sense of realpolitik by one UK guest commentator on Bloomberg, comments along the lines of "hey, spying happens since time immemorial, put on some big boy pants, yes there is shock but not horror when the Snowden revelations came out, the US does it, too, etc.". I disagree with this sentiment, as while the attack was quite targeted, it puts into question a quite large supply chain network.
Kudos to Bloomberg putting in the 12+ month investigative journalism to pull off this scoop. Yet another validation of the reasons I subscribe to listening/watching them.
> Supermicro could similarly be directed by Chinese national security interests to protect plausible deniability.
Supermicro is an American corporation, headquartered in San Jose. They're not directed by Chinese national security interests, they'll do anything the US Government tells them to do when it comes to US national security.
Fascinating. A company I used to work for dismissed this sort of situation as a problem and ended up using SuperMicro boards in a security crucial product. Their hardware has always been notably very crappy, with the IPMI interface defaulting to world-open unsafe parameters, but I'd not expected it to be this cleverly hardware backdoored. It's possible to neuter Intel ME, but that's only a small comfort with these motherboards.
Who else found the design of the page made the article difficult to read? I know darkness, spies and hacking go hand-in-hand but it's a bit too much imho.
I know this is going off the main topic. But it is very strange that I discover Black background, as in Dark mode in macOS, doesn't work at all despite what I have always thought it would be cool. But Black background on mobile devices work absolutely great!.
And I have no idea why, I searched on Google and couldn't find any decent answer. All results were either OLED being battery friendly with switched off pixels or other mobile UX for Dark mode. None of them describe the difference felt in Desktop and Mobile. Any link or explanation would be much appreciated.
As a fan of dark color themes and displays, I've shared a bunch of thoughts related to this on my personal blog. But I'd like to share a few quick thoughts below. I am not an expect on these matters, so these are just my opinions:
1. Glare. When reading on a small device, the amount of glare reflected in the screen space occupied by black pixels may be fairly minimal depending on your reading environment and positioning. Also, although mobile devices use glossy display surfaces, they tend to have several anti-glare layers in the display stack. You said your computer is macOS, so I suspect you're unlucky enough to be reading on a glossy laptop display. With "dark mode" color schemes, your eyes can more easily see the reflected scene (maybe your office lights, an exterior window, or even your own face). And the focal length of that reflected scene is 2x your reading distance to the screen. That reflected scene at an extended focal length is more relaxing for your eyes to focus on. So in order to read, you need to fight your natural tendency to relax and look at the reflected scene.
If you are lucky enough to be reading on a matte desktop display (typically a professional or prosumer monitor, such as a Dell UltraSharp or LG 43MU79-B), the glare will be minimal and it should be much easier to read.
2. Pixel density. I contend that one reason dark themes have become more prevalent in recent history is thanks to wider adoption of high pixel density displays. At a legacy density of approximately 75 dots per inch, the stroke weight of bright text on a dark background appears too faint if the strokes render as just one pixel in width. Higher pixel density allows for the strokes of letters to be wider than a single pixel, allowing for greater clarity. If you ever designed a dark theme in the days of ~75 dpi displays, you might naturally favor bold text as the default because it was considerably easier to read. (Interesting sidebar: many console oriented bitmap fixed-width fonts historically used two pixels for stroke weight, especially in the horizontal dimension, presumably because they were designed to be used bright-on-dark.)
Now, you did again say you were reading on macOS, so your display's pixel density is probably higher than ~75 dpi. But a MacBook Pro is still only ~220 dpi. A Surface Book is ~260 dpi. A Dell XPS laptop with a 4K display will be a little higher still (maybe ~300 dpi). But many phones are using 450+ dpi displays. The stroke weight of a character on a mobile device is several pixels wide, so it's highly defined and clear.
3. Font selection (related to above). Bloomberg has selected a serif font, presumably because they are a media organization and serif fonts are typically used for article bodies. However, combined with pixel density, the serifs will lose a lot of their definition and (in my opinion) reduce readability versus a sans-serif font. As an experiment, pull up the dev tools and change article[data-brand="businessweek"] .body-copy p to use sans-serif and see what you think. It may be marginally easier to read.
4. Don't discount OLED. The contrast that Bloomberg selected is maximal (pure black background and pure white text) and that works well for OLED since the background vanishes entirely. However, since most desktop and laptop monitors are not OLED, you're still getting backlight bleed, so the contrast is imperfect. Especially combined with the glare factor above, my experience is that given LCD backlight bleed, it is better to use a dark gray background instead of stark black. This makes the backlight bleed less distracting, for lack of a better word. The background ends up looking more uniform.
As with above, try adding "background-color: rgb(40,40,40)" or similar to .body-copy and see what you think.
This is because, contrary to what a lot of people say about dark themes (even though it's mostly a meme at this point), dark letters on white background are better to read than white letters on black background.
"People with astigmatism (approximately 50% of the population) find it harder to read white text on black than black text on white. Part of this has to do with light levels: with a bright display (white background) the iris closes a bit more, decreasing the effect of the "deformed" lens; with a dark display (black background) the iris opens to receive more light and the deformation of the lens creates a much fuzzier focus at the eye." - Jason Harrison – Post Doctoral Fellow, Imager Lab Manager – Sensory Perception and Interaction Research Group, University of British Columbia
This quote describes the situation well I think, why the opinions differ and why neither dominates UIs really.
Why aren't these attacks constrained by normal corporate firewalls? How does a random server on a navy ship start contacting baddie.china.com without raising red flags?
Right so all the stars need to align for it to go unnoticed - compromised server, firewall and other alerting/monitoring tools.
I would have thought one single unexpected packet in these high security environments would raise significant alarm bells and any anomaly would be found very quickly.
Intrusion detection involves connections coming from the outside. These attacks originate inside the network, from the compromised equipment.
Except on extremely controlled networks, this would be very hard to detect. It gets even worse when you consider that the Chinese had/have a distributed network of compromised machines. Imagine using a Google edge server as a dead drop...
We had something similar where anything outside "normal" generated a ticket. It was disabled after 1 week because the support teams where getting more then 5000+ tickets each day. And this was after filtering etc....
Now this does not make it impossible, just very complex. In a more "controlled" environment such as a naval ship, i could see this actually working better, especially if the system is supposed to talk to very few external systems.
It does. I've deployed systems that would not only notify staff when novel packets were observed but immediately isolate anomalous hardware through a combination of powerdown and network fabric reconfiguration.
The only way to create such an environment is to totally disconnect it from the outside world -- I'm talking even power source, phones, internet, all of it has to be disconnected or else I can exfil data all day long and nobody would ever know.
Only if you’re lucky. Priority is checking boxes and meeting audit requirements.
If you’re running an IDS on a big 100 Gb datacenter network, you’re literally processing millions of events. Very few places would notice such a thing unless they were investigating something related, and the ones with the capability are going to be for static workloads as getting anything done will be slow and painful.
Effectively nothing can be constrained by a whitelisting firewall if you have a sufficiently bored actor. You can smuggle data through a variety of benign looking protocols, things that wouldn't matter in the least generally. Your average server contacts hundreds of different public NTP servers, binary repositories, domain name servers every day. If the keys to the kingdom are a 32 byte ECDSA private key, you've lost if you think you can protect this from reaching the outside world.
A method that wouldn't show up on any firewall in the world is simply to delay or drop certain SYN packets. Even if you only intended to transmit a bit at a time through this, any unauthenticated host on the internet could use this without raising any suspicion or even printing log lines in most environments. As soon are you're making an assumption that you're trying to prevent what's inside from getting out things become substantially closer to impossible than anybody would like.
Firewalls in high security environments aren't just port/protocol based. You lock everything down - source ip/port and destination ip/port. You should know where it is coming from and where it is going to.
In the parent I described a system which would be able to communicate through those restrictions to another compromised host (remember we're assuming everything is compromised for the sake of this article, which actually seems like a good assumption now).
> remember we're assuming everything is compromised
I think Bloomberg (and all related) web servers displaying the article are compromised and they're leaving out critical facts the point the finger elsewhere.
Most likely it is using SPI https://en.wikipedia.org/wiki/Serial_Peripheral_Interface, that requires four pins and the two remaining ones are power and ground. SPI what is used to access EEPROMs and flash memory, so an attack that you can do is daisy chain such a device in the path to the EEPROM the board management controller uses as its firmware storage. Then you can very easily insert your own instructions and get the board management controller to execute whatever you want.
> Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not.
The Chinese government didn’t directly address questions about manipulation of Supermicro servers, issuing a statement that read, in part, “Supply chain safety in cyberspace is an issue of common concern, and China is also a victim.”
Essentially China ils saying "it was not me". Plausible
I Read that as "The US is also attacking our hardware supply chains". That is, the statement concerned supply chain attacks in general, not this specific one.
China is sourcing virtually nothing from US. If US is attacking the supply chain of China, it's doing it on Chinese soil. Which brings us back to my interpretation
The only interesting thing about this is left out. Who planted it is clear (someone told to do so) but not a single time is it questioned who they planted it for. Smells like false flag to me.
We think China does X Y and Z but we know the US does X Y Z and the rest of the alphabet. So unless something specific is leaked that shows who actually ordered this, logic would point at the US.
But then, since the logic points to the US it only stands to reason that is a false false flag! Why those wiley spies in Beijing really are clever. Discredit The USA and plant spy chips in important computers!
> ...but not a single time is it questioned who they planted it for...
The article and some accompanying reporting on Bloomberg audio/video segments says the attack seems targeted relatively specifically towards nearly 30 organizations (only US-based organizations were mentioned as targets, unknown if the list included organizations based in other nations). One known vector was through four subcontractors in China that built the boards for Supermicro's main Shanghai factory, specifically by bribing and/or coercing managers of those subcontractors' factories to go along with accepting the chip shipments and to make changes to the plant floor from the design to perform the chip insertions.
Designing and building a chip like this and then mounting the logistical effort to performing the insertions costing some non-trivial funds, coupled with the known targets, (Amazon didn't seem specifically targeted, Elemental a company they acquired was, who notably has US national security clients), form the circumstantial allegation that a PLA spy unit was behind the attack. You are correct that this doesn't entirely rule out a false flag possibility, but until we get more details about this, we're operating in the dark.
A false flag is an interesting supposition, but how would the US benefit from successfully convincing the world of the false flag's cover story?
The US has a few things to gain from this story: Economically, because Chinese products are perceived as compromised. Politically: because the Chinese government is seen in the offense.
The US has something to loose too: Being perceived as dependent on Chinese manufacturing and potentially compromised down to military hardware. (The first everybody knows, the second would be devastating for trust.)
All in all it would be a weird angle for a false-flag attack.
The article specifically mentions the PLA as the bad guy. So that part was not left out. The whole story is hearsay so far. It's plausible, but best read with a good dose of salt.
If we believe the story up to the point where subcontractors are forced into planting the chips, we must accept that it's easiest and least risk for Chinese government actors to force them into it. Mounting a false-flag attack at Chinese subcontractors would be exceedingly difficult for foreign agents. They'd likely blow their cover when trying to represent Chinese officials.
If you want an alternate version, I suggest you start with the easiest: The whole thing didn't happen. And Bloomberg is a victim of propaganda.
Probably much bigger. It's naive to assume Supermicro was the only vendor compromised in this way. Similar implants probably exist in equipment from every major vendor of information-processing equipment. Routers, mobile network equipment, you name it.
This is the leverage you give another nation-state when you let them control your supply chain.
>Amazon’s security team conducted its own investigation into AWS’s Beijing facilities and found altered motherboards there as well, including more sophisticated designs than they’d previously encountered.
>...the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached
>...that generation of chips was smaller than a sharpened pencil tip, the person says.
I hope this corrects the mistaken believe that China can't home grow sophisticated tech.
This sounds strange but it's commonplace to make parts like this. Some DCDC converter modules have silicon dies embedded into their PCBs to save physical footprint, most notably there's a family of Texas Instruments ones which use this technique.
> I hope this corrects the mistaken believe that China can't home grow sophisticated tech.
There's nothing particularly sophisticated about what they did, especially given what China has access to as one of the central hubs of tech manufacturing. There are two dozen nations (or more) that could do this from a strictly technical standpoint (few have the kind of required supply chain access to pull it off at scale in actuality). It's the audacity that is primarily impressive. China is encouraging all of their richest trading partners to further isolate them when it comes to supply chain and advanced tech. They're confirming for the 107th time what everyone already believes.
One question though. Would a politically correct, by-the-book US president have had the balls to sanction China? Considering such sanctions could affect the US economy.
Given that there’s video from 4 decades in which Trump advocates for tariffs, during which time he wasn’t a politician, it would seem that they are something he believes strongly in.
The goal, explicitly stated and often acted on in many contexts, is to disrupt international order and cooperation (including trade), and promote nationalistic competition by all countries. Recently his UN speech, for example, advocated it.
IC fabrication and packaged component assembly are very different things. China doesn't have anything near this level of sophistication, at least not without any outside help.
My gut feel about this for many years has been outsourcing chip manufacturing to another country is a serious security risk. This was a hardware device, hard to detect as it was, how much code is in the chips we already expect on a motherboard?
It seems flat out foolish for one country to own the world’s computer manufacturing.
It's been a few years I've given up on the idea of privacy with technology.
The number of security flaws that get discovered daily is only the tip of the iceberg.
I'm pretty sure some governments (or organizations) have had backdoors, be they hardware or software, in place for more than 20 years. We simply don't know about it yet (and probably never will). Would that actually be that far-fetched? I think not sadly.
Even the Intel Spectre et Meltdown fiascos are a sign that we have no idea how to actually secure this stuff. And that's normal, the very definition of IT security is that nothing can be secure.
Take the whole antiquated concept of processor rings for instance, we are adding a new level every other year now it feels like...
I find it way more interesting (even if it is ultimately "worse") to adapt to the mentality that "nothing is secure" than "let's try and make it secure", which as stated is in itself a fallacy...
I think the root of the problem is backwards compatibility and the fact that initially all these components were not designed with security in mind. So we're adding more insecure components in spirit of "move fast, break things" than fixing the debt incurred.
> It's been a few years I've given up on the idea of privacy with technology.
The problem with tech is that it's modularity and dependency on other people/tools/hardware/etc ultimately requires trust.
> in place for more than 20 years.
Far longer than 20 years. The idea of trust, privacy and security has been discussed for a long time. The complexity of security on just on one specific technology ( compilers ) was discussed in the early 1980s by Ken Thompson.
Compilers are just one part of a complex ecosystem. Now imagine having to check ABI or the physical chips/hardware themselves. Where you require sophisticated hardware.
The more I understand software the less I trust it (given the current state of engineering practices). Meanwhile all my friends/family are scrambling to install all the latest new "smart home" gadgets and I just look like a paranoid kook trying to talk them out of it.
This. I've even had an in-law say, "My brother works for the Defense Intelligence Agency, and he uses smart devices in his home, so they must be safe!", with no consideration that tech may not be his specialty, or he doesn't follow the daily IoT fiascos, or maybe he just thinks he won't get hacked. Dunno. Meanwhile, my year-old thermostat still wants me to connect it to wi-fi, and that will never happen.
While you may find a particular attack if you're looking for, in general, it's impossible for even the most thorough audits to check for the whole class of such attacks. You're not going to look into the chips. Well, you can, but that's prohibitively expensive and destructive - even if you could check that this chip was okay, then you still have to throw it out after analysis and plug in a different one.
The only feasible thing to do is thorough audits of all the supply chain for every component in your system, ensuring that your supply chain does not include even a single chip from an "untrustworthy" supplier, and even then it reduces the chances of an attack but does not eliminate it.
> Faggin: Yes, we were concerned about others copying the Z80. So I was trying to figure what we could do that that would be effective, and that’s when I came across an idea that if we use the depletion load the mask that doesn’t leave any trace, then I could create depletion load devices that look like enhancement mode devices. And by doing that we could trick the customer into believing that a certain logic was implemented, when it was not. Then I told Shima, “Shima, this is the idea how to implement traps. Put traps, you know, figure out how to do the worst possible traps that you can imagine,” and then Shima with his mind, that was steel mind, was able to actually figure out a bunch of traps that he could talk about.
> Slater: You want to tell us a little about that Shima?
> Shima: I didn’t count [on] talking about that mostly. I placed six traps for stopping the copy of the layout by the copy maker. And one transistor was added to existing enhancement transistors. And I added a transistor looks like an enhancement transistor. But if transistors are set to be always on state by the ion implantations, it has a drastic effect on very much. I heard from NEC later the copy maker delayed the announcement of Z80 compatible product for about six months.
> The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
Obviously this would still be possible without IntelME, but having an always on, highly privileged and remotely accessible baseband definitely makes the modifications easier and smaller...
Rather not, China's economic rise started in Jiang's era was both due to it being cheap, AND due to Jiang's era officials being more competent and business friendly than those of an average bantustan (really sorry having to use the term)
"Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."
Because it's so awesome. This must be one of this journalist's career highlights, to be able to put something like this in a mainstream serious reporting piece.
816 comments
[ 1.2 ms ] story [ 697 ms ] thread> During the ensuing top-secret probe, which remains open more than three years later, investigators determined that the chips allowed the attackers to create a stealth doorway into any network that included the altered machines. Multiple people familiar with the matter say investigators found that the chips had been inserted at factories run by manufacturing subcontractors in China.
That said, it's pretty scary that you can hide so much malicious functionality in such a small device, makes me wonder what might be hidden in my Lenovo. In any case it speaks highly of the auditing firm that they were able to locate this. I wonder if they performed an x-ray analysis of the board, as given the size of these chips it should be possible to embed such devices in one of the internal layers of the board as well, making them essentially invisible to optical inspection.
This stuff ends up being extremely difficult to disable. The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC invisibly to the OS and DHCPs on its own to open up an administration port, web interface, and some assorted call homes. You have to explicitly tell it to use the non-connected port, change credentials, and modify it so that it is not accessible within operating system as well. Hopefully while the machine is offline to prevented any automated scanning finding it within your network.
The number of times I'd end up nmaping our local networks and being able to remotely access production hardware with an interface that allowed me to reach this interface was maddening. The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely nu-authenticated avenues exposed. The course of action was always just to write off the hardware and bin it, because god knows what impact you could actually have using that interface.
> The organizations behind the new project each have already made substantial contributions to creating open source baseboard management controller (BMC) firmware. Now, working together, they will define the vision for a standard stack that can be used across systems and computing environments.
LinuxBoot and Open Compute OSF are working on open-source server firmware that can be measured on every boot and validated against hardware root of trust keys controlled by the server owner instead of the server OEM, https://www.platformsecuritysummit.com/2018/speaker/hudson/
> The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely un-authenticated avenues exposed.
https://www.itworld.com/article/2708437/security/ipmi--the-m...
https://www.virtuallifestyle.nl/wp-content/uploads/2016/08/S...
http://www.supermicro.com/products/nfo/SMS_SUM.cfm
At least on some boards you can boot the USB drive image containing the BIOS updater through the BMC and do a remote update that way.
Now even a hardware interlock could be subverted, but that's harder than sticking code in the bmc firmware, which does tend to get updated during the life of a server.
After popping an internet facing web server, I was able to compromise the IPMI system and use the management network to bounce around to any server in the enterprise completely bypassing all the firewalls and segmentation on the production network.
Management networks need rack level isolation.
https://my.mixtape.moe/sorqql.jpeg
cool, thanks for that info.
> just to write off the hardware
maybe you could just standup the mgmt network but blackhole route it at each switch port. The mgmt NIC thinks it's working properly but it can't talk to anyone nor can anyone talk to it.
at the expense of a dedicated switch.
The payload could be pretty small, since the server boards are likely using OS packages that match the chipset. This limits the software to a small set of well known targets, Linux, Windows, Apple. Target their kernels, and you only have to snip out a small chunk of bytes, and splice your own pre-defined package in. Splice in a miniature runtime, that operates a turing complete set of operations, and open up a listener that waits for network access, and now, your payload can enable arbitrary code execution, irrespective of permissions.
Now, to exploit, the payload needs to time the opportunity to splice itself onto the disk correctly. If certain well-known chunks of code will always exist in each given operating system, then with every disk access event one just needs to wait for the inevitable moment those magic system-specific bytes travel over the bus, in order to replace the known bytes with the poisoned modification. Events might target when the bytes are originally installed with the OS, or every time the OS reads those known bytes back into live memory, from any source.
The total payload package could probably fit inside a couple of megabytes, pack on a few more for the "listen & splice" part of the attack to round out the entire mass, and all we know how much data an SD card can fit into say... five grains of rice?
https://www.microchip.com/wwwproducts/en/ATtiny4
For scale, this alone is about the size of a large SMD capacitor and would basically be lost in most designs today.
https://en.wikipedia.org/wiki/Magic_number_%28programming%29
For example, looking for ELF or Portable Executable headers, as a crude estimate to determine attack opportunities. In this case, the magic numbers would probably be more selective and sophisticated, but still have an aspect of hard-coded values, since we're talking custom silicon.
https://www.digikey.com/en/product-highlight/t/taiyo-yuden/e...
They probably found it out when they were repeatedly tried to reflash the BMC flash, and saw that checksums did not match.
And prior to that, there were already persistent rumors in the Chinese interney of certain Chinese mobos sending "weird garbage on ICMP," and "BMCs that somehow boot and work with their flash memory soldered off"
Remembering that, I might even suggest that this is not a modchip that does something with signal on the go, but just a very tiny flash chip that has the modded firmware.
Going further from that, to pack, say, 16 mB on a sandgrain sized chip, the densities need to be like that of best flash chips out there, which also means that they have access to last gen flash fab.
https://www.itcreations.com/dist/landing/i/MBI-6128R-T2/MBI-...
Left of the sata connector. An empty space with 8 pads for an smt eeprom or flash. It is occupied by the thingy on bugged boards.
Right below is the Aspeed chip - the BMC
https://download.csdn.net/download/duanzhang512/10385038
The recovery overrides the primary if detected by default.
The place they put their "filter cap" is right on top the empty TSOP8 pad for the recovery flash. And they probably ordered the factory to sneak the traces just a little bit more, or put hidden vias under it, or simply had somebody very dexterous to solder it to pads with hair thin wires.
It also means that the extend of intervention into board design was minimal, and that a trivial automatic xray would not have picked it up. And as implied in the article, later they buried the bug to beat the AOI, if it was done higher upstream.
So, they would've been screwed even if they were doing board testing outside of China.
That's a clever trick.
But the sole fact that the chip has "to phone home" makes detection trivial, and puts the usefulness of the method to nil - anybody sees the router blink when it shouldn't and your bug's cover is blown.
That makes the whole thing gloriously simple. A part "stuck on" afterwards is obvious. A part fitted into a no-fit footprint after optical inspection is not, it looks exactly as if it was meant to be there.
But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
To me, that makes it sound like they could download from a remote host and inject code and do literally anything.
[1]https://patents.google.com/patent/US8346951B2/en
https://news.ycombinator.com/item?id=18144519
https://freebeacon.com/national-security/military-warns-chin...
Talking about things like this https://news.ycombinator.com/item?id=18074097
In other words, if you have plans to ever be a politician that china may not want in any position of power, don't store your nudes on icloud/dropbox/google drive, or they may suddenly get leaked when you least expect it and ruin your career in favor of a more... Shall we say "convenient"? ...alternative.
You are correct that individuals seeking personal infosec against state actors must be eternally vigilant.
There was a sense of realpolitik by one UK guest commentator on Bloomberg, comments along the lines of "hey, spying happens since time immemorial, put on some big boy pants, yes there is shock but not horror when the Snowden revelations came out, the US does it, too, etc.". I disagree with this sentiment, as while the attack was quite targeted, it puts into question a quite large supply chain network.
Kudos to Bloomberg putting in the 12+ month investigative journalism to pull off this scoop. Yet another validation of the reasons I subscribe to listening/watching them.
[1] https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...
Supermicro is an American corporation, headquartered in San Jose. They're not directed by Chinese national security interests, they'll do anything the US Government tells them to do when it comes to US national security.
Supermicro's vendors they use in their Chinese manufacutring are the likely vectors for the implantation.
Apple’s response isn’t great IMHO.
And I have no idea why, I searched on Google and couldn't find any decent answer. All results were either OLED being battery friendly with switched off pixels or other mobile UX for Dark mode. None of them describe the difference felt in Desktop and Mobile. Any link or explanation would be much appreciated.
1. Glare. When reading on a small device, the amount of glare reflected in the screen space occupied by black pixels may be fairly minimal depending on your reading environment and positioning. Also, although mobile devices use glossy display surfaces, they tend to have several anti-glare layers in the display stack. You said your computer is macOS, so I suspect you're unlucky enough to be reading on a glossy laptop display. With "dark mode" color schemes, your eyes can more easily see the reflected scene (maybe your office lights, an exterior window, or even your own face). And the focal length of that reflected scene is 2x your reading distance to the screen. That reflected scene at an extended focal length is more relaxing for your eyes to focus on. So in order to read, you need to fight your natural tendency to relax and look at the reflected scene.
If you are lucky enough to be reading on a matte desktop display (typically a professional or prosumer monitor, such as a Dell UltraSharp or LG 43MU79-B), the glare will be minimal and it should be much easier to read.
2. Pixel density. I contend that one reason dark themes have become more prevalent in recent history is thanks to wider adoption of high pixel density displays. At a legacy density of approximately 75 dots per inch, the stroke weight of bright text on a dark background appears too faint if the strokes render as just one pixel in width. Higher pixel density allows for the strokes of letters to be wider than a single pixel, allowing for greater clarity. If you ever designed a dark theme in the days of ~75 dpi displays, you might naturally favor bold text as the default because it was considerably easier to read. (Interesting sidebar: many console oriented bitmap fixed-width fonts historically used two pixels for stroke weight, especially in the horizontal dimension, presumably because they were designed to be used bright-on-dark.)
Now, you did again say you were reading on macOS, so your display's pixel density is probably higher than ~75 dpi. But a MacBook Pro is still only ~220 dpi. A Surface Book is ~260 dpi. A Dell XPS laptop with a 4K display will be a little higher still (maybe ~300 dpi). But many phones are using 450+ dpi displays. The stroke weight of a character on a mobile device is several pixels wide, so it's highly defined and clear.
3. Font selection (related to above). Bloomberg has selected a serif font, presumably because they are a media organization and serif fonts are typically used for article bodies. However, combined with pixel density, the serifs will lose a lot of their definition and (in my opinion) reduce readability versus a sans-serif font. As an experiment, pull up the dev tools and change article[data-brand="businessweek"] .body-copy p to use sans-serif and see what you think. It may be marginally easier to read.
4. Don't discount OLED. The contrast that Bloomberg selected is maximal (pure black background and pure white text) and that works well for OLED since the background vanishes entirely. However, since most desktop and laptop monitors are not OLED, you're still getting backlight bleed, so the contrast is imperfect. Especially combined with the glare factor above, my experience is that given LCD backlight bleed, it is better to use a dark gray background instead of stark black. This makes the backlight bleed less distracting, for lack of a better word. The background ends up looking more uniform.
As with above, try adding "background-color: rgb(40,40,40)" or similar to .body-copy and see what you think.
OLED works better with darkness than with bright backgrounds.
This quote describes the situation well I think, why the opinions differ and why neither dominates UIs really.
I would have thought one single unexpected packet in these high security environments would raise significant alarm bells and any anomaly would be found very quickly.
This sort of monitoring doesn't happen in the real world.
Except on extremely controlled networks, this would be very hard to detect. It gets even worse when you consider that the Chinese had/have a distributed network of compromised machines. Imagine using a Google edge server as a dead drop...
Now this does not make it impossible, just very complex. In a more "controlled" environment such as a naval ship, i could see this actually working better, especially if the system is supposed to talk to very few external systems.
Security engineering is about tolerable failure modes. - Dan Geer (2014)
If you’re running an IDS on a big 100 Gb datacenter network, you’re literally processing millions of events. Very few places would notice such a thing unless they were investigating something related, and the ones with the capability are going to be for static workloads as getting anything done will be slow and painful.
A method that wouldn't show up on any firewall in the world is simply to delay or drop certain SYN packets. Even if you only intended to transmit a bit at a time through this, any unauthenticated host on the internet could use this without raising any suspicion or even printing log lines in most environments. As soon are you're making an assumption that you're trying to prevent what's inside from getting out things become substantially closer to impossible than anybody would like.
Navy ships don't upload via Dropbox.
I think Bloomberg (and all related) web servers displaying the article are compromised and they're leaving out critical facts the point the finger elsewhere.
We think China does X Y and Z but we know the US does X Y Z and the rest of the alphabet. So unless something specific is leaked that shows who actually ordered this, logic would point at the US.
But then, since the logic points to the US it only stands to reason that is a false false flag! Why those wiley spies in Beijing really are clever. Discredit The USA and plant spy chips in important computers!
The article and some accompanying reporting on Bloomberg audio/video segments says the attack seems targeted relatively specifically towards nearly 30 organizations (only US-based organizations were mentioned as targets, unknown if the list included organizations based in other nations). One known vector was through four subcontractors in China that built the boards for Supermicro's main Shanghai factory, specifically by bribing and/or coercing managers of those subcontractors' factories to go along with accepting the chip shipments and to make changes to the plant floor from the design to perform the chip insertions.
Designing and building a chip like this and then mounting the logistical effort to performing the insertions costing some non-trivial funds, coupled with the known targets, (Amazon didn't seem specifically targeted, Elemental a company they acquired was, who notably has US national security clients), form the circumstantial allegation that a PLA spy unit was behind the attack. You are correct that this doesn't entirely rule out a false flag possibility, but until we get more details about this, we're operating in the dark.
A false flag is an interesting supposition, but how would the US benefit from successfully convincing the world of the false flag's cover story?
The US has something to loose too: Being perceived as dependent on Chinese manufacturing and potentially compromised down to military hardware. (The first everybody knows, the second would be devastating for trust.)
All in all it would be a weird angle for a false-flag attack.
If we believe the story up to the point where subcontractors are forced into planting the chips, we must accept that it's easiest and least risk for Chinese government actors to force them into it. Mounting a false-flag attack at Chinese subcontractors would be exceedingly difficult for foreign agents. They'd likely blow their cover when trying to represent Chinese officials.
If you want an alternate version, I suggest you start with the easiest: The whole thing didn't happen. And Bloomberg is a victim of propaganda.
This is the leverage you give another nation-state when you let them control your supply chain.
>...the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached
>...that generation of chips was smaller than a sharpened pencil tip, the person says.
I hope this corrects the mistaken believe that China can't home grow sophisticated tech.
https://www.electronicdesign.com/sites/electronicdesign.com/...
There's nothing particularly sophisticated about what they did, especially given what China has access to as one of the central hubs of tech manufacturing. There are two dozen nations (or more) that could do this from a strictly technical standpoint (few have the kind of required supply chain access to pull it off at scale in actuality). It's the audacity that is primarily impressive. China is encouraging all of their richest trading partners to further isolate them when it comes to supply chain and advanced tech. They're confirming for the 107th time what everyone already believes.
One question though. Would a politically correct, by-the-book US president have had the balls to sanction China? Considering such sanctions could affect the US economy.
Does Trump even have a goal in mind with his tariffs? It seems like he just wants to score political points, rather than achieve any actual outcomes.
It seems flat out foolish for one country to own the world’s computer manufacturing.
The problem with tech is that it's modularity and dependency on other people/tools/hardware/etc ultimately requires trust.
> in place for more than 20 years.
Far longer than 20 years. The idea of trust, privacy and security has been discussed for a long time. The complexity of security on just on one specific technology ( compilers ) was discussed in the early 1980s by Ken Thompson.
"Reflections on Trusting Trust"
https://news.ycombinator.com/item?id=13569275
Compilers are just one part of a complex ecosystem. Now imagine having to check ABI or the physical chips/hardware themselves. Where you require sophisticated hardware.
The motto of technology is "In Trust We Trust".
However, IMO, the "cost" of ignorance such area is not that bad either. I will let them live their way to be honest.
They likely needed to have the exact official schematic of the motherboard to compare every single detail of the hardware with.
The only feasible thing to do is thorough audits of all the supply chain for every component in your system, ensuring that your supply chain does not include even a single chip from an "untrustworthy" supplier, and even then it reduces the chances of an attack but does not eliminate it.
> Faggin: Yes, we were concerned about others copying the Z80. So I was trying to figure what we could do that that would be effective, and that’s when I came across an idea that if we use the depletion load the mask that doesn’t leave any trace, then I could create depletion load devices that look like enhancement mode devices. And by doing that we could trick the customer into believing that a certain logic was implemented, when it was not. Then I told Shima, “Shima, this is the idea how to implement traps. Put traps, you know, figure out how to do the worst possible traps that you can imagine,” and then Shima with his mind, that was steel mind, was able to actually figure out a bunch of traps that he could talk about.
> Slater: You want to tell us a little about that Shima?
> Shima: I didn’t count [on] talking about that mostly. I placed six traps for stopping the copy of the layout by the copy maker. And one transistor was added to existing enhancement transistors. And I added a transistor looks like an enhancement transistor. But if transistors are set to be always on state by the ion implantations, it has a drastic effect on very much. I heard from NEC later the copy maker delayed the announcement of Z80 compatible product for about six months.
http://archive.computerhistory.org/resources/text/Oral_Histo...
Good starting point.
Obviously this would still be possible without IntelME, but having an always on, highly privileged and remotely accessible baseband definitely makes the modifications easier and smaller...
There is only one I know of: Soyo motherboard
Their behavior, not their communication, has been overtly hostile for a while. Yet, very few politicians openly adress the issue.
Never, unless hardware manufacturing will take off somewhere else.
"Nobody's making you buy from China"
There’s a certain Casablanca-esque “what? There’s gambling happening here?” element here too — Huawei was shipping fake Cisco gear 20 years ago.
Hahaha funny thing to say after disclosers from NSA/CIA/FBI/GCQH/BND...