I thought they got access through an update sever with a weak password?
Was the attack otherwise sophisticated and just relied on an easy entry point? So the breaking in was easy, but the plan to steal once inside was sophisticated?
Maybe do just a few minutes of research before first posting on a thread about state sponsored cyber crime.
The actual "hack" was a supply chain hijack, the actors inserted malicious dll code through a solarwinds authorized developer workstation, thus ensuring the code would be signed, and then pushed as a legitimate update to the downstream implementations of solarwinds.
The whole weak password thing was simply a funnily timed thing some guy found. It had absolutely nothing to do with the malicious update, which was the source of this supply chain attack. For some reason a bunch of people online decided that two things happening around the same time means they are related. They are not
No, “a bunch of people online” decided that if you have jarring security practices in one avenue, it’s totally unsurprising if it turns out that you have other doors wide open too. Which is a completely reasonable assumption.
It can definitely be said that this may be an indicator of a larger security issue at the company. But assuming that issue is the cause for the backdoor, yes that is a leap, and an unfounded one. Correlation != Causation, and it's always unreasonable to assume otherwise.
See it like stealing from a museum: very easy to break in (weak password -> breaking a window), hard to get out of here with all the paintings unnoticed - and that, on a daily basis.
The amount of effort and different techniques they've put in to remain undetected for months (years?) while infecting a lot of actors is pretty impressive.
There had been a weak password on the update server in the past. The compromised update however was introduced via the build system before being signed and distributed
Most people know which countries developed stuxnet and used compromised human agents to place the infected drives into the target system.
The sophistication of the malware required state powers and assistance of the industrial company that manufactured the control systems. Those systems are not cheap and reverse engineering the control system software required detailed knowledge of the embedded hardware and software.
From what I've read and heard about the attack in the past, they had managed to find the supplier and had confiscated some blueprints or the actual structures (nuclear reactor?) that was used and then reverse engineered it from there with the help of nuclear scientists to find vulnerabilities in it.
from what I understand, they have the same controller that was being used in the centrifuges on their desk to develop the payload the rest was just a way to get to the controllers of the centrifuge which was rather standard.
It's not a zero sum game. It is interesting in that it marks the first _wide spread_ attack on a supply chain via cyber space. The malware's DNS signaling protocol is pretty neat, which you can look at here: https://www.fireeye.com/blog/threat-research/2020/12/sunburs...
The headline is "most sophisticated ever" not "totally boring". SolarWinds can both be a very interesting thing and the headline can also be hyperbole.
On December 8, 2020, the cybersecurity firm FireEye announced that red team tools had been stolen from it by what it believed to be a state-sponsored attacker.[106][107][108] FireEye was believed to be a target of the SVR, Russia's Foreign Intelligence Service.[27][109] FireEye says that it discovered the SolarWinds supply chain attack in the course of investigating FireEye's own breach and tool theft.[110][111]
After discovering that attack, FireEye reported it to the U.S. National Security Agency (NSA), a federal agency responsible for helping to defend the U.S. from cyberattacks.[1] The NSA is not known to have been aware of the attack before being notified by FireEye.[1] The NSA uses SolarWinds software itself.[1]
Largest impact, sure. But architecturally it was a relatively simple formula - compromise a widely used package and sleep on it until it was pervasive enough to be a valuable hack. I disagree with this being the most sophisticated though. Unless I'm missing something about this hack, the Stuxnet[1] architecture, complexity, and long term planning feel far more sophisticated than the SolarWinds hack.
I guess they mean latest hack we have been the target of? Stuxnet was custom written malware targeting Iran, so yes I would agree with you but perhaps what he was trying to say is still true.
The attack is unique in its usage of the supply chain. The malware is not 'sophisticated' in the same way stuxnet is because it has different goals. This actors goals align with stealth above all else, which is evident in both the design of the malware and the choice of the supply chain delivery vehicle. Also realize that the network comminication scheme used attempts to blend in with the legitimate SolarWinds software. If you design a stuxnet like malware and deploy it to 18k+ companies it will be found, because exploiting zero days left and right is super noisey. This malware was minimal. It's cool in a different way
Stuxnet was also a supply chain attack. The first infections were at Foolad Technic and Behpajooh, a pair of privately owned engineering firms.
The malware keeps a "breadcrumb" trail of each machine it infects, and based on investigations by Symantec [1] it was determined that every Stuxnet sample from Natanz originated from outside suppliers.
I don't feel that's really a counter-argument to it being less sophisticated. Sure SolarWinds might be a better match for its makers goals, but sophistication is not the same thing as fitting its purpose.
I would say that the sophistication of the Solarwinds breach is in the success of its scale, as opposed to the methods with which it used to successfully reached such scale.
Stuxnet was very sophisticated to hit a relatively narrow target by comparison.
So I think they are both sophisticated, but not comparing them directly.
Stuxnet infected close to 200,000 machines, over half in Iran, before being detected. It was so extremely stealthy and well designed that it could spread widely and avoid detection while making its way to the intended targets.
I would love to know how many machines got hit by SOlarwinds - however, its not about the machines - its about the value of the data it slurped... So even if it was one exchange server, with 10,000 treasury/nsa/whatever accounts, that one machines intel value is quite high.
Absolutely agree. Solarwinds focuses a disproportionate amount of effort in ensuring it shows up favorably in Gartner magazine reviews and trade publications. As a monitoring platform its a monolithic, expensive, slow and rather dated monitoring solution. Agile does not come to mind, and you certainly wouldnt use it for anything approaching "observability."
But the concerted marketing effort pays dividends. Solarwinds is almost the only choice for government. For potential attackers its a big red arrow. Exploit a rent-seeking company that writes mediocre software and exists mostly to cash in on "best practice" lock-in with government contracts.
as far as i know nobodys really addressed the elephant in the room. Solarwinds is still a preferred government purchase for monitoring. every company that was affected by it still uses it (or at least hasnt publically refuted it) and no government official has come forward to admit they will discontinue it. Solarwinds hasnt offered any remediation or major changes in their development, leadership or code. just patch, rinse, and repeat and try not to pay too much attention to the issue.
LOL the whole “fall guy” thing is how companies keep doing what they’re doing while making some symbolic penance. For a software company a fuckup of this magnitude should be: all customers leave, company dies, execs never work again. Anything less is an insufficient incentive to work extremely hard to prevent this from happening.
Sorry, I can’t agree with you here. What you’re saying is basically company’s dissolution of it is hacked. If this indeed becomes the case the companies will indeed kill to keep their secrets.
First, with GDPR and co, they can't keep their secrets if they get hacked.
Second, with such negligence? They deserve to be shut down. This isn't a highly sophisticated zero day exploit, there were multiple huge failure at Solarwinds that allowed the attack to happen.
And if you ever ignore a red light, pay your taxes late or do anything other than your absolute best to be an upstanding citizen you should be put into prison for the rest of your life.
Years ago, the company I worked at had a very big production issue which resulted in a customer's database being deleted. Of course, the customer was furious and called the CEO asking for the person responsible to be fired.
Calmly, our CEO said: 'No. If there's anyone in this company who will never make that mistake again it's him.'
I've seen this sort of Zen of CEO type story a lot but it doesn't match up with reality. Someone that deletes a production DB by being careless or reckless is likely to do something similar again. Perhaps not in the exact same way, but there's infinite ways to break things. Those that stumble upon one are likely to stumble upon another.
In that case everyone making 1 mistake is 'out'. Seems "cancel culture" is leaking into ops...
I was in the room when the CEO told that (one of the nice things about small companies) and have had contact with the person responsible for a few years until he got another job. True story ;)
There were changes though: the 'two pair of eyes' principle was enforced a lot stricter from then on.
Your method sounds good, but it makes things less safe in practice because it provides every possible incentive to hide, cover up, deny, etc., any problems.
A far better system is "no fault" where the company has incentive to be open about problems and finding solutions.
Anectodally; I have been a one man army CTO for a period. I could not have managed my Windows servers without Solarwinds. It was an excellent product which did everything I could ever want.
> nobodys really addressed the elephant in the room.
I think the elephant in the room is actually a more general issue that is cross platform and independent of the product implementation.
It’s 2021 and we are still ignoring the fundamental “best practice” that we’ve known about for at least 20 years.
Systems should be isolated from each other unless there is an overwhelming need for them to be connected and everything should run with the least privilege.
Centralised credential vaults and binary artefact repositories are an obvious example.
It's also important to have diversity. For example, on the Boeing 757 there are two computers that control the stab trim, that do the same thing. They must agree or both computers are automatically locked out.
The two computers are developed by two independent teams who are not allowed to talk to each other. Two different CPUs, two different algorithms, two different programming languages.
The idea, of course, is a design problem with one does not propagate to the other.
If one program is used in all your infrastructure, all your infrastructure is compromised when that program has a bug.
I wonder if there's a way to bring that approach, into the world of 1) software build servers, 2) third party dependencies, and 3) code review.
1) Build servers: Maybe by keeping it possible to build one's software on a laptop — then, when it's time to release something, one can build the software at the build server plus some people's laptops and compare hashes.
2) ???
3) Different people who don't talk with each other, do security code review, independently of each other? Hmm does that make sense
>> Systems should be isolated from each other unless there is an overwhelming need for them to be connected and everything
And we should have checks built into the build pipelines and checksum the files that go into the build. We have Merkle trees for a long time. I think, it would be possible to detect injected code.
The attacker in this case new the exact moves to insert a backdoor into .NET software. It wasn't hard to do, requiring no science.
But Microsoft's "system of trust" was undermined. The attacker was even inside Microsoft and Azure. No anti-virus, no "defender", no amount of basic or advanced telemetry caught this.
FireEye alone caught it... By accident.
The real "elephant in the room" is that software security continues to be marketing theatre. Closed-source software should not be trusted and consumers should not believe that any due diligence has been done to protect them.
I wouldn't say they caught it "by accident". They caught it the way that most organizations detect a compromise: they saw some suspicious network traffic and they investigated. They just happen to have among the most sophisticated investigation capability in the world.
From what I have read, the attacker attempted a second vector of infiltration, by e-mail. FireEye caught the second attempt, then noticed the SolarWinds activity.
> is that software security continues to be marketing theatre
Overall I tend to agree with your general points, but, this statement.
Would you say that seatbelts are "safety theatre" because some people die in car accidents while wearing a seatbelt?
I really don't think it's fair to attribute all software security as BS, especially given the resources state actors throw at breaking software. It's like arguing your seatbelts don't work because your 4 door sedan was hit by a bus.
The standard proviso about metaphors being given, I see your point. Seatbelts constitute an effort, seatbelts save lives, testing with dummies and survival results show it.
Seatbelts are real security measures. Still, more than 60K people die every year in motor accidents in North America and in the EU. Seatbelts cannot save life in all cases, such as outrageous speeds, but they are a sane measure for most -- if not all -- use-cases of driving.
In this compromise, the major purveyor of OS and security products itself was compromised. No company, no $agency did due diligence. Some of these victims are supposed to be in the business of high-assurance and due diligence.
None of the systems that we are conditioned to believe to be effective actually worked. And it was a standard use-case for all the victims.
An attack has multiple stages. Stuxenet's attack formula was:
inflitrate an airgapped network -> silently spread within -> silently destroy complex unique equipment
Solarwinds attack forumla seems to be:
compromise central infrastructure -> silently spread to customers via compromised updates -> silently exfiltrate useful data/create an advanced persistent threat (APT).
Because the initial compromise (oh a bad password, oh, a compromised remote code execution (RCE) engine) was not sophisticated this doesn't look sophisticated, but even stuxnet wasn't "sophisticated" in that sense, the initial attack vector was a USB stick in a parking lot, there's nothing sophisticated about that either. There is probably a lot of sophistication behind staying silent and the exfiltrating of data that we don't know about, even the silent exploration of multiple internal networks seems like a fairly non-trivial task to scale, especially to 18,000 potential targets, if more than a handful of them have been successfully compromised in a hard to resolve way, I would call that sophisticated.
The APT/Exfiltration work is much more close to the sophistication of the virus that attacked the centrifuges than the bad password. For all we know the hack could have levereaged information gained via kaspersky, telegram, or whatever other software to do sophisticated things much the same way we asked Siemens for their help.
We don't have enough information to make an assessment yet.
> the initial attack vector was a USB stick in a parking lot
That's not fair. It should sound like this: "the initial attack vector was a Windows 0day requiring 0 clicks, delivered by a USB stick in a parking lot."
People are saying that the solarwinds hack is not sophisticated because it is just a supply chain attack on a vendor with woefully insufficient security. I agree, that is not a very sophisticated sounding attack. What does appear to be potentially quite sophisticated is the post compromise activity.
“Our analysis indicates that these compromises are not self-propagating; each of the attacks require meticulous planning and manual interaction. Our ongoing investigation uncovered this campaign, and we are sharing this information consistent with our standard practice.”
You are saying my comparison is not fair because the USB stick actually had a 0 day which implies sophistication, but the supply chain build server was producing code and was also acting as a 0 day, in fact the actual payload appears to be significantly technically complex and custom.
Where do you draw the line between sophisticated and not?
USB in parking lot -> 0day -> horizontal movement 0days -> vertical movement 0days -> Payload delivery.
Bad admin credentials -> compromised build system -> supply chain 0day -> horizontal/vertical movement -> exfiltration of sensitive data.
Are we comparing:
[USB in parking lot -> 0day] to [Bad admin credentials -> compromised build system -> supply chain 0day] or
[USB in parking lot -> 0day] to [Bad admin credentials -> compromised build system] or
[USB in parking lot] to [Bad admin credentials -> compromised build system]
We don't know the extent and complexity of [horizontal/vertical movement -> exfiltration of sensitive data] in the solarwinds saga and therefore we have insufficient information to determine its sophistication or not. Meanwhile the vast majority of the people in this post have completely written off technical sophistication that occurs directly after the solarwinds software updates.
>silently exfiltrate useful data/create an advanced persistent threat
The chinese successfully pulled this off at Lockheed in the early 2000s
They would get lists of employees who attended various conferences, then email them a trojan saying "hey we met at conference X - here is something for you to click on"
And employees would fall for it.
The malware would TRICKLE out data very slowly so as not to be noticed.
They did this for a LONG time.
At the time, lockheed only had 3 egress points to the internet - and they had 110,000 laptops.
Another attack vector was the same as stuxnet - airgap; attack a supplier in taiwan, infect any USB stick put into the suppliers machines, then transfer once that USB stick was put into the Lockheed machine.
When the ruse was discovered - the chinese removed the throttle and they attempted to firehose out as much data as they could before they were cut off.
Stuxnet "worked by first causing an infected Iranian IR-1 centrifuge to increase from its normal operating speed of 1,064 hertz to 1,410 hertz for 15 minutes before returning to its normal frequency. Twenty-seven days later, the worm went back into action, slowing the infected centrifuges down to a few hundred hertz for a full 50 minutes. The stresses from the excessive, then slower, speeds caused the aluminium centrifugal tubes to expand, often forcing parts of the centrifuges into sufficient contact with each other to destroy the machine."
The solar winds hack is an otherwise unremarkable trojan that spread exclusively due to the bad security measures of solarwinds.
Stuxnet is a well understood attack from 10 years ago and its being compared with a poorly understood attack from 1 year ago.
>The solar winds hack is an otherwise unremarkable trojan that spread exclusively due to the bad security measures of solarwinds.
We don't have sufficient information to say what happened after the trojan landed is unsophisticated. All we have is the idea that microsoft benefits from making their attacker seem more competent than they are, which is true.
Sure, we could learn that they did something really impressive with the solarwinds hack that we have no idea about right now, and at that time it might be reasonable to compare it to stuxnet. The same could be said of any hack. However right now there is no evidence of sophistication anywhere near that scale, and thus calling it the largest and most sophisticated attack is clearly unwarranted, at least at this time.
> compromise a widely used package and sleep on it until it was pervasive enough to be a valuable hack.
Which makes me think: A bad actor could create a really good open source library or package, wait for everyone to use it and then introduce malware into it. Or they could "accidentally" add a security vulnerability and exploit it.
"There is another theory which states that this has already happened."
> A bad actor could create a really good open source library or package, wait for everyone to use it and then introduce malware into it.
This often happens with Chrome browser extensions that change hands and the new owner then injects malware or crypto mining or keylogging etc. into the newly acquired extension.
You're completely right, this wasn't a complicated attack at all. The two most notable items here are, Solarwinds has horrible security practices and those customers who were affected by the attack, also have horrible security practices. In a correctly secured environment it wouldn't have been possible for a infected Solarwinds server to connect out to a C2C server.
Yes, a secure server should not be able to make outgoing connections to arbitrary external machines on 80/443. If there's a specific need for a specific connection (e.g. the server needs to pull updates from the vendor) then that particular connection can be whitelisted.
SolarWinds is the most sophisticated hack by Russian/China/Nirth Korea. Stuxnet was the most sophisticated hack by the US/Israel.
The US does not need to hack SolarWinds because they probably could use a court order for the same outcome to distribute rigged binaries for political enemies.
It was the "largest and most sophisticated attack" that Microsoft has to make excuses for right now. (And that's regardless of their culpability or lack thereof.)
You are vastly underestimating the lengths that this attack when to avoid detection. They didn't just hijack Solarwinds updates and inject malware. They did things like concealing the commands of malware in legitimate looking Solarwinds packets. When they infiltrated a place, they had the malware sleep for months. They carefully avoid standard malware detection techniques and you can only do that if you have very sophisticated engineers. This isn't just some scripts someone pulled from the internet.
I guess nobody gets pwned by high school kids just screwing around anymore. Every hack is now the "most sophisticated ever" by the "most technologically advanced state actor ever" to break the "most secure six-character password ever."
Developed by network and systems engineers who know what it takes to manage today's dynamic IT environments, SolarWinds has a deep connection to the IT community.
The result? IT management products that are effective, accessible, and easy to use.
Solarwinds' former VP of Security posted a blog post entitled "Do Your Vendors Take Security Seriously?" [1] while, according to reports, being totally pwned from floor to ceiling.
They are vague because they want you to leave your contact details. Then a sales person will call you and gladly tell you everything you want to know and spam you with newsletters and marketing materials until eternity. That's B2B for you.
This is the case of the cobbler's children having no shoes, I think. If you're focused on your product, updating your website falls to the last priority (except for the marketing team).
Well the incentives arent there to blame those meddling kids and their dog.
Security companies benefit from having adversaries be big and scary
Hacked companies benefit from adversaries being big and scary. Nobody blames you if you get hacked by the russians. Getting hacked by teenagers looks bad.
Edit: that said, i feel like i should add, i personally dont feel, based on what we know, that this particular attack was a bunch of kids.
The supply chain compromise does not appear to be sophisticated. The "post installation of compromised software" activity seems to be somewhat sophisticated.
I find it curious that they would quote an executive on this. I doubt any of them have gotten their hands dirty on real code in years. Microsoft alone employs thousands of people more qualified to speak on the subject.
The SolarWinds incident was detect because of bad opsec by the operators who performed the FireEye op. I would image the capability was developed by an expert group in some intelligence agency, and then used as an entry point by a different operator group with lower standards.
But who is to day there aren't more of this kinds of attacks out there, just no one has made a foolish error using them yet? If we assume that, we have to assume this operation was somewhere in the middle of a normal curve of complexity, and there are even more sophisticated backdoored systems like that we just don't know about.
Imagine any medium-large code base (100+ of KLoCs), that is deployed widely, and has an auto update mechanism. Most companies don't have very strict access to the build process (and even if they do, all you need is to corrupt one employee), so it shouldn't be to hard to patch binaries before they are signed (especially bytecode in .NET and Java) , and add another URL and/or signature for verification (for sig only the attacker needs access to the web site/CDN too).
The change will be only a few lines, so is very hard to detect automatically - it will look like regular code for tools.
I just listened to this. Normally I like the A16Z podcast but this one was way below the normal standard. Just because they were organised, tested the exploit before they released it over a wider area it must have been a nation state?
It is as if a group of mates who codes a hack like this for a hobby/interest are disorganised have no experience in delivering software? Such ignorance for a self confessed expert. Scary.
I feel like this Solarwinds breach was a low effort, extremely high reward move, sort of like how Russia only had to spend 100k on ads in 2016 to get a massive reach on Facebook. I wouldn't think say this is the most sophisticated attack, this sounds more like they are embarrassed they were hacked for a year before they realized it and have to save face
> “I think from a software engineering perspective, it’s probably fair to say that this is the largest and most sophisticated attack the world has ever seen,” Smith said.
I mean, it's 100% going to happen again, and it was plainly obvious it was going to happen to begin with. We did a Black Hat talk about this (checks notes) 14 years ago, after being paid by a client to audit something like 12 different agent-based management systems:
Agent-based endpoint management is super convenient and is mainstream in modern IT management. But nobody grows up wanting to write the communications software for enterprise server inventory software, so the code quality on these products, which are ubiquitous, is awful. But 99% of enterprise buyers aren't even slightly motivated by software security when making purchasing decisions, and everyone knows it, so vendors buy fig-leaf audits from vendors who will sell public-facing documents for 2p1w engagements, and IT purchasers, whose performance reviews are based on completing projects and not protecting their companies from $500,000 purchase orders for what is effectively malware, accept those bogus reviews and get on with their lives.
Nothing is going to change about this, because nothing is going to alter the incentives. Reporters like Nicole Perlroth will try to blame it on ex-NSA hired guns (as if you needed the exploit chains NSA buys to break any of these systems), but I can't think of any realistic policy that could be applied to stop these kinds of attacks, not without massively disrupting the technology industry at the same time.
Best get used to it, is I guess my point. I mean that sincerely, not as a sort of appeal to right the ship; the ship is already upside down.
> but I can't think of any realistic policy that could be applied to stop these kinds of attacks, not without massively disrupting the technology industry at the same time
Why wouldn't Dan Geer's proposal to attach traditional products liability to closed source software improve the situation? Over time, source availability and reproducible builds should make this kind of thing a lot more difficult without wrecking anyone's budget. No?
That would work if we knew how to ship secure software at something resembling the cadence the industry demands, but we do not. We pretend to, and we get away with it because there isn't toothy liability attached to shipping bugs. We are all here, the software people on this site, the beneficiaries of that system.
Just very simple things, like reimplementing non-performance-sensitive C software from the 1990s and 2000s in simple memory-safe languages; it can't happen, the budget to make it happen would totally disrupt P&L at large companies; repeat with every well-known risk this kind of code is exposed to. I don't think we know how to solve this problem, which is why I tend to recoil from policy proposals to "solve" it.
What if the government directly paid the cost of reimplementing that old c software? If the market is failing here as it seems to be then perhaps the government should step in.
LOL nope. Humiliation works only on the level which target audience understands, like performance benchmarks. Security exploits are much more esoteric and tend to result in mudfights. It already did many times, like, how many businesspeople responsible for procurement have accurate understanding of spectre/meltdown/rowhammer vulns? Why do you think AMD isn't using it for marketing against Intel?
I think attacks like this show, that an intervention is very much needed.
And if it is to disrupt the technology industry big time, that might just be needed as well.
I mean you wouldn't have spared the tobacco industry, just because making them liable for lung cancer means disrupting the industry.
If the password for your "military grade" secured infastructure is literally "solarwinds123", I wouldn't say it's a sophisticated attack.
It's a more complete picture of how much modern businesses value security or pentesting audits. The answer to that is exactly zero effs.
As long as businesses think there is no ROI in security, this will stay the same.
The only thing sophisticated about this was that from code to deployment there wasn't any single audit of anyone. And that's a bad joke by any security measurement.
If any intruder can stay undetected THAT long, I don't wanna know what's up with MSFTs infrastructure. Must be an open door for everyone, and probably still uses the Perl pre-release based pipeline.
Initial compromise may not have been sophisticated or hard to achieve, the post compromise may have been somewhat sophisticated judging from the attackers operational security.
From casual reading alone, easily an order of magnitude, or two or three such orders, more sophisticated than the development and resources seemingly applied by SolarWinds.
Stuxnet seems to be far more sophisticated. I mean the entire idea of jumping an air gapped network was crazy but it worked.
Since Microsoft itself was compromised via SolarWinds angle I'd take the president's statement with a grain of salt and probably less objective than it would be otherwise.
“We have been actively looking for indicators of this actor and can confirm that we detected malicious SolarWinds binaries in our environment, which we isolated and removed,” the company said in a statement.
On "60 Minutes" they reported that the hardware in your computer is likely compromised as well, so new hardware will have to be bought.
This has a simple fix I've advocated for years.
Put the firmware for disk drives, USB sticks, embedded systems, etc., in ROM. Or at least provide a physical write-enable switch for updates.
I have no idea why people responsible for security do not demand this. I would expect them to be really sick and tired of "we have no idea if our firmware is infected or not, because we allow any piece of software to modify the firmware."
A literal turn-key switch on the case would work. Turn the key one way to enable firmware updates. Turn it the other to disable them. This should also be easy to replace and come as a standard sized device.
As an alternative, a 'security card' slot; similarly easily replaced or with an internal switch depressed re-'paired' to a new key.
Ceremony is part of the need for a key or a card. This needs to have weight to average, even sub-average, end users.
It must be out of the ordinary. The ceremony must have weight and differentiation from typical activities. True going inside and moving a jumper is fine for those willing to open the case, but it must work on Gradnma's Dell; while under warranty, without voiding the warranty.
293 comments
[ 5.0 ms ] story [ 54.2 ms ] threadWas the attack otherwise sophisticated and just relied on an easy entry point? So the breaking in was easy, but the plan to steal once inside was sophisticated?
The actual "hack" was a supply chain hijack, the actors inserted malicious dll code through a solarwinds authorized developer workstation, thus ensuring the code would be signed, and then pushed as a legitimate update to the downstream implementations of solarwinds.
The amount of effort and different techniques they've put in to remain undetected for months (years?) while infecting a lot of actors is pretty impressive.
https://www.sec.gov/Archives/edgar/data/0001739942/000162828...
The sophistication of the malware required state powers and assistance of the industrial company that manufactured the control systems. Those systems are not cheap and reverse engineering the control system software required detailed knowledge of the embedded hardware and software.
On December 8, 2020, the cybersecurity firm FireEye announced that red team tools had been stolen from it by what it believed to be a state-sponsored attacker.[106][107][108] FireEye was believed to be a target of the SVR, Russia's Foreign Intelligence Service.[27][109] FireEye says that it discovered the SolarWinds supply chain attack in the course of investigating FireEye's own breach and tool theft.[110][111]
After discovering that attack, FireEye reported it to the U.S. National Security Agency (NSA), a federal agency responsible for helping to defend the U.S. from cyberattacks.[1] The NSA is not known to have been aware of the attack before being notified by FireEye.[1] The NSA uses SolarWinds software itself.[1]
[1] https://en.wikipedia.org/wiki/Stuxnet
The malware keeps a "breadcrumb" trail of each machine it infects, and based on investigations by Symantec [1] it was determined that every Stuxnet sample from Natanz originated from outside suppliers.
1. https://tinyurl.com/1eswo98i
It was so sophisticated at not being detected that it went under everyone's radar for 5+ years.
Stuxnet behaved in exactly the same way.. neither did anything that would be detectable unless certain criteria was met and a secondary payload sent.
Stuxnet was very sophisticated to hit a relatively narrow target by comparison.
So I think they are both sophisticated, but not comparing them directly.
I would love to know how many machines got hit by SOlarwinds - however, its not about the machines - its about the value of the data it slurped... So even if it was one exchange server, with 10,000 treasury/nsa/whatever accounts, that one machines intel value is quite high.
But the concerted marketing effort pays dividends. Solarwinds is almost the only choice for government. For potential attackers its a big red arrow. Exploit a rent-seeking company that writes mediocre software and exists mostly to cash in on "best practice" lock-in with government contracts.
as far as i know nobodys really addressed the elephant in the room. Solarwinds is still a preferred government purchase for monitoring. every company that was affected by it still uses it (or at least hasnt publically refuted it) and no government official has come forward to admit they will discontinue it. Solarwinds hasnt offered any remediation or major changes in their development, leadership or code. just patch, rinse, and repeat and try not to pay too much attention to the issue.
Not accurate re: leadership. A new CEO just started in January, and their CTO was either terminated or he left just after the hack news.
Second, with such negligence? They deserve to be shut down. This isn't a highly sophisticated zero day exploit, there were multiple huge failure at Solarwinds that allowed the attack to happen.
Calmly, our CEO said: 'No. If there's anyone in this company who will never make that mistake again it's him.'
I was in the room when the CEO told that (one of the nice things about small companies) and have had contact with the person responsible for a few years until he got another job. True story ;)
There were changes though: the 'two pair of eyes' principle was enforced a lot stricter from then on.
A far better system is "no fault" where the company has incentive to be open about problems and finding solutions.
I think the elephant in the room is actually a more general issue that is cross platform and independent of the product implementation.
It’s 2021 and we are still ignoring the fundamental “best practice” that we’ve known about for at least 20 years.
Systems should be isolated from each other unless there is an overwhelming need for them to be connected and everything should run with the least privilege.
Centralised credential vaults and binary artefact repositories are an obvious example.
The two computers are developed by two independent teams who are not allowed to talk to each other. Two different CPUs, two different algorithms, two different programming languages.
The idea, of course, is a design problem with one does not propagate to the other.
If one program is used in all your infrastructure, all your infrastructure is compromised when that program has a bug.
As a marine saying goes, never take two chronometers with you to the sea. Take either one or three.
1) Build servers: Maybe by keeping it possible to build one's software on a laptop — then, when it's time to release something, one can build the software at the build server plus some people's laptops and compare hashes.
2) ???
3) Different people who don't talk with each other, do security code review, independently of each other? Hmm does that make sense
And we should have checks built into the build pipelines and checksum the files that go into the build. We have Merkle trees for a long time. I think, it would be possible to detect injected code.
On April 28th 2021 they will announce Q1 2020 results. THAT's when we'll know for sure how bad this hurt them.
They may release less than sanguine guidance on Feb 25th, we'll see.
The reason why I know this stuff I will leave as an exercise to the reader. :)
But Microsoft's "system of trust" was undermined. The attacker was even inside Microsoft and Azure. No anti-virus, no "defender", no amount of basic or advanced telemetry caught this.
FireEye alone caught it... By accident.
The real "elephant in the room" is that software security continues to be marketing theatre. Closed-source software should not be trusted and consumers should not believe that any due diligence has been done to protect them.
Overall I tend to agree with your general points, but, this statement.
Would you say that seatbelts are "safety theatre" because some people die in car accidents while wearing a seatbelt?
I really don't think it's fair to attribute all software security as BS, especially given the resources state actors throw at breaking software. It's like arguing your seatbelts don't work because your 4 door sedan was hit by a bus.
Seatbelts are real security measures. Still, more than 60K people die every year in motor accidents in North America and in the EU. Seatbelts cannot save life in all cases, such as outrageous speeds, but they are a sane measure for most -- if not all -- use-cases of driving.
In this compromise, the major purveyor of OS and security products itself was compromised. No company, no $agency did due diligence. Some of these victims are supposed to be in the business of high-assurance and due diligence.
None of the systems that we are conditioned to believe to be effective actually worked. And it was a standard use-case for all the victims.
The APT/Exfiltration work is much more close to the sophistication of the virus that attacked the centrifuges than the bad password. For all we know the hack could have levereaged information gained via kaspersky, telegram, or whatever other software to do sophisticated things much the same way we asked Siemens for their help.
We don't have enough information to make an assessment yet.
That's not fair. It should sound like this: "the initial attack vector was a Windows 0day requiring 0 clicks, delivered by a USB stick in a parking lot."
https://arstechnica.com/information-technology/2020/12/18000...
“Our analysis indicates that these compromises are not self-propagating; each of the attacks require meticulous planning and manual interaction. Our ongoing investigation uncovered this campaign, and we are sharing this information consistent with our standard practice.”
https://www.fireeye.com/blog/threat-research/2020/12/evasive...
You are saying my comparison is not fair because the USB stick actually had a 0 day which implies sophistication, but the supply chain build server was producing code and was also acting as a 0 day, in fact the actual payload appears to be significantly technically complex and custom.
Where do you draw the line between sophisticated and not?
Are we comparing: We don't know the extent and complexity of [horizontal/vertical movement -> exfiltration of sensitive data] in the solarwinds saga and therefore we have insufficient information to determine its sophistication or not. Meanwhile the vast majority of the people in this post have completely written off technical sophistication that occurs directly after the solarwinds software updates.The chinese successfully pulled this off at Lockheed in the early 2000s
They would get lists of employees who attended various conferences, then email them a trojan saying "hey we met at conference X - here is something for you to click on"
And employees would fall for it.
The malware would TRICKLE out data very slowly so as not to be noticed.
They did this for a LONG time.
At the time, lockheed only had 3 egress points to the internet - and they had 110,000 laptops.
Another attack vector was the same as stuxnet - airgap; attack a supplier in taiwan, infect any USB stick put into the suppliers machines, then transfer once that USB stick was put into the Lockheed machine.
When the ruse was discovered - the chinese removed the throttle and they attempted to firehose out as much data as they could before they were cut off.
Stuxnet "worked by first causing an infected Iranian IR-1 centrifuge to increase from its normal operating speed of 1,064 hertz to 1,410 hertz for 15 minutes before returning to its normal frequency. Twenty-seven days later, the worm went back into action, slowing the infected centrifuges down to a few hundred hertz for a full 50 minutes. The stresses from the excessive, then slower, speeds caused the aluminium centrifugal tubes to expand, often forcing parts of the centrifuges into sufficient contact with each other to destroy the machine."
The solar winds hack is an otherwise unremarkable trojan that spread exclusively due to the bad security measures of solarwinds.
>The solar winds hack is an otherwise unremarkable trojan that spread exclusively due to the bad security measures of solarwinds.
We don't have sufficient information to say what happened after the trojan landed is unsophisticated. All we have is the idea that microsoft benefits from making their attacker seem more competent than they are, which is true.
Which makes me think: A bad actor could create a really good open source library or package, wait for everyone to use it and then introduce malware into it. Or they could "accidentally" add a security vulnerability and exploit it.
"There is another theory which states that this has already happened."
This often happens with Chrome browser extensions that change hands and the new owner then injects malware or crypto mining or keylogging etc. into the newly acquired extension.
A thousand engineers is a huge project. You have no better knowledge than Microsoft about the attack.
The US does not need to hack SolarWinds because they probably could use a court order for the same outcome to distribute rigged binaries for political enemies.
You are vastly underestimating the lengths that this attack when to avoid detection. They didn't just hijack Solarwinds updates and inject malware. They did things like concealing the commands of malware in legitimate looking Solarwinds packets. When they infiltrated a place, they had the malware sleep for months. They carefully avoid standard malware detection techniques and you can only do that if you have very sophisticated engineers. This isn't just some scripts someone pulled from the internet.
https://www.solarwinds.com/
We’re Geekbuilt.®
Developed by network and systems engineers who know what it takes to manage today's dynamic IT environments, SolarWinds has a deep connection to the IT community.
The result? IT management products that are effective, accessible, and easy to use.
[1] https://www.solarwindsmsp.com/blog/do-your-vendors-take-secu...
This is every website of any company selling to enterprise customers. It's incredibly frustrating as a reader.
I imagine anytime someone actually answers the question "Yes, but what do you do?", they get fired.
Security companies benefit from having adversaries be big and scary
Hacked companies benefit from adversaries being big and scary. Nobody blames you if you get hacked by the russians. Getting hacked by teenagers looks bad.
Edit: that said, i feel like i should add, i personally dont feel, based on what we know, that this particular attack was a bunch of kids.
https://www.fireeye.com/blog/threat-research/2020/12/evasive...
No high school kid is doing that.
https://a16z.simplecast.com/episodes/solarwinds-anatomy-of-h...
It is as if a group of mates who codes a hack like this for a hobby/interest are disorganised have no experience in delivering software? Such ignorance for a self confessed expert. Scary.
The word “seen” is doing a lot of work there.
(Think, probably, and fair, are doing some too.)
https://web.archive.org/web/20061215050427/http://www.matasa...
Agent-based endpoint management is super convenient and is mainstream in modern IT management. But nobody grows up wanting to write the communications software for enterprise server inventory software, so the code quality on these products, which are ubiquitous, is awful. But 99% of enterprise buyers aren't even slightly motivated by software security when making purchasing decisions, and everyone knows it, so vendors buy fig-leaf audits from vendors who will sell public-facing documents for 2p1w engagements, and IT purchasers, whose performance reviews are based on completing projects and not protecting their companies from $500,000 purchase orders for what is effectively malware, accept those bogus reviews and get on with their lives.
Nothing is going to change about this, because nothing is going to alter the incentives. Reporters like Nicole Perlroth will try to blame it on ex-NSA hired guns (as if you needed the exploit chains NSA buys to break any of these systems), but I can't think of any realistic policy that could be applied to stop these kinds of attacks, not without massively disrupting the technology industry at the same time.
Best get used to it, is I guess my point. I mean that sincerely, not as a sort of appeal to right the ship; the ship is already upside down.
Why wouldn't Dan Geer's proposal to attach traditional products liability to closed source software improve the situation? Over time, source availability and reproducible builds should make this kind of thing a lot more difficult without wrecking anyone's budget. No?
Just very simple things, like reimplementing non-performance-sensitive C software from the 1990s and 2000s in simple memory-safe languages; it can't happen, the budget to make it happen would totally disrupt P&L at large companies; repeat with every well-known risk this kind of code is exposed to. I don't think we know how to solve this problem, which is why I tend to recoil from policy proposals to "solve" it.
Humiliating the competition is good marketing.
Were any in use at all? No
So, nothing really sophisticated... just exploiting the bad design of existing operating systems.
This will continue for the foreseeable future.
It's a more complete picture of how much modern businesses value security or pentesting audits. The answer to that is exactly zero effs.
As long as businesses think there is no ROI in security, this will stay the same.
The only thing sophisticated about this was that from code to deployment there wasn't any single audit of anyone. And that's a bad joke by any security measurement.
If any intruder can stay undetected THAT long, I don't wanna know what's up with MSFTs infrastructure. Must be an open door for everyone, and probably still uses the Perl pre-release based pipeline.
I would not be surprised to find hair saloons with better security practices than our military.
Pretty sure texans would not like to be caled balkans, even though balkan culture is somewhat similar to texan
Since Microsoft itself was compromised via SolarWinds angle I'd take the president's statement with a grain of salt and probably less objective than it would be otherwise.
“We have been actively looking for indicators of this actor and can confirm that we detected malicious SolarWinds binaries in our environment, which we isolated and removed,” the company said in a statement.
https://en.wikipedia.org/wiki/2020_United_States_federal_gov...
[1] https://www.microsoft.com/security/blog/2020/12/18/analyzing...
This has a simple fix I've advocated for years.
Put the firmware for disk drives, USB sticks, embedded systems, etc., in ROM. Or at least provide a physical write-enable switch for updates.
I have no idea why people responsible for security do not demand this. I would expect them to be really sick and tired of "we have no idea if our firmware is infected or not, because we allow any piece of software to modify the firmware."
As an alternative, a 'security card' slot; similarly easily replaced or with an internal switch depressed re-'paired' to a new key.
It must be out of the ordinary. The ceremony must have weight and differentiation from typical activities. True going inside and moving a jumper is fine for those willing to open the case, but it must work on Gradnma's Dell; while under warranty, without voiding the warranty.