They'll be somewhat networked at least together and if the operators want any convenience they'll be on a network with most other systems just so they can be monitored more easily.
Most control systems are not airgapped, because most people don't think of themselves as targets of that level of attack. Stuxnet was targeting national security infrastructure, which is much more likely to be airgapped, but your local powerplant doesn't (at present) think of themselves as national security infrastructure, so they don't take the same level of precautions.
Note: this only describes cases I'm familiar with, in the US. I bet that some countries with more experience on the receiving end of cyber-warfare (e.g. Ukraine) are better.
The airgap question comes up every time an industrial system is attacked. The thing is that these days all systems are connected via several layers to the outside world. Because a modern oil refinery is of no use if you cannot track what product is being made at that moment. And the people tracking the data are often in completely different locations or even different countries.
The main line of defense is these days is "layers of an onion" network with (physical) controls such that data easily can get out to a higher layer but that it is very difficult to get in from a higher layer back down into a deeper layer.
A completely airgapped network is not practical anymore because the alternative is even worse: nobody wants to have dozens or 100ths of operators, maintenance engineers and 3rd party contractors running around the facility with usb sticks because there is no network to move stuff around. If you have a network then you can control where the data comes into your network, who copies the data to where and what data is visible for which user.
It constantly horrifies me that they don't use data diodes for this systems like this.
Let data flow one way from the secure industrial equipment out to the general use network for monitoring, but you'd still have to go to a machine on the secure side to make any changes.
In a lot of industrial sites software security is a joke. Embedded systems tend to use very old, well proven technology, which in itself isn't a problem, it fits the market well, but the side effect is that security isn't always properly considered as it wasn't a concern when the software/hardware was developed.
I was involved in a project a few years ago delivering a series of monitoring systems running Windows XP to a brand new 700 million dollar oil rig. This was at the request of the client, they had software they needed that would only run on Windows XP. They had a fit when we had trouble sourcing Windows XP licenses. The expectation is that these systems will have a 20 - 30 year life.
It used to be that keeping every air gapped was enough, but organizations want easier monitoring, so more systems are being networked in an ad-hoc way without a lot of thought about security.
I expect we are going to see more things like this happening in the future until we start taking security in systems / embedded space more seriously. And even then there will be exploits of older systems for years afterwords since the replacement cycle is so long.
I wonder what a secure embedded system even looks like when I think about it. The environment isn't suitable to the kind of continuous patching that is done in the web world, but exploits will be found and dependencies will need to be updated. How do you square keeping things up to date with stringent testing requirements in systems that can kill people. Many of these systems / plants are unique, there is only one plant like it in the world, so testing becomes very hard.
How are mechanical components tested against requirements in critical systems? What is the process for changing those components, like upgrading pipes to a fancy new composite?
In my mind, we'll start treating silicon the same way... formal verification, rigorous real-world testing, trusted suppliers, and an expectation that change is slow, expensive, and risky.
Seems like there is a huge opportunity here for a startup that can navigate the industry and manage to solve some of these problems. There are huge players in the space, but from what I've seen they aren't solving these problems very effectively.
You're right, but my experience suggests that anything broadly in the 'manufacturing' field is treated as a cost center, and not something people want to spend money on.
In most places a rubber stamp of 'yep, definitely secure because we have a password (over http, oh by the way everyone uses the default which is in the manual)' is good enough. Then when something goes wrong there's a general shrug and they change the password.
I've friends who work as instrument technicians/engineers on embedded systems in manufacturing and have worked in Europe, UK and Australia in food manufacturing, mechanical, water collection and water processing.
The one constant I've heard is that all of their hardware is almost or is out of support, when it breaks, they expect band aid fixes and ironically none or very little of them can accept any downtime. There's no hardware redundancy for their production lines and when anything breaks, it's all hands on deck. Yet there's no funding going back into the production lines to pro-actively repair or minimise their risks. Manufacturing it seems to me is 100% a reactive industry.
The industries above work on incredibly small margins of profit and the sheer expense to outfit and refit these aging, decrepit (but still working!) production lines are quite honestly, massive.
These manufacturers won't invest in these engineering faults (whether it's security or production focused) until they've been fucked.
NB: This might be with the exception of Lego, my cousin who got employed by Lego after finishing his masters in Industrial Design & CS, and after reading and watching some articles on Lego. I'm convinced Lego's margins are a lot larger than most. They might be the closest thing to a FAANG company when it comes to investment in phu7sical engineering and manufacturing.
How are these places insured against industrial accidents?
I wonder if the way to get your foot in the door would be to partner with an insurer who would provide discounts to the client if they installed security technology meeting a certain standard. You could come up with some kind of bump on the wire type Linux device that proxied access to the old insecure system bringing them up to the standard. Then sell it as a return on investment through savings on industrial accident insurance premiums.
From what I've heard, one of the problems here is that Windows 10's support of parallel ports (widely used in automation) is really bad, and there's no way that live updates are going to be acceptable on something that needs to be running 24/7. What ways are there to properly air-gap something like this?
And of course once nobody's using Windows, attackers will pivot to target whatever everyone's switched to in its place. And they'll find holes. They always do.
We're talking about the difference between barely usable and unusable here. No system in the world has ever been slimefest of the scale of Windows. Even those which are deployed more than Windows.
It is much to easy to blame Windows for everything. If this is another TRITON level attack then it did not matter what operating system is used, the attackers would have come in anyway. With enough resources you can break anything.
Check out the DEFCON 2018 talk "Through the Eyes of the Attacker" to see to what lengths the TRITON attackers went. These guys were dumping eeproms from boards running obscure MIPS processors and were looking at raw ethernet packets in Wireshark and flipping bits in the packets to see what would happen. That their command&control was running on Windows was just coincidence.
I'll never understand selecting windows for any embedded system that is expected to have a 10+ year life. If you look at their history, they frequently blow up the development environment every 5-6(?) years. Yes they'll provide support, but it comes at being increasingly held hostage to whatever level of long term support costs are in vogue at the time. For a 20-30 year support life, that's a guaranteed headache.
You don't install any updates or change anything once the machine is setup and performing its duty. If the hardware fails you replace it with similar hardware and install the original software from CD again. I see windows NT on ships, windows 2000 in power stations, etc.
Improper maintenance of worn out components can cause a failure resulting in a crash, be they computer components, electrical components, or mechanical components.
Linux doesn't necessarily help here. Ubuntu LTS lifetime is 3-5 years, others are similar; outside this, distro maintainers think nothing of just dropping packages from the distro that the deployment may be relying on, so upgrading is not simple.
The difference is that with Linux you can self-support, or pay for someone else to do it, since the whole source code is available to everyone and the licence allows it.
Also, a better comparison would be with RHEL, which has a lifetime of 10 years or more.
I once encountered a guy who was setting up systems so that you could control a water treatment plant from your ipad at home. His attitude was, "Modbus on one side, ethernet on the other, what could possibly go wrong?" Lots, I told him. A lot of things could go wrong.
having setup many control systems over the years I can confirm that the state of the art is that bad. it's amazing just how blazé they are about security. also, terrifying.
>Also, all water meters are being replaced so they can be read remotely without a human physically using an electronic meter reader.
Those are extremely widespread already. And they're pretty open. I wouldn't personally be too worried about any potential exploits since they're simply broadcast only systems and the worst that could probably happen is your'd get a jacked up water bill.
And this is precisely why we can never consider nuclear power to be “safe”.
It’s just not worth the risk exposure. The worst case failure modes must be expected to occur, and they must be economically and ecologically acceptable when they do.
The idea that “this can theoretically happen but we promise it won’t” is simply not acceptable. Versus, “this is extremely unlikely to occur because of these numerous counter-measures, but when it does here’s what we do and what it will cost us.”
If you can do the later analysis on a nuclear plant and come away satisfied, then build baby build.
Yeah I used to work at a nuclear plant. They intentionally use all analog systems currently for fear of this. Almost nothing digital in the whole plant
51 comments
[ 3.3 ms ] story [ 46.1 ms ] threadNote: this only describes cases I'm familiar with, in the US. I bet that some countries with more experience on the receiving end of cyber-warfare (e.g. Ukraine) are better.
The main line of defense is these days is "layers of an onion" network with (physical) controls such that data easily can get out to a higher layer but that it is very difficult to get in from a higher layer back down into a deeper layer.
A completely airgapped network is not practical anymore because the alternative is even worse: nobody wants to have dozens or 100ths of operators, maintenance engineers and 3rd party contractors running around the facility with usb sticks because there is no network to move stuff around. If you have a network then you can control where the data comes into your network, who copies the data to where and what data is visible for which user.
Let data flow one way from the secure industrial equipment out to the general use network for monitoring, but you'd still have to go to a machine on the secure side to make any changes.
I was involved in a project a few years ago delivering a series of monitoring systems running Windows XP to a brand new 700 million dollar oil rig. This was at the request of the client, they had software they needed that would only run on Windows XP. They had a fit when we had trouble sourcing Windows XP licenses. The expectation is that these systems will have a 20 - 30 year life.
It used to be that keeping every air gapped was enough, but organizations want easier monitoring, so more systems are being networked in an ad-hoc way without a lot of thought about security.
I expect we are going to see more things like this happening in the future until we start taking security in systems / embedded space more seriously. And even then there will be exploits of older systems for years afterwords since the replacement cycle is so long.
I wonder what a secure embedded system even looks like when I think about it. The environment isn't suitable to the kind of continuous patching that is done in the web world, but exploits will be found and dependencies will need to be updated. How do you square keeping things up to date with stringent testing requirements in systems that can kill people. Many of these systems / plants are unique, there is only one plant like it in the world, so testing becomes very hard.
In my mind, we'll start treating silicon the same way... formal verification, rigorous real-world testing, trusted suppliers, and an expectation that change is slow, expensive, and risky.
In most places a rubber stamp of 'yep, definitely secure because we have a password (over http, oh by the way everyone uses the default which is in the manual)' is good enough. Then when something goes wrong there's a general shrug and they change the password.
I've friends who work as instrument technicians/engineers on embedded systems in manufacturing and have worked in Europe, UK and Australia in food manufacturing, mechanical, water collection and water processing.
The one constant I've heard is that all of their hardware is almost or is out of support, when it breaks, they expect band aid fixes and ironically none or very little of them can accept any downtime. There's no hardware redundancy for their production lines and when anything breaks, it's all hands on deck. Yet there's no funding going back into the production lines to pro-actively repair or minimise their risks. Manufacturing it seems to me is 100% a reactive industry.
The industries above work on incredibly small margins of profit and the sheer expense to outfit and refit these aging, decrepit (but still working!) production lines are quite honestly, massive.
These manufacturers won't invest in these engineering faults (whether it's security or production focused) until they've been fucked.
NB: This might be with the exception of Lego, my cousin who got employed by Lego after finishing his masters in Industrial Design & CS, and after reading and watching some articles on Lego. I'm convinced Lego's margins are a lot larger than most. They might be the closest thing to a FAANG company when it comes to investment in phu7sical engineering and manufacturing.
I wonder if the way to get your foot in the door would be to partner with an insurer who would provide discounts to the client if they installed security technology meeting a certain standard. You could come up with some kind of bump on the wire type Linux device that proxied access to the old insecure system bringing them up to the standard. Then sell it as a return on investment through savings on industrial accident insurance premiums.
Check out the DEFCON 2018 talk "Through the Eyes of the Attacker" to see to what lengths the TRITON attackers went. These guys were dumping eeproms from boards running obscure MIPS processors and were looking at raw ethernet packets in Wireshark and flipping bits in the packets to see what would happen. That their command&control was running on Windows was just coincidence.
Very few adversaries have enough resources to break anything. The threshold for roughing up a Windows setup is the lowest in the industry.
Also, a better comparison would be with RHEL, which has a lifetime of 10 years or more.
Also, all water meters are being replaced so they can be read remotely without a human physically using an electronic meter reader.
Those are extremely widespread already. And they're pretty open. I wouldn't personally be too worried about any potential exploits since they're simply broadcast only systems and the worst that could probably happen is your'd get a jacked up water bill.
You have unpaid WATER BILL of -2147483648 dollars.
Report immediately to INCINERATOR145 for processing.
Please have a kindly day,
ROBOTOVERLORD69420
Worst case you can tamper with the readings, but you can't actually cause damaging effects.
https://www.shodan.io/search?query=inventory+port%3A%2210001...
They take commands to change certain values.
Article spends too much time FUDing "plant explosions" for my taste
Because hacking systems to cause explosions would be unheard of?
https://www.telegraph.co.uk/news/worldnews/northamerica/usa/...
It’s just not worth the risk exposure. The worst case failure modes must be expected to occur, and they must be economically and ecologically acceptable when they do.
The idea that “this can theoretically happen but we promise it won’t” is simply not acceptable. Versus, “this is extremely unlikely to occur because of these numerous counter-measures, but when it does here’s what we do and what it will cost us.”
If you can do the later analysis on a nuclear plant and come away satisfied, then build baby build.