9 comments

[ 6.2 ms ] story [ 32.1 ms ] thread
Sadly this instantly became a political football, with the states pointing fingers at Iran, but Trump was not wrong in this case. This is gross incompetence at all levels — IT malpractice if you will.

CISA and its predecessors have been warning utility operators about critical infrastructure vulnerabilities for what, 15 years at this point? That goes back to the first Obama administration.

Yet here we are in 2026 and these utilities are still connecting these things to the raw Internet with default passwords. You cross a threshold where you're being deliberately careless.

When you are putting more effort into securing your Plex server on your home network then public utilities are taking on machinery that dumps chemicals into the local water, something is not right and finger-pointing isn't going to fix it.

> Censys ARC identified 4,148 Internet-exposed hosts that respond to EtherNet/IP and self-identify as Rockwell Automation/Allen-Bradley. The United States remains dominant at 71.0% (2,945 hosts), with Canada a clear second at 11.5% (476 hosts).

Describe the network security of the industrial automation industry and their customers in a single statement. Lol.

And how many more are on the same internal networks as dozens or hundreds of ordinary Windows desktops on which municipal workers are checking their email? Hardly better.
The numbers are a bit lower if you exclude honeypots (~2k hosts): https://www.shodan.io/search/report?query=rockwell+port%3A44...

And the problem has actually gotten better over the years:

https://trends.shodan.io/search?query=tag%3Aics+rockwell

The situation used to be worse with things like the Lantronix password recovery service (i.e. a UDP port that would just send you the device password without any auth). It's still not ideal and takings things offline isn't easy (https://blog.shodan.io/taking-things-offline-is-hard/) but it's getting better (slowly).

Some people allegedly say that the US should treat heads of companies like they do in China, when found guilty of certain crimes. Allegedly.
There can be surprisingly few critical components and processes in a water and sewerage network due to gravity-fed designs (far more cost efficient), large buffers (water reservoirs, sewage sumps, etc) allowing intermittent operation of otherwise critical components, retained ability for humans to manually operate equipment, and also the fields of availability and safety engineering which typically prefer elimination of software failure modes by designing equipment to not rely upon software.

The aim of a water network is to:

1. Take water from a water source (elevated dam -- strongly preferred, river, ocean) and as much as possible, gravity feed it to a treatment plant.

2. Treat the water using processes that are simplified/fail-safe as much as possible and could be operated manually by humans if necessary. This is where availability and safety engineers would design equipment to not be dependent on software and instead use mechanical or analogue electronics control.

3. Pre-position treated water as much as possible at ~50-90m hydraulic head (~500-900kPA) above the water faucets where people want to use the treated water and provide a buffer for X days of usage. Any pumps between the treatment plant and elevated storage therefore only need to operate intermittently to refill the buffer.

Sewerage networks have similar aims:

1. Let sewage flow as much as possible downhill to the treatment plant via gravity. Where a rising main (elevation gain) is required, place a large enough sump for X hours/days of usage and pump up to higher elevation from the sump.

2. Treat the sewage using processes that are simplified/fail-safe as much as possible and could be operated manually by humans if necessary. For example, a compressor used for aeration can be manually switched on/off with a mechanical switch and plugged into a diesel generator, and not require someone logging in with multi-factor authentication to a laptop to issue a command to a PLC to turn on the compressor.

3. Design overflows into the system for emergency release of partially or untreated sewage, and practice this process as part of disaster recovery exercises. This is generally an aim arising due to risk assessment process that says building a $1bn sump with 8 independent pumps is cost prohibitive versus the 1-in-200 year chance of untreated sewage messing up a downstream river for a few weeks.

Ultimately a lot of the cybersecurity risk comes down to government appetite to accept 1-in-1000 (or whatever) year failure modes. Is it worth investing now in triple modular redundant automated control systems (mostly used in safety-critical sectors such as aviation and space), or installing a just-in-case diesel generator at every one of 500 pumping stations across a region, or building $10bn of sewage sumps to hold sewage for up to a month, or building 2 treatment plants instead of 1 and using different technology for each, or hiring and training more humans to regularly exercise manual control and operation of a network, etc? Or just accept that once every 1000 years, some water rationing may be required, or a downstream river will be polluted for a few weeks?

The playbook, per federal officials, was crude but effective: attackers remotely accessed internet-facing operational technology, changed device IP addresses and passwords, and locked utility operators out of their own monitoring and control systems, NBC News reported. The PSA is now pleading with utilities to do the bare minimum - pull programmable logic controllers off the open internet and put them behind gateways and firewalls, use actual passwords, and restrict which devices are allowed to talk to one another.