Working for a company in Germany which is planing production 3 months in advance using printed Excel sheets. The migration of ERP system gone wrong and nobody knows how to fix it. Production management tries to hide this fact and does not talk to the engineering department. This will go for years, consultants will gather their fees for non functional system. Obviously IT infrastructure is not needed for manufacturing. It is just nice to have.
Working for a manufacturing company, you may be making drone parts, and you don’t really know which side of the war you’re making them for, because they can buy individual parts through different reputable-looking companies.
You could also be making surgical parts that help save lives.
Overall though, I think I’d rather be making nice practical furniture that hopefully people would never throw away. While I support people that want to protect, war is horrible.
Without software, drones are useless. I suppose they can still assemble manually operated quadcopters if they know their inventory by heart, but they will be unable to produce more parts by 3D printing or drones capable of stable flight, autonomous operations, surveillance or any more advanced use cases. Even remote control is probably out of the picture.
Reminds me of the Louvois[1] disaster in the French armed forces, they fucked the payment system so bad they had to roll back to manual accounting for a moment. Yes, for the whole French Army, which was at the time involved in multiple foreign operations…
> So the foreign intelligence services gave them a button push so it's not a direct cyber war on Russia.
What foreign intelligence services ? Also if you think there isn't a constant barrage of attacks coming from everyone, you're not ready for the real world.
I remember Steve Gibson saying some years back that the only reason USA doesn't (cyber-)'attack' Russia's train infra is the inability to 'hide the traces' of the attack, and it would be 'easily' attributed/mapped back to the USA causing (political) issues. Well, Ukraine doesn't have 'that' challenge.
On the other hand (and I'm not defending a drone company), anyone that has a business should know by now that ransomware (with our without deletion) is a real thing, and it's not an 'if' question, it's a 'when' question.
I have never worked with/for a Russian company, so it would be interesting to hear/read from someone who has, how 'well organized' are they? GRC-wise. Assuming that someone would run the COBIT framework on them (Russian companies), would the 'average' be 'ok' or it's a big mess (kinda like working for an EU company in early 00's)?
I did work for a Russian financial multinational just before COVID-19, as a native Russian speaker, and it was a free-for-all mess interally. The IT side had a load-bearing, old-school sysadmin type with a personality for heroics.
Obviously if there was opportunity for a supply chain attack like that they would have done it in addition to wrecking the IT infrastructure. Regardless, I imagine this will impact day to day drone operation as the Russians might re-flash the firmware to a known good version whether there is a backdoor or not
Not many companies explicitly prepare for the scenario where every single data storage unit in the company is effectively wiped and you have to redeploy from zero.
If you never bootstrap from zero (nor simulate this) then your systems probably have cycles in their deployment dependencies. Your config pusher is deployed from Jenkins/Puppet/Ansible but 2 years ago someone made Jenkins dependent on the config pusher for its own config. Now you cannot just deploy these systems in order, you have to replay the history before that change.
That happened to company I am familiar with a year ago. The main storage cluster,that everything depended on died. They recovered by deploying everything again from dev laptops.
This reminds me of troubles in a parallel universe.
Construction industry have products with typical lifetime of 50+, in some cases multiple hundreds. Computing and digitalization are hot topic now and for the past several decades with various buzzwords (probably 'digital twins' is the newest one) however when I am unable to open construction design files made in the beginning of my career less than 30 years ago due to obsolescence for various reasons then all those efforts seem for nothing eventually beyond immediate needs. Good old outdated 2D drawings seen as unfeasible practice might save the day in the future (... perhaps, assuming that current pdf files could still be opened some decades down the line, as that is a common 'digital paper' approach nowadays, actual physical world paper are used less and less).
Reminds me of compiler bootstrapping. If you only have the source code of, let's say, Rust and want to build it you first need to have a Rust compiler of a previous version. And for that you need another compiler. Then you reach times when rustc was written in OCaml. And know what? Now you need to build OCaml compiler!
That's a very odd website. Blocked by the Russian government so you get a TLS error, once you get past that, you get the Cloudflare "you are blocked" page, and then you use a VPN and... get the option to read this article in Russian.
The linked page is in English, but speaking of the option - residents of Russian Federation probably were not a target audience of Russian version of this website anyway. Contrary to some popular beliefs in Russia about language wars, many people still do speak Russian in Ukraine and media do publish articles in Russian.
One day I decided to change my main disk and used the opportunity to rebuild everything from scratch and from backups. I was up in about an hour.
And then I spent a week fixing this and that, ah yes I changed that too and, crap, I cannot remember why this thingie is set up this way. And some more.
This is a one-man lab, with simple services, all on docker. I also work in IT.
Recovering from scratch a whole infrastructure managed by many people over the years is a titanic task.
I helped to recover my nearby hospital as a volunteer when it was ransomwared. The poor two IT guys over there has no idea how to recover and the official help was pityful.
I also helped with a ransomware attack on a large company. The effort people had to do to remember why something was that way, or just remember whatever was colossal. Sure a lot of things were "documented" and "tested" but reality hit hard.
Modern IT practices don’t really contemplate disaster recovery. Even organisations with strict backup procedures seldom test recovery (most never at all).
Everything is quickly strapped together due to teams being understaffed. Preparing infrastructure in a way such that it can easily be recreated is easily twice the effort as “just” setting it up the usual way.
Say what you want about European financial organizations but they are legally obliged to practice their recovery strategies. So every other month production clusters with all user data get teared down in one cloud region and set up in another one over night. This works surprisingly well. I guess they would never do that without the legal requirements.
This is why documenting is so crucial. Even on a software architecture level.
A few months from now, I'd love to have written down decisions for my current project:
- Why did I decided to use Kysely over Drizzle, Knex, Prisma, TypeORM or other ORM/SQL tool?
- How am I going to do migrations?
- Why am I using one of Deno/Bun over sticking to nodejs?
- Why did I structure the project as a directory per feature over controllers/models/services directories?
- Why did I fork this library and what are the steps to keep this thing updated? Do I plan to upstream my changes? Is there a GitHub issue or PR about it?
- Why am I hosting in one of AWS/GCP/Azure? Why not lambda functions? Why docker?
- Why did I pick this specific distribution of kubernetes over the other also lightweight alternatives?
- Why did I even start this project and what do I aim to accomplish with it?
So I created a # Decisions section in README.md
This way I don't keep doubting my own decisions and wasting time opening 20 documentation tabs to compare solutions yet again.
On the other hand, I’ve worked in places where the total destruction of IT (so as to start again from a clean slate) was within the Overton window of options for how to transform the business.
> Recovering from scratch a whole infrastructure managed by many people over the years is a titanic task.
Half of the work is to know what you need, the other half is to know how you do it, while the third half is to cope with all the undocumented tinkering which happened along the way. So in that regard, starting from scratch can be acceptable, as long you are not starting from zero, and can build up on the knowledge and experience of the previous run(s). I mean, there is a whole gaming-genre about this, which is quite popular. And usually you have the benefit that you might be able to fix some fundamental failures which you had to ignore because nobody wanted to take the risk.
I had to rebuild a significant percentage of my homelab after my house was raided by the police and they took about $10k-worth of my gear; desktop, laptop, NAS, hard drives.
However, because in a previous life I'd been responsible for backups and involved in disaster recovery planning I was already kind of prepared with:
- a mirrored on site copy of backups (that they either didn't find or chose to leave behind)
- older hardware that had once been performing the duties of the existing seized gear (I'm a bit of a hoarder, I like repurposing or keeping for just such an occasion)
- multiple off site backups
- pretty good documentation of my setup
I was back up and running within a day or two and had lost maybe a couple of days of data. And it's a home lab, so nothing super important anyway, but a (not really) nice resilience test.
It also gave me the experience to work out a few structural changes to further limit the impact of an event that takes out a bunch of processing and storage.
(After 8 months they told me to pick up all my gear, they found nothing, but thanks for traumatising my kids)
In the 1990s mainframes got so stable and redundant that some were not rebooted in over a decade - they could even upgrade the kernel without rebooting. Then one company had a power failure andthe backup generators failed. When the power came back it was months before they figured out everything it was doing and then how to start that service where the guy who started it originally quit years ago.
most companies started rebooting the mainframe every six months to ensure they could restart it.
Chamath's new company 80/90 is targeting this pain. Large firms often have no idea what their software is trying to do. Rebuilding it is cheaper and leads to better software.
> I helped to recover my nearby hospital as a volunteer when it was ransomwared.
I'm curious about how you got in the door here. Very cool, but isn't healthcare IT notoriously cagey about access? I've had to do PHI training and background checks before getting into the system at my (admittedly only 2) PHI-centered jobs.
Granted, if it was such an emergency, I could see them rushing you through a lite version of the HR onboarding process. Did you have a connection in the hospital through whom you offered your services?
If you are professionally responsible for infrastructure, and you haven't thought very hard about the "how do we rebuild from scratch" case, you're committing professional malpractice.
The problem is environments like hospitals, who'll "cost center" their IT department to death, where even the most seasoned pro has no chance to ever do the right thing. There should be liability at the board level. There never will.
>This is a one-man lab, with simple services, all on docker. I also work in IT.
TBH your mistake was only running one layer of virtualization. What I do on my home setup is run a docker in a VM in a VM in a docker in a docker in a VM in a docker in a VM in a VM in a docker. This, I feel, ought to be the minimum level of indirection and virtualization in any technical configuration in perpetuity. Anything less is bush league and prone to errors.
I've worked in tech all my life, and long ago learned how important it is to be impeccably diligent in documenting build processes whenever creating, deploying or adjusting new architecture.
Now it's simply become part of my engineering hygiene - as natural and effortless as brushing my teeth.
Actually drilling your DR is also crucial. If you never put it to the test, your documentation isn't worth the paper it's printed on.
In fact the last few years I've been thinking about ways for these systems to rebuild themselves on a continuous basis. Eg. I'd love a smartphone that competely restores itself from backup every night, even to brand new identical hardware, including secure element artifacts (either via private keys I securely control or reregistering everything in an automated fashion), with no user-noticeable impact.
I wonder to what extent either side is worried about the firmware on the drones. Somehow getting tampered-with firmware onto the drones that your enemy is using seems like it would be valuable.
AI translation (to English) is off in places. "Ukrainian cybercriminals" is not in the original and was picked as the translation of the closest sounding full word.
> The attack destroyed over 47 TB of critical data,
I'm very dubious that there would be such an amount of “critical” data pretty much anywhere, besides the banking and insurance sector. And particularly not at a drone manufacturer.
The source of this news is the Ukrainian military which seems to exaggerate and spread propaganda (as does every other country of course). I don't know why we accept this information as reliable.
Here is a translated version of the telegram message posted by the hacking team:
> LLC “Gaskar Integration” (Gaskar Group)—one of the largest UAV manufacturers in Russia—has just been penetrated right down to the tonsils in the course of demilitarization and denazification.
> VO Team, together with the Ukrainian Cyber Alliance (https://t.me/UCAgroup) and another very well‑known organization whose mere mention makes the vatniks’ bottle‑openings burst (https://gur.gov.ua/), carried out large‑scale operations: we seized all of Gaskar Group’s network and server infrastructure, gathered valuable data on their current and prospective UAVs, destroyed that data, and knocked the entire infrastructure offline.
> By the way, from the information we obtained, the PRC is helping Gaskar Group with production and staff training. China transferred technology for the newest UAVs—technology that is now in our hands .
> VO Team focused on wiping out the production complex’s infrastructure. On‑site we erased more than 250 hosts (4 ESXi servers, 46 virtual servers, 200+ workstations) and bricked about 20 MikroTik devices. In total we destroyed 47 TB of valuable data at Gaskar Group—including 10 TB of backups—and disabled all production and auxiliary systems.
> The scum at Gaskar Group have the blood of hundreds of Ukrainian children, women, and elderly on their hands. That’s why we went after this target with special zeal. We now possess the lists of ALL employees, their home addresses, information about their family members, and much more… We’re in your home computers and phones—we’re everywhere . Not a single bastard from Gaskar Group will escape responsibility!
> The sword of Damocles already hangs by a thin thread over your heads. It’s too late to spew excuses like “we’re apolitical” or “we were just making money”….
> The whole world can see that the so‑called Russian Federation has strategically lost everything. Defeat and collapse of that unwashed entity are only a matter of time. VO Team is collecting data on everyone involved in Putin’s criminal war—the deaths of our children, mothers, and all Ukrainians. Retribution is inevitable and is drawing near!
Cyber warfare is really reaching a new peak in Ukraine, and not just the more-traditional cyberattacks like this. The target is of note; the drones themselves are the thing that's setting this war apart from all of the primarily human powered wars of the past centuries.
Drones have revolutionized reconnaissance, sabotage, and munitions interception. Relative to their material cost, they can be terrifically destructive, and with the advances in image recognition in the past decade some are able to operate even when affected by electronic signal jamming. This is some very cyberpunk shit going on right now.
This was obviously a very high-value target, and Ukraine has shown themselves again to be masters of asymmetric warfare: taking out a sizable chunk of Russia's long range bombers using drones smuggled across Russia, and now impacting one of the centers of Russia's drone manufacturing. It is difficult to see how the war will end, but it is clear that Ukraine is not about to stop fighting.
I am working for a medium sized Swiss company. We're coding our own ERP, based on a nightmare of a stack. We call it "security by confusion". An attacker would maybe find its way in, but he'll never find the way out. If he destroys 90% of our code, we'll still be up and running, because 95% of the codebase is obsolete.
There was an interview with some U.S. army rep. that was pretty famous a few years back. He said that even if hackers broke in to the U.S. systems they wouldn't achieve anything due to how old, convoluted and buggy the systems were - even we who have full access can barely get anything to work!!
Of course, that was said purposefully, and may not reflect the truth at all.
But still, the thought that that is security for some (and I am sure it is for someone, somewhere) is kind of an extension of security by obscurity that is scary.
79 comments
[ 3.2 ms ] story [ 83.4 ms ] threadYou could also be making surgical parts that help save lives.
Overall though, I think I’d rather be making nice practical furniture that hopefully people would never throw away. While I support people that want to protect, war is horrible.
Is this sarcasm?
I swear this is SAPs main business model
Just endless consulting bills to set it up then fix it when it's delivered in a broken state.
[1]: https://fr.wikipedia.org/wiki/Logiciel_unique_%C3%A0_vocatio...
I was acquired by a company that was working on Sales Force integration for 3 years, I left before it was fully functional.
They had 4 full time devs working on Sales Force, meanwhile we had built the entire company in a year with 4 devs.
So the foreign intelligence services gave them a button push so it's not a direct cyber war on Russia.
Do you have any evidence that it was foreign intelligence services?
What foreign intelligence services ? Also if you think there isn't a constant barrage of attacks coming from everyone, you're not ready for the real world.
On the other hand (and I'm not defending a drone company), anyone that has a business should know by now that ransomware (with our without deletion) is a real thing, and it's not an 'if' question, it's a 'when' question.
I have never worked with/for a Russian company, so it would be interesting to hear/read from someone who has, how 'well organized' are they? GRC-wise. Assuming that someone would run the COBIT framework on them (Russian companies), would the 'average' be 'ok' or it's a big mess (kinda like working for an EU company in early 00's)?
They should have checked the source codes and added some changes to make drones unpredictably unreliable
"Oh this totally innocent code change? Oh look it makes the gps act weird if longitude is between a certain range how weird"
If you never bootstrap from zero (nor simulate this) then your systems probably have cycles in their deployment dependencies. Your config pusher is deployed from Jenkins/Puppet/Ansible but 2 years ago someone made Jenkins dependent on the config pusher for its own config. Now you cannot just deploy these systems in order, you have to replay the history before that change.
Construction industry have products with typical lifetime of 50+, in some cases multiple hundreds. Computing and digitalization are hot topic now and for the past several decades with various buzzwords (probably 'digital twins' is the newest one) however when I am unable to open construction design files made in the beginning of my career less than 30 years ago due to obsolescence for various reasons then all those efforts seem for nothing eventually beyond immediate needs. Good old outdated 2D drawings seen as unfeasible practice might save the day in the future (... perhaps, assuming that current pdf files could still be opened some decades down the line, as that is a common 'digital paper' approach nowadays, actual physical world paper are used less and less).
Other way is to build a stripped down version of rustc only capable of compiling latest rustc, e.g. using https://github.com/dtolnay/bootstrap
Great post on bootstrapping and its problems: https://bootstrappable.org/
Actual project capable of bootstrapping Linux system from scratch: https://github.com/fosslinux/live-bootstrap
- know your threats
- assess your risks based on identified threats
- backup 3-2-1 strategy (3 copies of your data on 2 independent storage places with 1 copy offline and offsite)
- "build the world from scratch" plan with the assumption that all infra is completely and irreversibly destroyed.
- assume you have already been hacked but you don't yet know about it. Build your indicators of compromise based on that simple assumption.
Observing how some "groups of people" act in a totally ignorant fashion is amusing.
https://archive.ph/jg9Mg
Somebody saved it four hours ago.
One day I decided to change my main disk and used the opportunity to rebuild everything from scratch and from backups. I was up in about an hour.
And then I spent a week fixing this and that, ah yes I changed that too and, crap, I cannot remember why this thingie is set up this way. And some more.
This is a one-man lab, with simple services, all on docker. I also work in IT.
Recovering from scratch a whole infrastructure managed by many people over the years is a titanic task.
I helped to recover my nearby hospital as a volunteer when it was ransomwared. The poor two IT guys over there has no idea how to recover and the official help was pityful.
I also helped with a ransomware attack on a large company. The effort people had to do to remember why something was that way, or just remember whatever was colossal. Sure a lot of things were "documented" and "tested" but reality hit hard.
Everything is quickly strapped together due to teams being understaffed. Preparing infrastructure in a way such that it can easily be recreated is easily twice the effort as “just” setting it up the usual way.
A few months from now, I'd love to have written down decisions for my current project:
- Why did I decided to use Kysely over Drizzle, Knex, Prisma, TypeORM or other ORM/SQL tool?
- How am I going to do migrations?
- Why am I using one of Deno/Bun over sticking to nodejs?
- Why did I structure the project as a directory per feature over controllers/models/services directories?
- Why did I fork this library and what are the steps to keep this thing updated? Do I plan to upstream my changes? Is there a GitHub issue or PR about it?
- Why am I hosting in one of AWS/GCP/Azure? Why not lambda functions? Why docker?
- Why did I pick this specific distribution of kubernetes over the other also lightweight alternatives?
- Why did I even start this project and what do I aim to accomplish with it?
So I created a # Decisions section in README.md
This way I don't keep doubting my own decisions and wasting time opening 20 documentation tabs to compare solutions yet again.
How did they prevent threat actors presenting themselves as volunteers, were you vetted?
Half of the work is to know what you need, the other half is to know how you do it, while the third half is to cope with all the undocumented tinkering which happened along the way. So in that regard, starting from scratch can be acceptable, as long you are not starting from zero, and can build up on the knowledge and experience of the previous run(s). I mean, there is a whole gaming-genre about this, which is quite popular. And usually you have the benefit that you might be able to fix some fundamental failures which you had to ignore because nobody wanted to take the risk.
However, because in a previous life I'd been responsible for backups and involved in disaster recovery planning I was already kind of prepared with:
- a mirrored on site copy of backups (that they either didn't find or chose to leave behind)
- older hardware that had once been performing the duties of the existing seized gear (I'm a bit of a hoarder, I like repurposing or keeping for just such an occasion)
- multiple off site backups
- pretty good documentation of my setup
I was back up and running within a day or two and had lost maybe a couple of days of data. And it's a home lab, so nothing super important anyway, but a (not really) nice resilience test.
It also gave me the experience to work out a few structural changes to further limit the impact of an event that takes out a bunch of processing and storage.
(After 8 months they told me to pick up all my gear, they found nothing, but thanks for traumatising my kids)
- daily backup locale + remote (blackbaze with 60 readonly retention strategy, separated bucket by service)
- monthly offline backup
- a preprod server where my users can restore entiere environment for testing purpose (CI)
in case of full house fire, i can be back online in an working day.
PS: i have only some TBs of data so quite easy to do.
most companies started rebooting the mainframe every six months to ensure they could restart it.
I'm curious about how you got in the door here. Very cool, but isn't healthcare IT notoriously cagey about access? I've had to do PHI training and background checks before getting into the system at my (admittedly only 2) PHI-centered jobs.
Granted, if it was such an emergency, I could see them rushing you through a lite version of the HR onboarding process. Did you have a connection in the hospital through whom you offered your services?
The problem is environments like hospitals, who'll "cost center" their IT department to death, where even the most seasoned pro has no chance to ever do the right thing. There should be liability at the board level. There never will.
TBH your mistake was only running one layer of virtualization. What I do on my home setup is run a docker in a VM in a VM in a docker in a docker in a VM in a docker in a VM in a VM in a docker. This, I feel, ought to be the minimum level of indirection and virtualization in any technical configuration in perpetuity. Anything less is bush league and prone to errors.
Now it's simply become part of my engineering hygiene - as natural and effortless as brushing my teeth.
Actually drilling your DR is also crucial. If you never put it to the test, your documentation isn't worth the paper it's printed on.
In fact the last few years I've been thinking about ways for these systems to rebuild themselves on a continuous basis. Eg. I'd love a smartphone that competely restores itself from backup every night, even to brand new identical hardware, including secure element artifacts (either via private keys I securely control or reregistering everything in an automated fashion), with no user-noticeable impact.
I'm very dubious that there would be such an amount of “critical” data pretty much anywhere, besides the banking and insurance sector. And particularly not at a drone manufacturer.
> LLC “Gaskar Integration” (Gaskar Group)—one of the largest UAV manufacturers in Russia—has just been penetrated right down to the tonsils in the course of demilitarization and denazification.
> VO Team, together with the Ukrainian Cyber Alliance (https://t.me/UCAgroup) and another very well‑known organization whose mere mention makes the vatniks’ bottle‑openings burst (https://gur.gov.ua/), carried out large‑scale operations: we seized all of Gaskar Group’s network and server infrastructure, gathered valuable data on their current and prospective UAVs, destroyed that data, and knocked the entire infrastructure offline.
> By the way, from the information we obtained, the PRC is helping Gaskar Group with production and staff training. China transferred technology for the newest UAVs—technology that is now in our hands .
> VO Team focused on wiping out the production complex’s infrastructure. On‑site we erased more than 250 hosts (4 ESXi servers, 46 virtual servers, 200+ workstations) and bricked about 20 MikroTik devices. In total we destroyed 47 TB of valuable data at Gaskar Group—including 10 TB of backups—and disabled all production and auxiliary systems.
> The scum at Gaskar Group have the blood of hundreds of Ukrainian children, women, and elderly on their hands. That’s why we went after this target with special zeal. We now possess the lists of ALL employees, their home addresses, information about their family members, and much more… We’re in your home computers and phones—we’re everywhere . Not a single bastard from Gaskar Group will escape responsibility!
> The sword of Damocles already hangs by a thin thread over your heads. It’s too late to spew excuses like “we’re apolitical” or “we were just making money”….
> The whole world can see that the so‑called Russian Federation has strategically lost everything. Defeat and collapse of that unwashed entity are only a matter of time. VO Team is collecting data on everyone involved in Putin’s criminal war—the deaths of our children, mothers, and all Ukrainians. Retribution is inevitable and is drawing near!
Drones have revolutionized reconnaissance, sabotage, and munitions interception. Relative to their material cost, they can be terrifically destructive, and with the advances in image recognition in the past decade some are able to operate even when affected by electronic signal jamming. This is some very cyberpunk shit going on right now.
This was obviously a very high-value target, and Ukraine has shown themselves again to be masters of asymmetric warfare: taking out a sizable chunk of Russia's long range bombers using drones smuggled across Russia, and now impacting one of the centers of Russia's drone manufacturing. It is difficult to see how the war will end, but it is clear that Ukraine is not about to stop fighting.
Of course, that was said purposefully, and may not reflect the truth at all.
But still, the thought that that is security for some (and I am sure it is for someone, somewhere) is kind of an extension of security by obscurity that is scary.