Who else would be paying for it…? Genuinely, if a government entity has costs (even costs incurred by their own foolishness), who else should pay for it? Are you suggesting the councillors/admin staff should pay for it out of pocket? What other entity works like that?
What if the workers get more competent as the result of the attack? What if the worker competence's effectiveness was limited by the organizational structure?
Incompetence is just a fact of life, not a moral failure.
If you hire incompetent people, you can't reasonably expect them to do their jobs properly. You may save some money and avoid some inconvenience that way, but you may also have to face crises that would have been avoidable. Alternatively, you can spend more money proactively and hire more competent people. Either way, taxpayers will pay the price.
Yes, a county in the south Texas desert with fewer residents than a mid-size office park should just up and fire the one person in the county who could kind of do the job and was willing to do so each year for probably what a FAANG senior staff engineer makes each month.
This county, by the way, was bankrupted from having to deal with the corpses of dead migrants who were desperate enough to try to sneak in through it.
I empathize with the instinct to point out a correct solution in a space you specialize in, but as another commenter pointed out, this is a county of about 5000 adults, with a likely very small government. It's very possible that they were under-resourced for this kind of incident, and under-resourced to have a rock-solid backup solution in place.
Given the usual IT chops of a small local government, I give them points for having backups at all. Funding from higher levels of government (either state/provincial or federal depending on the region) earmarked for anti-ransomware initiatives, could help mitigate this sort of disaster in the future.
It seems that state level governments should be providing the setup needed to run a county. These small places have no hope of operating a government software stack on their own.
Yup, that's my current opinion. Counties can be extremely well-funded if they encompass a city, and may have almost no funding at all if they're rural. The needs of each county government are obviously different as well, but there's still a bunch of fixed costs that take up a disproportionately large part of smaller counties' budgets.
There's more to backing up public and government records than a big hard drive and someone who can set up rsync.
Because they're government records, there's the very good chance that there are compliance/certification requirements, plus there's infrastructure you have to put in to make it auditable while mitigating the risk of a ransomware attack affecting backups, too.
Setting up total, compliant, and working backups for a small county can reasonably require a small headcount and some specialized long-term planning.
E: that's not to mention that in some lower-population, lower-income counties, $400 isn't trivial to spend. There are definitely places in the States where deciding whether to upgrade a single computer from 2006 for $800 is a long conversation.
As for spending $800, how is spending $37,000 more feasible?
I find these sorts of exchanges astonishing. A 1Tb drive can be bought for $30 on ebay. Backing up to it is a simple xcopy or rsync command. Then put the drive in the safe.
Ok, how often do you take it out of the safe to do periodic backups?
How's access to the safe managed? What happens if you get a vindictive (ex-)employee with access to it? If the person with access to the safe loses the key, or gets in an accident and can't open it anymore, what do you do?
How do you audit access to the safe to make sure nobody's pulling down data and selling it for profit on the side?
How do you make sure you don't lose all your backups by having a butterfingers IT person drop the drive?
How sure are you that the latest backup doesn't contain files hit by ransomware, or worse yet, the ransomware itself?
If you're storing financial or medical data about the county's citizens, how are you going to handle the possibility of being legally challenged down the line about data storage practices?
Handling any data in an org of almost any size is surprisingly complicated, especially if it's other peoples' data. Most people don't realize the scale of the issue because they're coming from a place of mostly working with at-home backups.
In terms of spending feasibility: really depends on how funding gets allocated by the state. Like I said in another comment, the solution would be for the state to subsidize the fixed costs of putting this kind of infrastructure in place.
Have you ever worked with small city governments? Usually the people in charge of IT were assigned or put there, not because they have or know any IT skills. Not to mention where and how would they put a 16T backup drive in something newish and back up everything.
I worked for a company that got crypto-locked. We had backups. We had tested that representative backups and restores worked; do you test restore every backup you take? We did not test that restores worked across all affected machines concurrently in a reasonable amount of time.
We did end up restoring some large percentage of machines from backup, eventually, then after retrieving/reviewing data we trashed those machines and rebuilt them. We also ended up paying - some of the backups didn't work, and some were taking so long to restore it didn't make sense to wait for them - and we rebuilt those as well.
At least for us, we probably could have done a better job with testing backups. The behavior we saw with individual test restores was not the same as the what happened with mass restores, and we were surprised by that. Also, beyond just restoring from backup, we spent a lot of time rebuilding machines; if you get compromised, your backup might be clean, but the restored state is probably ready to be compromised again. I don't remember how long it was before our security people said they figured out how we got compromised in the first place.
> A recent ransomware attack on Brooks County’s Justice of the Peace and district courts, and finance department, cost it more than $37,000.
> “We had determined if we didn’t, then it would take us anywhere from six months to a year to reconstruct our software program,” Ramos said.
How much would it have cost for 6-12 months of engineer time plus the court system slowing to a crawl in the meantime? Thank god these hackers don't know how to negotiate.
> Thank god these hackers don't know how to negotiate.
I think that's a feature, not a bug. They have exemplified that if they get paid, you get your data back. No negotiating, no games, just a hard time limit with consequences and a relatively low payout cost to avoid them.
This is some tiny town, not surprised it was that low really. Someone here could have an idea how much it might cost. They said it was their financial software and it was the only one boy backed up. Which means it’s probably ancient. Idk much about municipal software.
Buying new software is one thing you could price out. Losing historical financial data is the one that’s tricky to price.
Knowing the state of municipalities in Texas, it was probably some legacy Tyler Technologies database. Maybe even one of the versions that still used COBOL for backend storage.
Source: worked for a Texas municipality in a previous career life, but definitely not Brooks County
That name is giving me nightmares. I divorced in a Texas county I no longer live in, and at some point, some agency would not accept the copy of the divorce decree I had from the original action because it said "Copy - Original signed by judge" on the last page, and they wanted something with that signature. I ended up in a nightmare loop trying to find how to request records, where I could look up my case and see the link, but not use it without registering, but if I registered and logged in, my case no longer showed up in their search at all, making it so I could see my case record was clearly there, but I had no way to retrieve it.
>There are several different platforms out there that can shut a machine off if it becomes infected, and then keep all the other machines safe from the machine that is infected... Ramos said this is the type of software the county purchased after the attack, to protect the county’s server
So I'm not an expert by any means, but I can't picture how that is an effective protection. Either the detection can catch it before infection, in which case why shut down? Or it catches it after infection when it could have already spread.
Setting up that last backup sounds like a better use of money.
Yeah it sounds like a glorified manual kill switch, I doubt it’s automated in a meaningful way. If it was it’d be easy to bypass unless they figured out phishing AV and are keeping it a secret.
But I guess they thought it was worth buying to appease the public.
Every time a hospital system pays off hackers (which happens all the time)... I assure you the ransom payment is also mostly coming out of your tax dollars.
For all of us shocked shocked about the county’s preparation and response, it’s worth noting that this is a county of population ~7000 and only about 5000 adults.
The number of people involved in government, and the resources available for professional support staff (and solution products) are probably modest to say the least.
A lot of you probably went to high schools that were bigger and more funded.
Does it mean that like a corporate M&A, maybe some cities/counties could also indulge in M&A for better value to citizens? Because with bigger base you can afford more employees dedicated to specific tasks.
While it might be politically unpalatable, it's appalling to me that there aren't any standards for government websites. With ~3000 counties and I don't know how many municipalities, trying to synthesize data between different local governments is a nightmare for journalists, citizens, legal actors, and who-all else. Political science types like to mention 'laboratories of democracy' with a misty-eyed gaze into the past, but frankly it's far more common to end up with meth labs of bad governance, and the idea that great software solutions for the public sector are going to bubble up of their own accord has been shown to be wrong. There needs to be a system that is open source, accessible, and standard enough that city managers have to come up with a decent excuse as to why they're not using it.
For a country that gave birth to nearly all of the technologies the world is using, seeing the U.S. response to covid in terms of vaccination proof was an absolute shock. My American friends were walking around Europe with a piece of paper saying they got vaccinated. So yeah if you guys can't get your shit together and create a centralized vaccination certification system amid a global pandemic, a standard website kit isn't happening either.
You think that’s bad, wait until you hear about our federal identification system…
I think people from other countries don’t understand that Americans don’t want (federal) government knowing anything about them besides taxes. There’s a whole bunch of reasons why, but for the most part it boils down to various forms of liberty - and tales of the mark of the beast.
… but it’s usually fine when your state or a corporation is doing it.
The uk (at least at the moment) has no federal ID card a bit like the US. Much like in the US it’s a point of contention when voter ID requirements are being mooted. That being said they managed to roll out a vaccine passport pretty easily.
> I think people from other countries don’t understand that Americans don’t want (federal) government knowing anything about them besides taxes
I'm not American, I know that, have heard various reasons for it, and i consider it to be profoundly stupid. It might have made sense in the 19th century, but today? Your federal government already knows all there is to know about you. They can tap in healthcare, financial, state government information, all that online stuff. The cat is out of the bag, why not use it for something practical? A common unified ID which gives access to e-government services is the norm across the EU, and it's pretty cool and practical.
I do not find it stupid at all. I would prefer to live a life without a government, that issues proof of identity for me. And rather have a working economy, where the problem of ID independent of government is solved. I know, it is convenient to have a single persona, that i can use everywhere and they are interoperable. But i AM. That is proof enough. It does not need a government for the proof of that, imho.
> And rather have a working economy, where the problem of ID independent of government is solved
That sounds profoundly stupid. You mean like the credit rating agencies that provide credit worthiness identity, but where you have no control of how and with whom they store your data? So how would that work, how would the companies providing that service make money and how could they prove your identity? They'd still need to use government info like birth certificate/drivers license to verify who you are, but they'd have a profit motive. Of course that sounds better than the government, which doesn't need to sell who you are, providing an identity service.
Why have a useless middleman trying to make money off you?
Just take the government and a centralized ID out of the equation and it becomes an interesting problem. And why do i need a big guy, who hands out an ID with a silly watermark, prooving, that it is me? We have families, we have offspring, we have friends, we have relations, that we build over time, this should be proof enough?
> We have families, we have offspring, we have friends, we have relations, that we build over time, this should be proof enough?
So fuck any homeless, people living alone, orphans, etc. ? Not to mention the utter unreliability of relying on random people (who claim they know you) for information like place/date of birth, etc.
Yep. And i am not saying its a bad or good thing to do it that way. Just that it is one solution to a problem. Another could be, that people in need are part of their communities, people know you, and there is a charity, where people pay in and they organize themselves, to pay out benefits for the less lucky. Many things were tried out in the past, so other ideas may come up in the future. But it doesnt always need a centralized government, a big guy, far, far away.
I think the big difference there is in what capacities for violence exist within these two entities. One has a monopoly on legal violence and is armed (government) the other has no such powers (corporations). You want to limit how much power you give to an entity that has a legal monopoly on violence. Behind this desire to limit power is the recognition of the sad recurring reality of what is known as democide. Also on the table here is the distinction between tyranny, trending tyranny, and potential tyranny. If you’re not a member of a society in the state of potential tyranny then it may be too late to reverse the trend. So the question arises what safeguards and obstacles can be set in place that keep people in a state of peace and potential tyranny and how do you upset things from devolving into trending tyranny. These are pretty deep questions that I’m starting to scratch the surface in learning about.
> Really the federal government should be publishing and distributing guidance to local governments about how to run IT.
Or even running the IT for them. Lots of smaller municipalities and counties, even with all the guidance on the world, wouldn't have the resources to properly manage everything. Not to mention what a colossal waste it is, financially or practically, for every single administrative entity to reinvent the wheel for each of their IT needs.
I'm only shocked that this tiny district didn't have backups. Even schoolgirls do have backups.
Maybe they should learn from Zambia and restore their data from their backup, and tell the ransomware guys some nice things, just as the central bank of Zambia did.
37,000$ really is not a huge amount.. surprised how low this amount is. Securing the systems and hiring more IT support services .. would have cost way more money honestly .
Perhaps ransomware is evolving to fine just right price point niche?
I've felt like they aren't asking for just under the repair cost, they're asking for just under the "governments get serious about stopping this" cost.
And that is 90% of the actual demand for bitcoin. The rest was speculation, and that is going away as prices fall. So imagine how bitcoin would crash if people stopped paying these ransoms.
This may be a very dumb crypto / ransomware question, but if someone knows the answer, I'd appreciate it.
Why don't businesses (or systems) seed their drives with files with known text / content and then use those files to reverse the method used to encrypt? It seems like having an adversary encrypt a set of known "canary" files should provide information to reverse the encryption?
Again, there may be a good reason (or many many good reasons) why this would not be a good solution, especially since I'd expect most OS installations have enough standard files to do this if it worked, but I am curious if someone knows.
Edit: From the helpful comments, this is a known class of attack on a cryptosystem called a plaintext attack. Using that information, I looked into how ransomware systems address this attack, and several, apparently, use per-file keys as, in part, a defense against this type of attack.
What you're suggesting is known as a known-plaintext attack. All modern and properly implemented cryptosystems are designed to be highly resistant to it.
Your edit is still a bit confused. Any halfway decent encryption algorithm and mode can encrypt terabytes of data and still be resistant to known plaintext attacks. You can pretty safely use a single key to encrypt an entire hard-drive after all - that's how Bitlocker and LUKS, etc work. You don't need per file keys for encryption security or defense against known plaintext - it gives more flexibility in selectively decrypting though -- ie pay me a little money and I can prove I can return some of your shit, pay more if you want the rest.
“The only data that we had that wasn’t backed up was in our auditor's office, where we have our financial software,”
Seems to me that in modern times failing to back things up is incompetence and dereliction of duty. The auditor should be billed for at least part of the payment.
86 comments
[ 0.30 ms ] story [ 164 ms ] threadMaybe /this/ is the real web3.
If you hire incompetent people, you can't reasonably expect them to do their jobs properly. You may save some money and avoid some inconvenience that way, but you may also have to face crises that would have been avoidable. Alternatively, you can spend more money proactively and hire more competent people. Either way, taxpayers will pay the price.
This county, by the way, was bankrupted from having to deal with the corpses of dead migrants who were desperate enough to try to sneak in through it.
> “The only data that we had that wasn’t backed up was in our auditor's office, where we have our financial software,” Ramos said.
Seems like it should’ve been a priority.
(a component of my work in risk mgmt is business continuity and disaster recovery, excuse my saltiness)
Given the usual IT chops of a small local government, I give them points for having backups at all. Funding from higher levels of government (either state/provincial or federal depending on the region) earmarked for anti-ransomware initiatives, could help mitigate this sort of disaster in the future.
A 16T drive costs around $400. Shouldn't be so expensive to copy the contents of the disk to it now and then. No special software is needed, either.
Because they're government records, there's the very good chance that there are compliance/certification requirements, plus there's infrastructure you have to put in to make it auditable while mitigating the risk of a ransomware attack affecting backups, too.
Setting up total, compliant, and working backups for a small county can reasonably require a small headcount and some specialized long-term planning.
E: that's not to mention that in some lower-population, lower-income counties, $400 isn't trivial to spend. There are definitely places in the States where deciding whether to upgrade a single computer from 2006 for $800 is a long conversation.
As for spending $800, how is spending $37,000 more feasible?
I find these sorts of exchanges astonishing. A 1Tb drive can be bought for $30 on ebay. Backing up to it is a simple xcopy or rsync command. Then put the drive in the safe.
It's as simple as that.
How's access to the safe managed? What happens if you get a vindictive (ex-)employee with access to it? If the person with access to the safe loses the key, or gets in an accident and can't open it anymore, what do you do?
How do you audit access to the safe to make sure nobody's pulling down data and selling it for profit on the side?
How do you make sure you don't lose all your backups by having a butterfingers IT person drop the drive?
How sure are you that the latest backup doesn't contain files hit by ransomware, or worse yet, the ransomware itself?
If you're storing financial or medical data about the county's citizens, how are you going to handle the possibility of being legally challenged down the line about data storage practices?
Handling any data in an org of almost any size is surprisingly complicated, especially if it's other peoples' data. Most people don't realize the scale of the issue because they're coming from a place of mostly working with at-home backups.
In terms of spending feasibility: really depends on how funding gets allocated by the state. Like I said in another comment, the solution would be for the state to subsidize the fixed costs of putting this kind of infrastructure in place.
We did end up restoring some large percentage of machines from backup, eventually, then after retrieving/reviewing data we trashed those machines and rebuilt them. We also ended up paying - some of the backups didn't work, and some were taking so long to restore it didn't make sense to wait for them - and we rebuilt those as well.
At least for us, we probably could have done a better job with testing backups. The behavior we saw with individual test restores was not the same as the what happened with mass restores, and we were surprised by that. Also, beyond just restoring from backup, we spent a lot of time rebuilding machines; if you get compromised, your backup might be clean, but the restored state is probably ready to be compromised again. I don't remember how long it was before our security people said they figured out how we got compromised in the first place.
> “We had determined if we didn’t, then it would take us anywhere from six months to a year to reconstruct our software program,” Ramos said.
How much would it have cost for 6-12 months of engineer time plus the court system slowing to a crawl in the meantime? Thank god these hackers don't know how to negotiate.
Buying new software is one thing you could price out. Losing historical financial data is the one that’s tricky to price.
Source: worked for a Texas municipality in a previous career life, but definitely not Brooks County
So I'm not an expert by any means, but I can't picture how that is an effective protection. Either the detection can catch it before infection, in which case why shut down? Or it catches it after infection when it could have already spread.
Setting up that last backup sounds like a better use of money.
But I guess they thought it was worth buying to appease the public.
The number of people involved in government, and the resources available for professional support staff (and solution products) are probably modest to say the least.
A lot of you probably went to high schools that were bigger and more funded.
Really the federal government should be publishing and distributing guidance to local governments about how to run IT.
I am wondering about how the ransom payout is going to show up in next year's budget. "Miscellaneous"?
Off topic, but there is a documentary about migration through Brooks County and the perils faced by people trying to migrate: https://en.wikipedia.org/wiki/Missing_in_Brooks_County
I think people from other countries don’t understand that Americans don’t want (federal) government knowing anything about them besides taxes. There’s a whole bunch of reasons why, but for the most part it boils down to various forms of liberty - and tales of the mark of the beast.
… but it’s usually fine when your state or a corporation is doing it.
I'm not American, I know that, have heard various reasons for it, and i consider it to be profoundly stupid. It might have made sense in the 19th century, but today? Your federal government already knows all there is to know about you. They can tap in healthcare, financial, state government information, all that online stuff. The cat is out of the bag, why not use it for something practical? A common unified ID which gives access to e-government services is the norm across the EU, and it's pretty cool and practical.
That sounds profoundly stupid. You mean like the credit rating agencies that provide credit worthiness identity, but where you have no control of how and with whom they store your data? So how would that work, how would the companies providing that service make money and how could they prove your identity? They'd still need to use government info like birth certificate/drivers license to verify who you are, but they'd have a profit motive. Of course that sounds better than the government, which doesn't need to sell who you are, providing an identity service.
Why have a useless middleman trying to make money off you?
So fuck any homeless, people living alone, orphans, etc. ? Not to mention the utter unreliability of relying on random people (who claim they know you) for information like place/date of birth, etc.
At the same time they don’t have any issues with commercial companies knowing everything about them, without any kind of control or oversight.
https://constitution.congress.gov/constitution/amendment-10/....
[1]: https://www.gov.uk/service-toolkit
Or even running the IT for them. Lots of smaller municipalities and counties, even with all the guidance on the world, wouldn't have the resources to properly manage everything. Not to mention what a colossal waste it is, financially or practically, for every single administrative entity to reinvent the wheel for each of their IT needs.
Maybe they should learn from Zambia and restore their data from their backup, and tell the ransomware guys some nice things, just as the central bank of Zambia did.
Perhaps ransomware is evolving to fine just right price point niche?
https://statescoop.com/ransomware-map/
And that is 90% of the actual demand for bitcoin. The rest was speculation, and that is going away as prices fall. So imagine how bitcoin would crash if people stopped paying these ransoms.
Why don't businesses (or systems) seed their drives with files with known text / content and then use those files to reverse the method used to encrypt? It seems like having an adversary encrypt a set of known "canary" files should provide information to reverse the encryption?
Again, there may be a good reason (or many many good reasons) why this would not be a good solution, especially since I'd expect most OS installations have enough standard files to do this if it worked, but I am curious if someone knows.
Edit: From the helpful comments, this is a known class of attack on a cryptosystem called a plaintext attack. Using that information, I looked into how ransomware systems address this attack, and several, apparently, use per-file keys as, in part, a defense against this type of attack.
Seems to me that in modern times failing to back things up is incompetence and dereliction of duty. The auditor should be billed for at least part of the payment.