I started on a help desk, from my experience the management knew which ones were smart enough to do real damage and triggered enough to actually do it.
If I had any advice it would simply be to avoid letting a corporate environment allow you to forget you have real people as employees with feelings. "It's just business" doesn't translate to everyone.
A trigger is an event that precipitates other events, aka it sets things in motion. It could be paralysis, or it could be ejecting a bullet from a barrel or wiping a set of routers.
The commenter above you is trying to make you aware that "triggered" is an internet SJW dog-whistle to attract attention to an event. His statement is forcing you to acknowledge the definition of the word "triggered" in its most literal sense.
Can you explain this more? It doesn't make any sense to me.
Note, I am aware of the use of trigger as in "trigger warning" for potentially disturbing content. Even with that context I'm lost. Did you mean "literal" literally?
This is a total nightmare scenario for a lot of organizations, and is quiet doable for a lot of systems. Specifically on the router side (thanks Cisco) I haven't seen a lot of controls that would stop this. Network Engineers need full access to get things done in a timely fashion, and the limitations of control in TACACS. Recover from insider attacks of this nature are all based on how much to drill to recover from this style of attack.
So why is the router attack any worse than a "Delete exchange server + all backups" command, or the "use ansible to remote format entire server farm in 1 click" command?
You have to trust employees at some level. If people know that doing things like this will equal jailtime, I would assume that would stop most people.
Now think if this employee lived in Russia and did this remote. What would the recourse be?
At the scale of companies like this one it should be physically impossible to bring down a critical server and all its backups. That's what offsite backups, preferably on write only media are for.
In Unix, yes, but there have been systems without a single ultimately-privileged user.
One could imagine a system in which ultimate authority belongs to a 51% share of stockholders, whose keys delegate authority to the board of directors, who delegate authority to the CEO or CTO, who delegates authority on down the line. Each certifying party could revoke (or allow to expire) authority prior to firing a delegee.
> If people know that doing things like this will equal jailtime, I would assume that would stop most people.
What I want to know is how people grow to physical adulthood without realising that this is wrong. One simply doesn't destroy others' property.
For the same reason that people still do things knowing they result in jail time, people will still do things knowing they are wrong. There's probably a few root causes for actions like that (rage, hopelessness, greed, etc), with different levels of how much each one plays in the decision.
In the end, society only functions because the vast majority of people think they are better off following the norms set by that society. When they no longer think this because of emotional tunnel vision or a real lack of hope, whether it be real or imagined, or pervasive or lasting just long enough, their actions are no longer predictable as a rational member of society. Unfortunately, that means in some cases, it doesn't matter what the consequences are, there will still be the occasional incident.
> Okay so a deploy to prod should be able to require a stockholder voter?
Well, every deploy to production in any company is the result of a shareholder vote — it's just made obvious.
By using delegation and certificates, an employee who is delegated authority to deploy to production can do so without requiring a majority of shareholders to actually vote on that particular deploy.
That's pretty awesome.
> This sounds like a Utopian paradise not the realities of IT in 2016.
We've had the ability to do this for almost twenty years: it was made possible by RFCs 2692 & 2693 in 1999. The necessary processing could be performed in a split second.
The system able to do that will have to be designed by someone and maintained by someone, because it cannot just appear out of thin air, work perfectly and never have bugs. So, ultimately there will be someones with enough power to change the way every single thing in the system operates and therefore enough power to destroy the system.
"Someone has to have root" is not entirely true - you can easily have root accounts on key systems accessible only by requiring two separate authentication tokens, and have them be held by separate people. Sure, it's inconvenient, makes changes slower and requires more people and thus is more expensive, but that does prevent any single person from doing too much damage.
I would think the first and easiest thing to do to prevent situations like this is to make it so a single person cannot run certain commands. Sort of like the digital version of two keys on the opposite side of the room to launch a missile.
It's not the kind of problem, that can be solved with technology. There will always be people with access not loyal to the company and pissed at management.
Backups could be maintained on a regular schedule, including full router backups. Others in charge of this could be given create and append access to the backup systems, but not delete.
Once you delineate split rights and controls, then you can mitigate the severity of these attacks. Yes, douchebag-router admin can still zero out the routers. But you can be in business in less than a day with those configs logged.
Right - going down for a day can be damaging enough for some companies. Not allowing one person to click the "erase" button in the first place should be the focus. How hard could it be to implement a system where two or more people have to issue a command before it runs?
Not at all. If a human can do some damage, so can a script. Giving appropriate accounts permission so that no 1 account can destroy the whole system is just smart.
But thanks for assuming I'm some inhumane business exec, or a VC.
Honestly, it's not a bad idea even if the only reason is to just prevent accidental execution.
In the company I work at, we have something similar for database executions. In Dev you can run anything you want. However in prod, everyone only has read-only access. If you wish to execute something with higher authority, there is a web page we go to, where we can paste the code, a review board request, a ticket number (if available) a business authorizer, and a technical authorizor (other than yourself). It sounds like a lot, but it's usually pretty quick. Every command is then logged (which I believe is more for SOX reasons). It's annoying, but it's not that terrible of a system.
It's always the same problem: Who will deploy such a system?
At some point, you'll get somebody with full access to your system. You can reduce it into a very small window (at big cost), but you can't make it go away.
Have a few people at a single computer perform the following ritual:
1. deploy the system
2. create new admin account with a long random password and/or 2FA physical token
3. print the password out, seal it in an envelope and put it in a safe not accessible to any IT person; place the 2FA token in a separate safe controlled by different person than the first safe;
4. remove all other admin accounts - with the other present technicians checking that no other account will have access
5. if needed, with proper authorisation you can retrieve the password, and implement the required changes (again, with a second person controlling what exactly is being done).
If the systems require changes rarely, then this cost isn't that high.
That's what it's like from your side of the glass - on the other side there are still sysadmins who either execute those commands, or have access to the API keys privileged to run those commands.
As someone who works as an identity and access governance consultant and developer, I endorse this article.
I mean, there's very little you do about this particular scenario other than some small mitigations (at the end of the day you'll have to trust someone with something) but it's genuinely amazing how vulnerable the processes inside large, multi-million companies are to any internal attacker.
> I mean, there's very little you do about this particular scenario other than some small mitigations (at the end of the day you'll have to trust someone with something) but it's genuinely amazing how vulnerable the processes inside large, multi-million companies are to any internal attacker.
Maybe the scenario is too rare for companies to care, so when it happens they just accept the cost of once in a while instead of setting up additional security policies which will turn procedures cumbersome.
Maybe you're working at different firms than I am, but the typical reaction I see from management is that there is no problem that can't be solved with yet another layer of process.
It's not actually amazing if you think about any aspect of human life and consider how it all hinges on continued cooperation, good will, and concern for your own future.
What stops bad person in finance from making a huge wire transfer to a hidden account? What stops the CEO from plundering the company? What stops the average driver from taking out a bunch of pedestrians? What stops a doctor from purposefully poisoning one of her patients? What stops a parent from abusing their children?
Only good will and care for the future. And any system in place to stop that stuff from ever happening would cost more than the benefit.
What I find curious in this article is that they gloss over that this is also a people management problem. Sure you can put restrictive policies in place but apparently things went down at Citibank that caused so much ire for the person in question to take action like that.
I like to think that even though people see sysadmins are grumpy rage-machines in a basement most are super-decent people caring deeply for the systems under their control. Perhaps I've got a too rosy view of my fellow engineers.
I'd be very curious to hear what went on there on the people side of things. What led to this person feeling so angry that they would take an action like this. I'm also curious how it could even get so far. I have the benefit of being able to regularly talk to my manager so even if we did performance reviews in that way they wouldn't come as a surprise.
I disagree: If the employee is committed to his/her job and (ab)used by 'upper management', it is likely they reflect that on their job, up to the point of abuse.
A healthy work environment is as needed as compartmentalization and other schemes to manage abuse.
We do not know the extend and circumstances of the employee and his/her related story and the story of the company, so no judgement on that.
People have different reactions to bad news, especially life altering news like "you're about to be fired." These reactions stem from personality differences that form early in development and throughout life.
Someone could be a great team member, love their peers, and still react badly to the news they might be fired soon.
I don't think any amount of "people management" can fix someone who is already a bad egg, "configured" to react poorly to bad news. Some people will be out for revenge no matter what.
The more important thing to analyze here might be the technical measures in place to respond to configuration files being wiped out. Sure, an "insider" could maliciously wipe out the files... but couldn't a technical glitch just as easily do the same? The important lesson from this should be including adequate detection measures to immediately know when configuration files have been changed outside of the normal build process, and the ability to rollback when changes are detected.
One person should not be able to do any more damage than a technical glitch.
With organizations like banks and such, usually all your logins stop working just as security shows up and escorts you out... that's how you find out you've been fired.
If he left his review thinking "they're going to fire me" then there was a miscommunication somewhere along the line.
There's a reason employers don't say "We're firing you at the end of the week... so uh... play nice until then, k?"
> I'd be very curious to hear what went on there on the people side of things.
It's important to study that in these cases. I was convicted in a similar case and am actually a chapter in a CERT book, but they never reached out to me for my input, so they're completely oblivious to my motives and missing key facts about the case. I'd write more about it or speak at conferences but I've been able to bury that past and move on. Maybe some day.
I think he had a similar case to mine where he felt he was stuck in a situation with his manager(s) and felt like he had no recourse. HR should be involved in employee reviews and should provide a way for the employee to give feedback on his own review.
Another aspect of it is education about the law. I see case after case where the defendant had no idea he could face federal charges at all, much less one that can result in such stiff penalties. (The penalties he was facing were much, much worse if he had been tried and convicted rather than taking this plea deal.) That could easily be part of any degree program and/or employee orientation.
If a company is bringing in child labor in dangerous factories, and employing prostitutes to gratify the upper management, while you're being whipped daily, would that, maybe be an excuse?
I know it's a really out there example, but there are times where outright rebellion, even destructively, is morally acceptable. I don't know the specifics of this example, however.
That's a pretty big strawman. A better recourse would be to whistle blow in that scenario. Though clearly that carries a great deal of personal risk too.
Destroying a company damages all of that companies customers and employees as well. The difference between a surgical strike of a "bad guy" and carpet bombing a country.
Corrupt stuff in third world countries do not get entire chapters in CERT books - that's where things like these are expected to happen. Over and over again.
The NSA spying incident that Snowden revealed wasn't a third world country. Then again, he didn't take down the NSA's routers or spying aparatus on the way out either.
> HR should be involved in employee reviews and should provide a way for the employee to give feedback on his own review.
One issue I see here is that HR usually sides with management and tend to label you as a troublemaker whenever you disagree with something or someone. HR should always be neutral and they are usually far from it.
If you follow the link to the justice department website, you'll see that the guy saw himself as a martyr.
"They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team. Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
I did read that part, it's even quoted in the article posted.
However, you don't go from happy trotting elf in candy rainbow land to martyr by the flip of a switch (at least not that I know of). So regardless I'd still want to understand what contributed to creating an environment that caused someone to retaliate like this.
It's a little unsettling to think about how the biggest threats to an organization can be from within. The inside hacker doesn't even have to be disgruntled: they could just download the wrong link or put in a flash stick from the parking lot and BAM! More hacking in a minute than any Hollywood effort could get you.
The worst part about it is that to be completely on the ball with internal threats, you need to be constantly paranoid, assuming by default that any employee with enough access is a threat, or a threat waiting to happen. Such things do not add to team morale or cohesion, but in this case, a pound of prevention is still better than a metric ton of cure.
Regarding morale, there's always a "bad country" making headlines each week. A disgruntled employee and an employee coerced by Bad Country's operatives can do pretty much the same damage, so you might as well frame the risks and policies in terms of protecting employees from the latter.
I don't want to know about what he did unless there's some information on why he did it also. Was he pushed over the edge after years of mistreatment? Or was he mentally unstable and it was petty revenge?
Most of us have the keys to the kingdom in one way or another and while I (and others) could never do this it doesn't mean I don't have sympathy for someone who does - for some justifiable reason. I can't imagine any that are justifiable... but I can ask.
I found the other linked articles about, "The Malicious Insider", a bit fucked in the head. Unapproved hardware! Email misuse! Unapproved workaround! Unapproved software! Are you fucking kidding me? That's a pretty far cry from this guy.
From the SMS quoted in TFA, seems like he at least thought he and his colleagues were being continually mistreated by upper management:
“They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.”
“Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
This may sound petty, but it is sincere nonetheless; I'm curious how someone with his command of the English language rose to such a position of power to be able to cause this much damage.
I had an IT internship where my manager handed me a master key to the building and root logins for quite a few boxes. I also had the admin credentials to every user computer in the building.
Literally the "keys to the kingdom".
Sometimes the default level of trust is just really high...
It really depends on the company (size, number of staff) and the position itself. I've been in similar situations... I've also worked in smaller orgs that operated tighter operations than fortune 500 companies.
> This may sound petty, but it is sincere nonetheless; I'm curious how someone with his command of the English language rose to such a position of power to be able to cause this much damage.
Not having a great command of the English language doesn't seem to be a huge barrier to technical roles that are fairly insulated from the business side of things in enterprise environments, IMO.
What precautions can one take against their sysadmin? I can't think of many places that have stricter security policies than 3-letter agencies, still Snowden got away with a USB stick (man, can't wait to see the movie - I wonder if he really passed the USB stick out using the rubik's cube as in the movie trailer).
The thing is that that you can't do much against someone who has root access for a living.
There are other ways to make a point, especially when you're prepared to get fired. Damaging everyone else is selfish and stupid.
> The thing is that that you can't do much against someone who has root access for a living.
In banks, significant decisions (like personally authorising multimillion dollar transfer) require two signatures (of two senior officers), so they can't be abused by just one person going rogue.
It can be imagined that we apply the same approach for system administration - "pair-programming" work only, perhaps with proximity cards which detect when one of the admins is actually not at the desk.
That's very theoretical and not analogous because the difference is that sys admins have to do things quickly and under pressure, sometimes at ungodly hours.
In contrast the banker has a more relaxing experience putting his signature on a document. It's stressful enough running root commands sometimes, and I can't imagine how horrible it would be to require a second person to be on-call/pair/double-check/etc. I would have long ago quit the profession, which would be impossible to do anything in. Also systems would take longer to fix, so organizations would be displeased.
Vet your sys admins correctly before hiring, keep an eye on people's stress levels, which in 2016 America apparently doesn't matter to anyone anymore in IT, and don't piss off the sys admin.
I think the LISA paper does a good job describing an approach.
basically don't allow people to commit changes directly to the underlying system, require the changes from 2 administrators to be captured and then compare them for equivalence. (what equivalence means can be bit for bit equality at the most basic binary level, or perhaps some parsing that can verify equivalence at language level for configurations)
if the changes are equivalent commit them. otherwise point out the differences.
As the paper also discusses, can be used with a single administrator to make the administration auditable, simply capture all changes before they are committed.
Wasn't Snowden's trick to use an inkjet printer to make blank CDRs look like music CDs, bring to work and burn data to them, and then take them home again?
If that's true, it's pretty unbelievable security would allow someone to walk out of an NSA building with a CD, just because the label had an album cover on it.
Are you going to make people get naked and cavity search them? Physically smuggling data seems not that hard in this day and age. MicroSD is so dense, can fit one in my nose.
> All he needed, said multiple intelligence community sources, was a few thumb drives and the willingness to exploit a gaping hole in an antiquated security system to rummage at will through the NSA’s servers and take 20,000 documents without leaving a trace.
> Finally, Snowden’s physical location worked to his advantage. In a contractor’s office 5,000 miles and six time zones from headquarters, he was free from prying eyes. Much of his workday occurred after the masses at Ft. Meade had already gone home for dinner. Had he been in Maryland, someone who couldn’t audit his activities electronically still might have noticed his use of thumb drives.
I believe you're thinking of Chelsea Manning (Formerly Bradley Manning). If I remember correctly, she took a blank CDRW printed to look like a Lady Gaga album into the secure area to burn files. Even put on headphones to bop her head and mime listening to the music
I've heard, first hand that Citi is extremely toxic. People are mistreated, are abused (verbally) on a frequent basis. There's no accountability within the ranks and management doesn't care and encourages it
I almost accepted an offer to work in their NY offices. When I was being interviewed, the Manger of the group was divulging his dislike for the person I was going to report to.
It's a sick place & the IT guy in the article sounded like he was being abused. He shouldn't have done what he did, He should have documented the stuff he went through and maybe get a lawyer and sue for pain and suffering.
One might say that these things happening are the market telling Citi to stop being a dick. Unfortunately it seems like they're not really taking the hint.
I wonder if the fact that this was a bank had something to do with the short sentence. Back when I was a pr activist, it was known that the level of public anger - the public's trigger - on an issue would sway the decision of high-profile cases - that is, the public's perception would factor into the judge's decisionmaking. If crimes were committed inside certain "pathways", those crimes were easier to get away with under certain constraints. This man has sabotaged a bank using a computer and made pro-union statements. If he had made broadly anarchist statements I wonder how this sentence would have changed.
I argue that these dynamics are relevant from a criminal psychology and a government-hacking standpoint. Being mostly derived from firsthand resources, Toffler, Bernays, et. al, along with some authors whose mention is straight-up dangerous, I am constantly seeking additional resources.
Yes. The AT&T hacker 'weev' was sentenced to 41 months prior to the vacation of his conviction. As far as I know no systems were disrupted in his case. He expressed vaguely anti-authoritarian leanings without appealing to a stronger base.
This is arguably what keeps shows like Mr. Robot on the air btw. We will tolerate hackers down certain pathways and within certain constraints. Even if they're "insane". Leave the pathway and God help you.
Sometimes staying on the pathway is dangerous too, but my job is to motivate for great justice.
Edit: Don't kill or torture anyone. This should be in your base operating code anyway; if it isn't, enforce it down to pith level.
Federal sentencing guidelines are pretty rigid and most computer fraud and abuse cases get the lowest end of the range.
The EFF has a good breakdown of weev's sentence and the D&D-like charts and calculations used to reach it. He got hammered for "special skill", "sophisticated means" and "means of identification" bonuses.
Carmen Ortiz, the Federal Attorney General who continues to terrorize the Boston area, thought she was threatening him with 35 years in jail:
"If convicted on these charges," said Ortiz, "Swartz faces up to 35 years in prison, to be followed by three years of supervised release, restitution, forfeiture and a fine of up to $1 million."
Not totally different from, say, Edward Snowden. Down-trodden information technology workers becoming radicalized and committing acts of sabotage, like the factory floor workers also did a century ago.
A society where IT workers have great power over corporate fortunes but little political representation is going to be unstable...
I'd personally argue that there's quite a bit of a difference. Snowden, to my understanding, wasn't protesting a bad performance review and perceived neglect from upper management; he was protesting what he judged to be an illegal action on the part of his employer.
That being said - I do agree with your sentiment that the more dependent society becomes on technology, the greater the risk posed by those who possess knowledge of how to operate and maintain that technology. This doesn't just apply to human beings too - electronic actors (e.g., automated systems or even AI) may someday if not already hold just as much power.
The question in my mind then is how to "democratize" this power, so to speak.
This article is clickbait titled spam. There's a snippet of an interesting story there but not enough for a real story. The "analysis" at the end of the article is not really analysis at all.
There's interesting details behind this; were they really firing him? Did Citibank make the horrible error of telling an IT worker that they are fired but not immediately revoking their privileges? Could management have handled his performance issues better, or were there red flags that he should have been let go sooner?
All of that could be interesting but because this is just clickbait blog spam none of it is in the article.
You cannot control the human factor so easily but you can definitely keep you employees happy! I don't get why the article does not mention the quality of working environment. IT systems can be extremely stressful if there is not help from management, and most of us know what bad management means...
I agree. There's a lot of focus on punishment, consequences, and all the lockdowns for frustrating insider threats, but not so much on the flip side: reaching out to people, valuing them and their needs and opinions, working together, defusing problems so that there are fewer threats.
There are analogies to broader threats, say, to entire countries, and these days a lot of people are more interested in being antagonistic and hostile in dealing with the threats than working to understand them and their grievances. The two sides may never agree, but the effort makes all the difference.
It's a perfectly reasonable principle to write into law, a password or key or whatever doesn't give an employee the right to damage the systems of their employer.
This is reminiscent of the San Francisco network administrator who took it upon himself to "protect" the city from his supervisors and would only divulge his password to mayor Newsom.[1]
IIRC it was also precipitated by perf reviews or discipline.
Hope AWS has better systems in place to make something like this nearly impossible to carry out.
I worked for a large company that went through drastic downsizing at one point. Some ppl who were getting let go were "well positioned" to do some damage. The way the company handled this is there was no advance notice. Your manager and a couple of security dudes shows up at your desk with boxes and you're outta there in 5 min.
There were things like managers getting punched, ppl screaming etc but no malicious hacking that I'm aware of.
Somehow one dude (I'm assuming :)) managed to take a dump in the elevator on the way out :)
I consulted at a Fortune 100 investment bank in New York City 15 years ago. It's difficult to hide that layoffs are coming - managers are pulled into meetings to rank their employees, they tell their friend, word gets around.
Someone set up a digital timebomb on the systems. They were one of the people let go, and a few weeks later, things crashed all over the place.
One amusing bit - a few months later, another digital time bomb went off. He had hidden another one that they had not found, even knowing of the first one.
Despite laying off about 10% of the staff, management had a good idea of who it was, but they had been clever so it was difficult to get evidence it was them.
Sorry for the naive question, but in that kind of situation, how do you assure the continuity of the organization ? What if the people being let got were supporting customers, about to close a deal, and so on ? How do you make sure people are not leaving with a critical "know-how" ?
People in the middle of a deal or with crucial know-how will be called to serve out their notice period for handover - which if the HR is switched-on is 1-3 months. Usually some extra severance money will be offered as a sweetener compared to running away immediately.
However, potential to do damage usually supersedes potential to benefit the business in the notice period. A company with sane HR has preventative policies in place:
- Never have single-person key-man risk in the first place; always have a backup/deputy designated well in advance because anyone can be hit by a bus. Helps with covering holidays too.
- For highly sensitive jobs like sysops or people dealing with customer money, revoke access to systems first thing in the morning, then tell them, then immediately escort them out.
- Don't have shared or master passwords anywhere, but have user-based ACL instead so access can be revoked.
In theory the manager is supposed to determine who's needed for a skeleton crew. But there's managers that retain the people they thought were competent but because they're blinded by ass kissing they end up with less than ideal staff. At the end of the day it probably doesn't usually matter that much though, a drastic downsizing is usually a death knell anyway.
At financial institutions I've worked at, if you're getting fired you are taken to a room (managers office, conf room etc.) and immediately escorted out of the building after. No returning to your desk at all. Someone else will gather the belongings that you need and bring them like wallets/handbags. The rest of your stuff will be mailed/couriered to you later after someone has had the chance to go through them to make sure that there is no IP in them.
judging from the article, I was under the assumption that he had a poor review, not a firing. Possibly set to go on an improvement plan or some-such.
This leads to a messy set of compensations:
* When we fire people, they may take revenge, so we escort them out immediately
* ...but then people start assuming the worst when we tell them they are doing poorly, so we stop doing that and just fire them once it's bad enough (escorting them out immediately at that point)
* ...so then they start trying to guess when we are unhappy with them, and now we have employees that weren't at any risk of firing taking preemptive revenge...
It's a mess, and the less respectfully you treat people in the name of "policy", the more it can escalate things.
This happened when I was at Google, but it was an accident.
On someone's last day they ran rm -rf / to wipe their desktop machine. I can't remember whether they typed it into the wrong terminal window or had mounted an external filesystem somewhere deep under their home directory and forgot about it, it was something like that. They wound up wiping some unfathomably large amount of data (I think most of a data center) before noticing what they'd done.
There were backups, but it was still a huge hassle.
I managed to do that one time but on a much smaller scale (home server). Forgot about my network mounts, rm -Rf, noticed that it was taking longer than expected to delete just a few dirs... "oh crap." :) Luckily I run zfs with zfs-auto-snap on my home server so everything was back the way it was after a quick 'zfs restore'.
I once left a "production" connection string in my local config while fixing a bug in my last few days on a job, this got checked into source control (yeah, configuration in source control is bad)... a couple weeks after I left, someone had used that connection string to connect to the database server, and truncated a table.... needless to say, there were some things that changed after that.
Sounds like a management issue more than security. However, post-Snowden, I did some exploration of a strategy for countering sysadmin risk in organizations whose IT activities were pretty regular.
> “They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.”
> “Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
Ah yes, the old double 'they are harming me by holding me accountable' and 'I am helping them by holding them accountable' defense.
What I'd like to know is how mindsets like this are formed in life, and how to prevent them. One's employer doesn't owe one a job; one owes one's employer fair labour in return for one's wages. The world does not consist of shadowy forces plotting against one. Don't minimise one's own agency: be an active force for good, not a passive subject of whatever happens.
Labor laws disagree to a certain degree -- at least in that you can't fire someone for any reason you choose (see the EEOC poster in your breakroom/mailroom/kitchen for examples). Also employment contracts can stipulate various things that prevent a company from firing you, particularly in union jobs.
I've found going by the motto "don't be a dick" tends to be a decent deterrent against doing something stupid like maliciously wiping company data and getting sued and/or going to jail for it. If it's your employer doing it to you, there is plenty enough opportunity in IT to find work at a place that doesn't have a hostile work environment.
One of my clients, a well known fixture in American finance, has been hacked by employees or contractors several times over the past 10 years. These cases have been public and supposedly cost millions (it's so hard to know how accurate the assessed damages in these cases are).
Because they now fear employees and contractors more than external threats it takes 30 tickets to different groups to set up a server. You can't chown a file in a directory you own without a ticket. You can use one of two old text editors. No new IDE's or modern text editors. You can't upgrade language runtimes without corp. approval. It's nuts. Ironically it leads to more attack vectors and it's ground their software dev to a halt.
> has been hacked by employees or contractors several times over the past 10 years
It's simply an overreaction to risk. Just like insisting the company doesn't treat its employees like human beings is also an overreaction (and gross oversimplification of what is actually a complicated topic).
Considering we're talking about Citi, who does have a track record of treating their IT staff poorly, you can't call my statement an overreaction or a gross oversimplification.
I am seeing much the same attitude out of FOSS these days. Taking away options in the name of "user confusion" and "security". In effect they do not trust the end user to not shoot themselves in the foot. aka, paternalism on a grand scale.
189 comments
[ 757 ms ] story [ 2592 ms ] threadIf I had any advice it would simply be to avoid letting a corporate environment allow you to forget you have real people as employees with feelings. "It's just business" doesn't translate to everyone.
What does this mean? Do you mean "provoked"?
Note, I am aware of the use of trigger as in "trigger warning" for potentially disturbing content. Even with that context I'm lost. Did you mean "literal" literally?
So why is the router attack any worse than a "Delete exchange server + all backups" command, or the "use ansible to remote format entire server farm in 1 click" command?
You have to trust employees at some level. If people know that doing things like this will equal jailtime, I would assume that would stop most people.
Now think if this employee lived in Russia and did this remote. What would the recourse be?
In Unix, yes, but there have been systems without a single ultimately-privileged user.
One could imagine a system in which ultimate authority belongs to a 51% share of stockholders, whose keys delegate authority to the board of directors, who delegate authority to the CEO or CTO, who delegates authority on down the line. Each certifying party could revoke (or allow to expire) authority prior to firing a delegee.
> If people know that doing things like this will equal jailtime, I would assume that would stop most people.
What I want to know is how people grow to physical adulthood without realising that this is wrong. One simply doesn't destroy others' property.
In the end, society only functions because the vast majority of people think they are better off following the norms set by that society. When they no longer think this because of emotional tunnel vision or a real lack of hope, whether it be real or imagined, or pervasive or lasting just long enough, their actions are no longer predictable as a rational member of society. Unfortunately, that means in some cases, it doesn't matter what the consequences are, there will still be the occasional incident.
Well, every deploy to production in any company is the result of a shareholder vote — it's just made obvious.
By using delegation and certificates, an employee who is delegated authority to deploy to production can do so without requiring a majority of shareholders to actually vote on that particular deploy.
That's pretty awesome.
> This sounds like a Utopian paradise not the realities of IT in 2016.
We've had the ability to do this for almost twenty years: it was made possible by RFCs 2692 & 2693 in 1999. The necessary processing could be performed in a split second.
Which is fine, but seems like a roundabout way to do it.
Backups could be maintained on a regular schedule, including full router backups. Others in charge of this could be given create and append access to the backup systems, but not delete.
Once you delineate split rights and controls, then you can mitigate the severity of these attacks. Yes, douchebag-router admin can still zero out the routers. But you can be in business in less than a day with those configs logged.
Point in case, as parent commentator said, it's not the kind of problem that you solve with technology.
But thanks for assuming I'm some inhumane business exec, or a VC.
In the company I work at, we have something similar for database executions. In Dev you can run anything you want. However in prod, everyone only has read-only access. If you wish to execute something with higher authority, there is a web page we go to, where we can paste the code, a review board request, a ticket number (if available) a business authorizer, and a technical authorizor (other than yourself). It sounds like a lot, but it's usually pretty quick. Every command is then logged (which I believe is more for SOX reasons). It's annoying, but it's not that terrible of a system.
Of course a sane company wouldn't have a bus factor of one for such a process, but not all companies are sane.
At some point, you'll get somebody with full access to your system. You can reduce it into a very small window (at big cost), but you can't make it go away.
1. deploy the system
2. create new admin account with a long random password and/or 2FA physical token
3. print the password out, seal it in an envelope and put it in a safe not accessible to any IT person; place the 2FA token in a separate safe controlled by different person than the first safe;
4. remove all other admin accounts - with the other present technicians checking that no other account will have access
5. if needed, with proper authorisation you can retrieve the password, and implement the required changes (again, with a second person controlling what exactly is being done).
If the systems require changes rarely, then this cost isn't that high.
I assume one could refine the process by binding sets of commands or capabilites to specific cards.
Would also be rather handy in preventing remote access attacks.
I mean, there's very little you do about this particular scenario other than some small mitigations (at the end of the day you'll have to trust someone with something) but it's genuinely amazing how vulnerable the processes inside large, multi-million companies are to any internal attacker.
Maybe the scenario is too rare for companies to care, so when it happens they just accept the cost of once in a while instead of setting up additional security policies which will turn procedures cumbersome.
What stops bad person in finance from making a huge wire transfer to a hidden account? What stops the CEO from plundering the company? What stops the average driver from taking out a bunch of pedestrians? What stops a doctor from purposefully poisoning one of her patients? What stops a parent from abusing their children?
Only good will and care for the future. And any system in place to stop that stuff from ever happening would cost more than the benefit.
I like to think that even though people see sysadmins are grumpy rage-machines in a basement most are super-decent people caring deeply for the systems under their control. Perhaps I've got a too rosy view of my fellow engineers.
I'd be very curious to hear what went on there on the people side of things. What led to this person feeling so angry that they would take an action like this. I'm also curious how it could even get so far. I have the benefit of being able to regularly talk to my manager so even if we did performance reviews in that way they wouldn't come as a surprise.
I'd be interested to hear it too, but in the grand scheme of things, it doesn't matter one iota.
A healthy work environment is as needed as compartmentalization and other schemes to manage abuse.
We do not know the extend and circumstances of the employee and his/her related story and the story of the company, so no judgement on that.
Someone could be a great team member, love their peers, and still react badly to the news they might be fired soon.
I don't think any amount of "people management" can fix someone who is already a bad egg, "configured" to react poorly to bad news. Some people will be out for revenge no matter what.
The more important thing to analyze here might be the technical measures in place to respond to configuration files being wiped out. Sure, an "insider" could maliciously wipe out the files... but couldn't a technical glitch just as easily do the same? The important lesson from this should be including adequate detection measures to immediately know when configuration files have been changed outside of the normal build process, and the ability to rollback when changes are detected.
One person should not be able to do any more damage than a technical glitch.
If he left his review thinking "they're going to fire me" then there was a miscommunication somewhere along the line.
There's a reason employers don't say "We're firing you at the end of the week... so uh... play nice until then, k?"
It's important to study that in these cases. I was convicted in a similar case and am actually a chapter in a CERT book, but they never reached out to me for my input, so they're completely oblivious to my motives and missing key facts about the case. I'd write more about it or speak at conferences but I've been able to bury that past and move on. Maybe some day.
I think he had a similar case to mine where he felt he was stuck in a situation with his manager(s) and felt like he had no recourse. HR should be involved in employee reviews and should provide a way for the employee to give feedback on his own review.
Another aspect of it is education about the law. I see case after case where the defendant had no idea he could face federal charges at all, much less one that can result in such stiff penalties. (The penalties he was facing were much, much worse if he had been tried and convicted rather than taking this plea deal.) That could easily be part of any degree program and/or employee orientation.
http://www.infoworld.com/article/2653004/misadventures/why-s...
Still no excuse to take down 90% of a company's connectivity.
I know it's a really out there example, but there are times where outright rebellion, even destructively, is morally acceptable. I don't know the specifics of this example, however.
Destroying a company damages all of that companies customers and employees as well. The difference between a surgical strike of a "bad guy" and carpet bombing a country.
Also, blowing the whistle can mean many things... in this case, it was a temporary disruption, not much different than an organized strike would be.
Yeah, isn't that what I said?
One issue I see here is that HR usually sides with management and tend to label you as a troublemaker whenever you disagree with something or someone. HR should always be neutral and they are usually far from it.
"They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team. Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
However, you don't go from happy trotting elf in candy rainbow land to martyr by the flip of a switch (at least not that I know of). So regardless I'd still want to understand what contributed to creating an environment that caused someone to retaliate like this.
The worst part about it is that to be completely on the ball with internal threats, you need to be constantly paranoid, assuming by default that any employee with enough access is a threat, or a threat waiting to happen. Such things do not add to team morale or cohesion, but in this case, a pound of prevention is still better than a metric ton of cure.
Most of us have the keys to the kingdom in one way or another and while I (and others) could never do this it doesn't mean I don't have sympathy for someone who does - for some justifiable reason. I can't imagine any that are justifiable... but I can ask.
I found the other linked articles about, "The Malicious Insider", a bit fucked in the head. Unapproved hardware! Email misuse! Unapproved workaround! Unapproved software! Are you fucking kidding me? That's a pretty far cry from this guy.
“They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.”
“Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
Literally the "keys to the kingdom".
Sometimes the default level of trust is just really high...
Not having a great command of the English language doesn't seem to be a huge barrier to technical roles that are fairly insulated from the business side of things in enterprise environments, IMO.
The thing is that that you can't do much against someone who has root access for a living.
There are other ways to make a point, especially when you're prepared to get fired. Damaging everyone else is selfish and stupid.
He did, however, hold a Rubix cube as a way for Greenwald and Poitras to identify him in an early meeting. http://www.cnet.com/news/edward-snowden-and-the-rubiks-cube-...!
In banks, significant decisions (like personally authorising multimillion dollar transfer) require two signatures (of two senior officers), so they can't be abused by just one person going rogue.
It can be imagined that we apply the same approach for system administration - "pair-programming" work only, perhaps with proximity cards which detect when one of the admins is actually not at the desk.
In contrast the banker has a more relaxing experience putting his signature on a document. It's stressful enough running root commands sometimes, and I can't imagine how horrible it would be to require a second person to be on-call/pair/double-check/etc. I would have long ago quit the profession, which would be impossible to do anything in. Also systems would take longer to fix, so organizations would be displeased.
Vet your sys admins correctly before hiring, keep an eye on people's stress levels, which in 2016 America apparently doesn't matter to anyone anymore in IT, and don't piss off the sys admin.
"Two-Person Control Administration: Preventing Administation Faults through Duplication" by Shaya Potter, Steve Bellovin, and Jason Nieh
https://www.usenix.org/conference/lisa-09/two-person-control...
The video isn't there for some reason, but the paper and audio of the presentation are.
basically don't allow people to commit changes directly to the underlying system, require the changes from 2 administrators to be captured and then compare them for equivalence. (what equivalence means can be bit for bit equality at the most basic binary level, or perhaps some parsing that can verify equivalence at language level for configurations)
if the changes are equivalent commit them. otherwise point out the differences.
As the paper also discusses, can be used with a single administrator to make the administration auditable, simply capture all changes before they are committed.
Inquiring minds want to know.
> Finally, Snowden’s physical location worked to his advantage. In a contractor’s office 5,000 miles and six time zones from headquarters, he was free from prying eyes. Much of his workday occurred after the masses at Ft. Meade had already gone home for dinner. Had he been in Maryland, someone who couldn’t audit his activities electronically still might have noticed his use of thumb drives.
http://www.nbcnews.com/news/other/how-snowden-did-it-f8C1100...
I almost accepted an offer to work in their NY offices. When I was being interviewed, the Manger of the group was divulging his dislike for the person I was going to report to.
It's a sick place & the IT guy in the article sounded like he was being abused. He shouldn't have done what he did, He should have documented the stuff he went through and maybe get a lawyer and sue for pain and suffering.
Take note - just in case, don't wear a 'hoody' as people may think you are a 'hacker'.
I argue that these dynamics are relevant from a criminal psychology and a government-hacking standpoint. Being mostly derived from firsthand resources, Toffler, Bernays, et. al, along with some authors whose mention is straight-up dangerous, I am constantly seeking additional resources.
Uh. Hack the planet.
Edit: or don't. U du u.
This is arguably what keeps shows like Mr. Robot on the air btw. We will tolerate hackers down certain pathways and within certain constraints. Even if they're "insane". Leave the pathway and God help you.
Sometimes staying on the pathway is dangerous too, but my job is to motivate for great justice.
Edit: Don't kill or torture anyone. This should be in your base operating code anyway; if it isn't, enforce it down to pith level.
The EFF has a good breakdown of weev's sentence and the D&D-like charts and calculations used to reach it. He got hammered for "special skill", "sophisticated means" and "means of identification" bonuses.
https://www.eff.org/deeplinks/2013/03/41-months-weev-underst...
https://en.wikipedia.org/wiki/United_States_v._Swartz
I wonder where they started the negotiation before settling on a 21 month plea deal
"facing 35 years in jail" is inaccurate in the sense that it would've been impossible for him to receive such a sentence.
"If convicted on these charges," said Ortiz, "Swartz faces up to 35 years in prison, to be followed by three years of supervised release, restitution, forfeiture and a fine of up to $1 million."
A society where IT workers have great power over corporate fortunes but little political representation is going to be unstable...
That being said - I do agree with your sentiment that the more dependent society becomes on technology, the greater the risk posed by those who possess knowledge of how to operate and maintain that technology. This doesn't just apply to human beings too - electronic actors (e.g., automated systems or even AI) may someday if not already hold just as much power.
The question in my mind then is how to "democratize" this power, so to speak.
There's interesting details behind this; were they really firing him? Did Citibank make the horrible error of telling an IT worker that they are fired but not immediately revoking their privileges? Could management have handled his performance issues better, or were there red flags that he should have been let go sooner?
All of that could be interesting but because this is just clickbait blog spam none of it is in the article.
There are analogies to broader threats, say, to entire countries, and these days a lot of people are more interested in being antagonistic and hostile in dealing with the threats than working to understand them and their grievances. The two sides may never agree, but the effort makes all the difference.
I'm genuinely curious, is this the actual charge? What specific law did he break here?
Edit: He plead to I think a single count of the following 18:1030(a)(5)(A) and 18:1030(c)(4)(A)(i)(I) and (B)
http://www.leagle.com/decision/In%20FDCO%2020160411834/U.S.%...
https://www.law.cornell.edu/uscode/text/18/1030
It's a perfectly reasonable principle to write into law, a password or key or whatever doesn't give an employee the right to damage the systems of their employer.
IIRC it was also precipitated by perf reviews or discipline.
Hope AWS has better systems in place to make something like this nearly impossible to carry out.
[1]http://m.sfgate.com/bayarea/article/S-F-officials-locked-out...
There were things like managers getting punched, ppl screaming etc but no malicious hacking that I'm aware of. Somehow one dude (I'm assuming :)) managed to take a dump in the elevator on the way out :)
They had EMS parked outside too. Wasn't pretty.
Someone set up a digital timebomb on the systems. They were one of the people let go, and a few weeks later, things crashed all over the place.
One amusing bit - a few months later, another digital time bomb went off. He had hidden another one that they had not found, even knowing of the first one.
Despite laying off about 10% of the staff, management had a good idea of who it was, but they had been clever so it was difficult to get evidence it was them.
However, potential to do damage usually supersedes potential to benefit the business in the notice period. A company with sane HR has preventative policies in place:
- Never have single-person key-man risk in the first place; always have a backup/deputy designated well in advance because anyone can be hit by a bus. Helps with covering holidays too.
- For highly sensitive jobs like sysops or people dealing with customer money, revoke access to systems first thing in the morning, then tell them, then immediately escort them out.
- Don't have shared or master passwords anywhere, but have user-based ACL instead so access can be revoked.
This leads to a messy set of compensations:
* When we fire people, they may take revenge, so we escort them out immediately
* ...but then people start assuming the worst when we tell them they are doing poorly, so we stop doing that and just fire them once it's bad enough (escorting them out immediately at that point)
* ...so then they start trying to guess when we are unhappy with them, and now we have employees that weren't at any risk of firing taking preemptive revenge...
It's a mess, and the less respectfully you treat people in the name of "policy", the more it can escalate things.
On someone's last day they ran rm -rf / to wipe their desktop machine. I can't remember whether they typed it into the wrong terminal window or had mounted an external filesystem somewhere deep under their home directory and forgot about it, it was something like that. They wound up wiping some unfathomably large amount of data (I think most of a data center) before noticing what they'd done.
There were backups, but it was still a huge hassle.
Always a fear of mine. I always try to pause for a second before I run a command like that.
I used to think all the banks would do that. Like UPS it is a very important insurance.
Also the man who shuts down the bank will face serious consequences.
https://www.schneier.com/blog/archives/2013/08/nsa_increasin...
> “Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
Ah yes, the old double 'they are harming me by holding me accountable' and 'I am helping them by holding them accountable' defense.
What I'd like to know is how mindsets like this are formed in life, and how to prevent them. One's employer doesn't owe one a job; one owes one's employer fair labour in return for one's wages. The world does not consist of shadowy forces plotting against one. Don't minimise one's own agency: be an active force for good, not a passive subject of whatever happens.
That kind of thinking is exactly what breeds this kind of stuff, though.
Labor laws disagree to a certain degree -- at least in that you can't fire someone for any reason you choose (see the EEOC poster in your breakroom/mailroom/kitchen for examples). Also employment contracts can stipulate various things that prevent a company from firing you, particularly in union jobs.
I've found going by the motto "don't be a dick" tends to be a decent deterrent against doing something stupid like maliciously wiping company data and getting sued and/or going to jail for it. If it's your employer doing it to you, there is plenty enough opportunity in IT to find work at a place that doesn't have a hostile work environment.
Because they now fear employees and contractors more than external threats it takes 30 tickets to different groups to set up a server. You can't chown a file in a directory you own without a ticket. You can use one of two old text editors. No new IDE's or modern text editors. You can't upgrade language runtimes without corp. approval. It's nuts. Ironically it leads to more attack vectors and it's ground their software dev to a halt.
It's simply an overreaction to risk. Just like insisting the company doesn't treat its employees like human beings is also an overreaction (and gross oversimplification of what is actually a complicated topic).