This would never have happened in the first place had they used an encrypted complex password and a simple password manager.
The whole company takes the hit with blunders like this. It's everyone's fault responsible for the infrastructure allowing this to happen in the first place. Those who pass the blame on others very quickly are equally to blame which means the CEO is just as to blame as the 'intern'.
Clearly the whole company doesn't train their interns.
The password was coded into a file that was checked in to git.
The git repository just happened to be public.
It's entirely reasonable to think that the person in question possibly didn't even stop to think that Solarwinds123 was an actual secret that needed to be kept, as it is the equivalent of common passwords that are published publicly in manufacturer documentation.
I think the point is at no point is it acceptable to be in a position to be able to do something deeply damaging to a company with something as simple as a intern leaking a password. The intern should never been put in a position where this was even possible.
I’d say in all the ways that matters this was basically everybody BUT the interns fault.
Yes, an individual was responsible: the CEO. Ultimately that job comes with the responsibility for everything happening inside the company and as a CEO you ensure that you have the proper culture and control mechanisms in place that that sort of thing does not happen, and that if it should happen that it gets detected.
By your standards, zero companies have a proper “culture” then. That’s just not how things work anywhere. You might think or even assume they should, but they don’t.
The story line may be being twisted by CNN but the idea of a CEO blaming his staff is fundamentally wrong. The message should be “I created a culture which meant an intern could put the whole company at risk. This was my mistake”
Ted Cruz claimed it was his children's fault for pressuring him into their trip, e.g. there is no lacking in incompetence and integrity in many systems, organizations, companies, communities, families. It starts with the individual which requires someone caring about how they look, about accountability, chain of command/responsibility.
How do we solve this though? Who is the CEO responsible to - who do they care, if anyone, about how they appear to? How do we shift this behaviour for the examples of CEOs and politicians who maybe haven't been outed yet by such a security breach but are vulnerable to such lacking of a rigid structure of command/responsibility?
Edit to add: I'm beginning to think HN is full of really lazy people.
> Because of downvotes? That's probably just because you brought partisan politics into the discussion.
Wut? Mentioning a politician’s name isn’t partisan politics. He said nothing about the politics of Cruz, he compared the Senator shifting blame onto his kids with this CEO blaming an intern. Don’t be like that.
Thank you for pointing this out - it's unfortunate that people can be immediately triggered to react after simply seeing a name that prevents/blocks them from continuing on to understand or critically think to understand what's actually said; it's definitely a sign of the times and state of our health and our thinking ability as a society.
This is uncharitable. First, you assume people were offended (and you used the loaded term 'triggered' to describe it). Then you go on to assume this means that society must be degrading, because so many people are losing their ability to think (the implication being that you are not part of that group).
A much simpler explanation is that people who read and comment on HN do not want to see every conversation degenerate into yet another political fistfight, since it seems to have infected every other part of our lives. One of the core values of this community is that we mostly manage to avoid that.
My perspective is valid, you've done nothing to prove it's not - other than putting your own "much simpler explanation;" where you're also making assumptions. Hope you enjoyed the thought exercise writing out your reply.
It's interesting how I can reference, with no personal opinion attached to it, a factual, recent situation that's comparable to the thread - about a well-known person who happens to be a politician - happens to be a politician, and as you said, because you're exhausted of hearing about them, you'd prefer they be censored/avoided in conversation. It's interesting to say the least.
P.S. Triggered is a valid word, I used it in the correct context - and you were in fact triggered by using it and are trying to gate keep its use.
What you are doing here is a pretty good real world demonstration of Russell conjugations. Also very non-productive, so I don't think there is any reason to try continuing the discussion. Have a great day!
Re: Russel conjugations - there was no "spin" to anything I said, I stated facts that were neutral - yes, that neutrality which is certainly a better light than you're appearing to believe Ted Cruz deserves (which has nothing to do with my own perception of him - it's your own belief or reaction you're applying to the facts). There is a difference - and if you read the examples at Google for searching "what is russell conjugation" - https://tomdehnel.com/what-is-russell-conjugation/ - then none of them fit what I said because I was just listing fact, so my points still stand and I wasn't using any fallacies or bad faith arguments. I won't analysis the conversation further because it'll become inflammatory. I'd love for you to show exactly where I am applying bias to the facts I listed.
> Mentioning a politician’s name isn’t partisan politics
Of course it is, why would you think otherwise? By calling out a particular politician by name, especially a controversial one, OP introduced a political slant to the discussion. Could just as easily have said "this is a problem we commonly see with politicians, or other people in leadership positions" without calling a particular one out.
> Don’t be like that.
Like what, exactly? OP made a comment, and when it started to go grey he maligned the HN commentariat as a whole as being full of lazy people. All I did was help him understand what aspect of his comment was the likely trigger for the downvotes. HN by and large does not like political discussions, it's even right there in the moderation rules.
> I'm beginning to think HN is full of really lazy people.
> Ted Cruz claimed it was his children's fault for pressuring him into their trip...
I'll bite. (Stick with me; I'll get to the point after some analysis that may be objectionable or seem unrelated.)
He's got responsibility to two parties: the citizens of his state and his children. Both are important in different ways, and the needs of each conflict in certain circumstances.
Did he want to get somewhere warm? Probably.
Did his children want their father with them? Probably.
Was he unable to advise while in Cancun? Doubtful.
Was he unable to legislate while in Cancun. He was not.
So, despite the inappropriate optics of his trip (and reflection of poor character for a leader of the state), it appears as though he made a judgement that favored his family and his self, presumably because he judged no apparent harm to the state by his actions.
I bring all this up because it's relevant to the parallel you bring: he blames his children for the choice. The problem is, neither you nor anyone else I'm aware of have demonstrated harm by his choice. So his "blaming" his children appears to reflect his choice between two conflicting priorities.
The intern, on the other hand, caused demonstrable harm and the CEO blames his own decisions on said intern.
Cruz did not blame lack of preparedness on his children. He blamed bad optics on his choice to acquiesce to his children. Far as I can tell, you're bringing a straw man to a knife fight.
I don't think your assertion is incorrect, that incompetence and lack of integrity are responsible for the CEO's choice, but I don't think you demonstrated that properly with your parallel. I find Ted Cruz to be pretty unpalatable, but I believe the hype around his trip is exaggerated, and your use of it to affirm your assertion is misplaced.
However, in an effort to make this more interesting, let's say the parallel is appropriate. I would, for the sake of argument, challenge that the fault, instead, lies in the hands of the people of Texas. If "the buck stops here", then let us remember that there is an entity above that of Ted Cruz, and that is the people of Texas.
If we truly believe that mistakes are not the fault of the individual who made the mistake, but rather the person at the top of the chain, then it stands to reason that politicians are not at fault, but rather the citizens that voted that politician in.
Since voters have ultimate responsibility to decide the way in which their community is governed—just as a CEO makes choices that bubble down—then it seems to me the people are ultimately responsible.
And in this context, your parallel seems even more misplaced, because the people of Texas (and the rest of the country) are blaming the "little guy" when those in charge of actually deciding who's hired and who's fired are the responsible parties.
Personally, I find the ultimate lack of integrity in "the people", because everyone blames "government" when "government" goes wrong, but fail to acknowledge that they are, in fact, said government.
I charge you to challenge my assertion and justify your parallel.
I see this, though I've exhausted my mental focus for today - I have severe chronic pain to contend with every day. I promise you I will respond likely by tomorrow, perhaps only will get a draft done and then Monday will reply. First though, thank you for taking a bite and spending energy engaging - I appreciate that alone.
Thank you, and I hope you feel at least a little better tomorrow.
I'd like to clarify one thing that I think was nebulous in my post: while I think "the people" are the responsible party, I meant to suggest that choosing who is the responsible party at all is close to arbitrary.
That is to say, arguments over who's held responsible more reflects who we want to be responsible than who may actually be responsible, which is in fact shared among several people. (The CEO, the intern, the person who gave the intern the password, the person who configured the system to require a password that could be leaked, etc.)
We choose the CEO (or Ted Cruz, or whomever) because of our own biases, in spite of the fact that responsibility is shared, and in spite of the fact that who's "in charge" is nebulous at best.
We do the same when blaming the 12 US billionaires (or however many there are) for income inequality in the country, or blame all white people for racial injustice, and so on.
That doesn't mean there aren't responsible parties, or that people and groups should be free from criticism. The truth of the matter is that—no matter how justified it is to assign blame—it's exceedingly rare for anyone to take responsibility for their own failings.
This shouldn't necessarily contort to whom and how we assign blame. But, rather, it should inform us of the breadth of the problem, our own contributions to it, and help us become better citizens, instead of shift the blame as the CEO or Cruz did.
If we find that unpalatable, but don't acknowledge our own complicity, then all we've achieved is a pitchfork-laden witch hunt, and not actual direction towards resolution.
Cruz using his children as an excuse shows either of these:
1) He's a pushover and his children, that we can assume he loves, he doesn't have control or authority over them for whatever reason; and in fact they'd have control over him.
2) He's using his children as an excuse to avoid taking accountability, and the problem down the whole line is a lack of accountability.
I agree the necessity of being in-person is debatable, however regardless, the lack of common sense to understand how it would be easily perceived is quite astonishing.
You're blaming the intern here - in fact it was the policies and protocol that allowed that whole shenanigans to unfold how it did: it shouldn't have been the interns responsibility to begin with - protocol should have prevented it, protocol and process that in a chain of command/accountability should reach the CEO if the protocol is adequate.
Of course the hype about his trip is exaggerated, that's to be expected - perhaps we've all been attempted to be gaslighted by Trump for so long that this doesn't seem like a big deal, voted in representatives not being actively engaged in military fashion to protect and help "his" people. We're so used to unqualified, inadequate, incompetent politicians being voted into power - that this seems okay - how many people died so far, how much damage was caused leading to economic damage and reducing productivity of those harmed - especially if they have to fight insurance companies for damage; it speaks to Eric Weinstein's commentary that it's only during wartime that the true leaders appear out of necessity.
Indeed, the people of Texas have been manipulated, arguably controlled subtly for many decades, directed by the duopoly - crafted heavily by industrial complexes - to put relatively bad people into the spotlight, for the mainstream media complex, controlled by a handful of conglomerates, to then amplify the two main narratives of either side of the duopoly. So is it really the people's fault? Yes and no, of course with the right policy proposals all of this can be countered strongly, and the change will cascade quickly, exponentially - policy proposals such as Andrew Yang's core policies; it's all the yin-yang dance of the universe, people need to suffer to learn - to pay attention, and hopefully we're at a part of a cycle where healing and creation is rapidly approaching, hopefully this isn't just a foreshock - where Trump's wave is still building to a tsunami - and we're yet awaiting the major earthquake before everything destabilizes and perhaps we're going to lead into a civil war - and perhaps even global war will start in parallel, if democracies can't be stabilized enough to create multi-lateral trade agreements to economically funnel resources away from the known bad actors who aren't behaving or falling in line to peace and freedom.
It's complex, there are systems that most don't know or understand or consider - and these systems self-perpetuate like a virus to keep counter attacks at bay; the autonomous immune system of America and the world is failing, inadequate. Luckily the purpose and value of distributing power, control - separating Federal government from the States - and further diversifying risk via having many States with their own varying rules - allows neighbouring States to shine light/the truth onto the States that have been darkened, influenced, lead too far astray by bad actors (domestic and foreign).
It's a lack of integrity in the systems - these flows that lead to multi-generational trauma, karma - action, consequences - passed down from one generation to the next. It's why my focus for all my projects is health: there are still ripples passed through stress-trauma of parents of war onto their children, and onto their children, and so on. We are generating new active trauma while not doing enough to support new people born or migrating to our more/rel...
Your comment was down voted because it mentioned a political figure. In today's hyper-polarized society, any mention of politics in a discussion about something apolitical preemptively draws silence or disregard. No one wants to be dragged into an adjacent unproductive argument.
Looking at your other posts, it also seems that you easily show a pessimistic attitude, similar to how old people will complain about younger generations. No one responds positively to such generalizations.
What you seem to be claiming as generalizations I can only assume you speak to the holistic view that I often speak from.
Mentioning a political figure doesn't necessitate baiting one into a conversation - however yes, it's clear that people are tired, exhausted - and no longer actually read to comprehend, stunting critical thinking throughout what they're reading - and then making an assumption as to what they're reading, and then doing a lazy downvote for a dopamine hit to feel fulfilled - as if they were diligent, reaffirming their assumptions.
And I disagree that "no one responds positively to such generalizations" - it doesn't make them untrue, and usually I also reference the solution, policy proposals, to solve the problems.
This is a silly standard that nobody is held to in the real world. Nearly every company can have a breach due to an intern or another employee leaking credentials. There’s nothing the CEO can do about that.
It is often the case after these breaches that HNers start assigning blame in ways that indicate their lack of practical experience in the security world. After 20 years working in security, from startups to fortune 500s, including the most well funded security teams in the world, EVERYONE has stupid password problems in some corners of their company. The password leak was absolutely not the CEOs mistake.
With all due respect to your experience in the industry, if knowing that an intern’s password is solarwinds123 is enough for an attacker to sign and release an unauthorized binary and publish it to hundreds of customers, then that’s an operational problem that the CEO is ultimately responsible for.
Well, nothing like that even happened, so why are we talking about it? solarwinds123 was a password for a public ftp server. It did not let anyone sign anything and it didn’t push updates out. It isn’t even related to the solarwinds breach.
That’s the reason they’re having the hearing, the point is that the CEO is bringing this up and blaming an intern for the massive breach when it’s pretty clearly either a red herring or a complete lapse in security.
Sure, SolarWinds is a corporation, it will have a Board of Directors representing the interests of its shareholders, it has at times been a private company (so its shareholders were some handful of private equity investment companies) and public (so they will have included funds and the great unwashed) but in either case the Directors are responsible.
So in terms of specific people that's Bill Bock, the chairman of that board at the time.
They did indeed replace their CEO. I would assume after agreeing to pay off the old one as this is the usual practice and, unlike some no-name intern, a handsomely compensated CEO can afford expensive lawyers if you try to kick them out uncompensated for incompetence based merely on the evidence that their inadequate oversight cost you billions of dollars.
No, we're all mostly just distracted by so many problems and failures throughout our society. Someone commented the other day mentioning how it doesn't take a genius to see the West is degenerating. To me it's obvious it's the industrial complexes involved in regulatory capture - fuelled by bad actors, greed, targeting the richest nation in history - leading to bad policy for people, individuals, for too many decades now; including policy allowing the duopoly to exist, thrive, and the tightening and expanding conglomerates of a handful of mainstream media companies also then perpetuating the two main narratives of the duopoly, along with whatever other for-profit interests decide to influence/manipulate us through shallow, cheap advertising. Thank Goodness for the internet, technology, to allow anyone with a voice a better chance at gaining a following and a communication channel - so we can bypass the control of those systems to educate people, and share different narratives - truths.
I worked at two big, well known software companies which had the same default password - company123, albeit for low security stuff, where you wanted wide access but a password was required for some reason.
Well, a lot of companies are doing it in reverse , by not properly securing test sites, and bringing live sensitive data from prod to test . I don't know which is worse
The point of being a CEO is that you are ultimately responsible for what decisions are made. Even if decisions are delegated. Because guess what, you as CEO are responsible for delegating correctly.
More recently, our politicians simply “don’t recall”, or worse directly lie to us.
For C-level folks, its simpler. Take no responsibility ever, unless forced to by the courts. Even then, taking actual responsibility is so rare that I have no examples.
That this is common in most organizations shows that most larger organizations end up being selection systems for filtering sociopathic to the top.
It is not because their are deliberately designed this way, but because this is what sociopaths seek, and the organization fails to actively filter against it.
Great article - I'd seen another on the S/C/L model, possibly by the same guy, and this one went into more depth - it's a really interesting insight - thx!
My pleasure. It was posted a long time ago on HN and also a few months ago. Actually I didn't notice the recent repost. See
https://hn.algolia.com/?q=Gervais
>It is not because their are deliberately designed this way, but because this is what sociopaths seek, and the organization fails to actively filter against it.
The sociopaths are the ones designing the organization and creating the legal and bureaucratic frameworks ostensibly meant to filter them out. That the end result nurtures and rewards them and allows them to use their subordinates as a bullet sponge seems entirely deliberate.
Exactly. It may not be direct responsibility - but it's still your fault that password policies weren't made obvious on the onboarding process for IT hires for example. It's your fault there isn't a culture of technical supervision, or regular auditing, etc etc.
I have never been a CEO so take this as a question rather than a statement. Someone in that position can lay our rules, regulations, governance standards etc for people under you, but how realistic is it that a CEO has any visibility on how they are implemented or enforced?
I think they know this is a pretty desperate story. Some of the other coverage [1] suggest Solarwinds had so many different overlapping security failures they may never be able to attribute to a single cause.
Still there's some interesting things that could help a (much smaller, less critical) software vendor decide where to focus their security efforts. Perhaps near the top of the list should be:
1. Who in your organisation has access to your build and distribution toolchain, and how secure are their credentials?
2. How good is your record keeping? Are all your builds traceable back to a specific revision in your source control, and are you keeping build logs somewhere they can't be tampered with?
As an intern, you usually work for free or for subpar money, and expect to be taught the basics of the job and some good fundamentals. The fact this intern lacked the security knowledge any mom and pop company employee should have suggests the company simply wanted free labour from them.
If your companies existence relies on interns not making mistakes. Security breach is the smallest of your issues.. Company board is probably like, fingers crossed, hope the other interns are good...
The last time an intern made such a disproportionate impact was in 1996.
So, who is responsible for giving an intern - likely not even strictly under an employment contact with the company - access at that level. And who, in turn is responsible for failing in their oversight duty of this situation? And who in turn is responsible for a the department that this fell under where such oversight duty failures are possible? And so on. Before long, you're back at the CEO, but that time the question will stick.
As a CEO you've got nobody to blame after your first 90 days of employment.
I don't think I have ever been more than one week in to a job before I was given some form of admin level rights.
In small companies I can understand that. But I've had it also happen in government jobs, in big companies and startups that collect a lot of sensitive personal data.
In one case I was with a startup that had names, addresses, DOBs, phone numbers and debit/card details in their DB. When they hired me they didn't ask for references or ID. I was given full admin rights to their Azure account on day one.
If I were in charge I would at the very least want to validate a new employees ID before giving them any form of access to IT systems, and only elevate access privileges once they had an established track record within the company.
As an intern? I would find it somewhat worrisome if an intern at some random company would be given access to mission critical systems and data within a week. That would definitely qualify as an oversight failure, and 'small' is a pretty relative affair, if the company has four people then I could see your point - maybe - but if it had 30 or more than it really isn't ok.
The fun thing about being in charge is that everything is your fault. Blaming the intern, the person least able to defend themselves, is ridiculous. If an intern can destroy your business, you're doing it wrong.
What’s more damaging is that a simple password is the only thing keeping your business from destruction, not to mention exposing all Americans with a security risk.
According to the article, SolarWinds doesn't seem to think that the password itself is a problem, only that it was leaked. And they "took it down", that sounds as if they deleted the leaked document. It doesn't sound as if they changed the password.
Seriously. Blaming the intern is a bad look, but here it reinforces the idea that they don’t even understand what was wrong. The fact that it leaked is only relevant is that it shows 1) how bad their password is and 2) how deficient their process is for dealing with (let’s face it, inevitable) leaks.
> Current and former top executives at SolarWinds are blaming a company intern for a critical lapse in password security that apparently went undiagnosed for years.
The SolarWinds execs have provided us with keen insight for the root cause of the SolarWinds attack, but the insight they conveyed is probably not the insight they intended to convey.
> "They violated our password policies and they posted that password on an internal, on their own private Github account," Thompson said.
So let's analyze the various scenarios under which the "intern" might have been responsible, and why each one is bullshit.
1) The intern exposed a company password on their own account.
Counter: What kind of "password policies" allowed such a weak password in the first place?
2) The intern came up with the weak password themselves, thus violating "password policies" not just for secrecy but strength/security. This password was then used for several critical, production applications.
Counter: Why was an intern in charge of deciding a password used for anything critical?
3) The intern came up with the weak password and exposed it, but it was only used for the intern's own corporate accounts (e.g. their Windows workstation).
Counter: Why did an intern have a level of access such that their account being breached could lead to this level of compromise/exfiltration?
Conclusion:
There is no conceivable scenario under which this makes sense.
Worst scenario: The intern is malevolent. Counter: Credentials should be given progressively as trust is built (and as lessons are learnt), and an intern can’t have access to production.
I have an employee who I thought I could give more responsibilities, but he keeps not locking his computer when he walks away. He has very limited access to everything and it would impede his career if he didn’t also have the same attitude about other issues. (My question is, how do I make him diligent — It’s real potential wasted).
Further, have you really listened to his explanation of why he isn't bothered about locking his computer?
It might be that in your organisation this is a cargo-cult security practice that he's not bothered by because he knows it's not an effective practice.
Or it could be that he knows he doesn't have enough permissions to do any damage, so he doesn't bother locking his computer. Trusting him with a little responsibility might change that.
This isn't a solution you can implement yourself since it sounds like you're in a position of power over the employee and it would be harassment. But having a culture where leaving computers unlocked and unattended is an open invitation to getting embarrassing YouTube videos opened on full screen by your peers works wonders for getting people to lock their computers.
I wish I had an answer for the second part; it's hard to see someone with talent be the one to get laid off after months of asking them to be more diligent in their work.
> I wish I had an answer for the second part; it's hard to see someone with talent be the one to get laid off after months of asking them to be more diligent in their work.
Oof, this hits close to home. I don't have an answer either.
The harassment would be the doing embarrassing things with the unlocked computer. A manager or team lead shouldn't be picking on their employees like that. It's something more OK for a peer to do. Ironically, it's the interns who I've seen get into it the most and make it a game. They really got a kick out catching the full timers.
Harassment has to be a repeated and unwanted action against someone. If it's one time, it's not harassment. If you don't speak up against the perceived harassment, can it even be considered harassment? Also would you rather be fired instead? There's real life implications to not locking his computer.
You were replying to laurent92, who didn't say anything like that (picking on employees or embarrassing them); that was this other comment, which was replying to laurent92:
At a company I used to work for, if someone left their computer unlocked, we'd send the donut emoji into the #everyone channel on slack from their computer, and they'd have to buy donuts for the office
For us, it’s teapots. Inspired by the info set who noted his cube mate couldn’t learn to lock his screen and started sending, from his open laptop, “I’m a little teapot...” first to just him and then the wider org when he failed to adjust his behavior.
We had a similar thing at one gamedev place that I worked at where and email would go out to the team if you left your computer unlocked(I forget the exact phrase but it was fairly silly).
We had shared offices and one of the programmers had the office right next to the kitchen. One day we all heard the senior programmer shout "WHAT THE FUCK!" and all ran over to see what had happened.
It turns out one of our engineers had walked into the kitchen and left his computer unlocked. The senior developer seeing this had opened up outlook, started a new message and began typing in the subject. What he didn't know is the developer had hand-rolled a keylogger with a match pattern for the message that everyone would send and dispatched Windows+L via key injection to the main window loop.
The trap was sprung and the machine locked right in front of him as he typed the last letter unable to send the email.
There was all sorts of other shenanigans at that place(like a fake "April 2nd" firing, they got the person who did that back with an annoy-a-tron over a 6 month period) but that was one of the more memorable ones.
Well I've heard of one office that had an established rule that any unlocked laptop could be used to promise the rest of the team free cake on behalf of the person that forgot to lock their laptop.
If nothing else at least it promotes awareness (and cake!).
Why is locking your computer at work a security concern? It's certainly another layer in the onion but a rather weak one and certainly not one worth losing someone over. If it's so big an issue, get him a smart watch that works with their OS of choice and enforce a bluetooth device locking policy.
I agree, I’m sure it helps but seems like if a malicious coworker wants access they can trivially steal your password with a keylogger or just watching you type it in. May prevent spontaneous acts I guess but feels like if those are really a risk you’ve hired the wrong people.
> One company had an employee they fired for bad behaviour. A few months later all their production servers got wiped out. ... A month later, it happened again! Everything gone.
> What happened was that the rogue employee, months before they were fired, had waited until everyone went to lunch, walked around from one PC to the next, and collected the ssh private keys from any of them that weren’t locked. ... Then, a while after getting fired, he used a co-worker’s ssh key to login and destroy everything.
Of course, screen locking would not prevent a determined, disgruntled (or desperate) employee from pulling off something like this, but it would stop the casual collection of secrets while everyone is at lunch. Add full disk encryption to PCs and you have a reasonable chance at defending.
Oh, and if you're doing anything with credit cards, PCI DSS requires you to enforce a workstation locking policy. Failing to do so opens you up to massive liability in the event of a breach.
I strongly recommend just sitting him or her down and explaining why this is a problem (you've articulated it well here). Either they get more diligent (great) they can't get more diligent (not great) or they refuse to be more diligent (really not great).
I think that there's sometimes a tendency to overthink management topics - there's no need to overthink this one.
Also what is the password rotation policy if this password was valid for years and intern had access to it? Reasonably I would expect all of the shared passwords to be rotated every time some person leaves.
If Solar Winds CEO, CTO, and everyon below them had taken the responsibility of security even slightly seriously, they would have made the system secure against such leaks. Some really simple steps:
1) access restrictions such that even malicious interns, and certainly careless interns can do little damage *when* the inevitable leak happens
2) Actively scan everyone's online presence, and let them know that this is a requirement of employment.
3) Require 2FA
4) Much better training so it is reduced
5) Internally firewalled and airgapped systems
I could go on... but none of these were done
The fact that they blame the intern shows that they are insanely unqualified for any job related to any sort of security. These CxOs are active hazards in the industry.
> 1) access restrictions such that even malicious interns, and certainly careless interns can do little damage when the inevitable leak happens
this should be something that is implemented in any organisation beyond a very small scale, mainly because even if not malicious, people should not be able to make critical mistakes in systems they have no know how off.
The intern leaked the password, but how was he able to know this was critical information? Not to mention he should never have been put in that position in the first place.
Years ago, not long after I started at Amazon, there was a huge Netflix outage[0]. It surfaced - or at least was widely speculated - that the cause was a pretty green employee running a DROP TABLE command against a prod database instead of a dev environment.
One morning when I came in and sat down at my desk, all of the old-timers were having coffee and discussing the fiasco. I was very happy to hear all of them talk about how mistakes happen, and the last person to be blamed for such an outage is the poor guy or gal that hit the ENTER button. Rather, blame falls (to various degrees) on: the engineers in their orbit who should be backing them up; the managers helping to onboard them; the chain of command; the entire system that is in place to prevent inappropriate access.
One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way.
> One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way.
owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion.
Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers.
Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous.
Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents.
Something we tell employees during our onboarding in a technical comes down to this.
- reason about a problem by yourself first
- think about impact before you do a change, if in doubt, ask and doublecheck.
- admit mistakes when you realise them, explain your reasoning and why you did the action.
- learn from your mistakes, but accept that being error-free is simply not possible.
To be fair sometimes people hyper focus on refactoring rather than fixing business problems, which isnt really helpful either, have to find a decent middle ground.
> owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers.
If you are in the right company. I made a big mistake at one point. We had all of the people responsible for one of the payment methods away on holiday (this one payment method was managed by another team in another country, as it wasn't using our normal payment gateway and provider that my team was maintaining).
Huge panic, someone needs to fix this, you can't use X any more for completing your order. I'm in the right team that's handling payments and fulfilment, only one at work at that point so I'm told to fix it. I do, I fix it, send it over to testing, get the green light, fix is deployed, everyone is happy.
2 hours later, we figure out the payments are working, but the orders are not being finalised and are still in unpaid. We realised that that single payment method done by this other team in another country was not using our standard payment processing workflow and it has a different way of actually getting the confirmation from the payment provider. This was quite a big company, we had around tens of thousands of Euros blocked in those 2 hours. I own up to it, I admin I made a mistake, go deeper (we did not have anything about this documented) fix it again, we unblock everything, all good. Until 1 month later I got fired (there were layoffs because of larger financial issues, but I was on the list because of the incident), it was the only time in my career this happened to me.
In the same time, someone else in my team made a mistake, hid it from management, even though we knew about it internally, fixed it and bragged about fixing the issue (no mention of him being the one who caused it) and got somehow (didn't even know we had such a thing) employee of the month and big praise in the next department meeting from management.
My lesson from this? Screw these companies and the people running them. I was asked to help, I did it, I made a mistake, fully aware of that, but then I'm the only one thrown under the bus for it.
Leaving that company was one of the best things that happened to me. It also thought me that when a company, especially a big one, blames a low level employee for a big mistake or some visible incident, there is something very wrong there. There is usually so much politics, so many layers of management, that blaming one person that does some actual work is the easiest way of hiding your actual problems.
in my experience this is detrimental for a company in the long term. It results in people who are in charge of taking leadership not actually leading the company and it shows everyone in the company taking risks is instant failure in their eyes.
To give you a counterexample.
The same company my prior example came from, also had some other "silly mistakes" made by an intern.
He had to do inventory of a couple of old servers and remove hard disks from these servers. The servers where to be sold.
Sadly, no one told him we had additional servers in the back of the storage room which he forgot to check because they where not on the same pallet as the batch he was told to check.
Result, an couple of servers got sold with disks still in them. Luckely the company we sold to was friendly enough to give us a headsup about it and it resulted in no further issues, but still.
Our company director personally took this as a reason to spearhead a plan about improving operational security and change processes (Aka, remove the hard disks when the machines are put out of service instead of half a decade later when their sold).
The intern felt pretty bummed and thought he was responsible for the mistake, but in my opinion he done the job that was asked of him, he just got incomplete instructions. This was also explicitly communicated with him by his direct supervisor.
In my experience, not throwing people under the bus to hide organizational or process failure, but simply admitting the processes could be better and striving for improvement does absolute wonders for morale and team building.
Being perfect is impossible, organizations should keep people to impossible standards, especially to hide incompetence.
Great for you. The company that I currently work hired a new manager, he is amazing on how he shows that we are a team and a "person" mistake, is actually, the process which need better thinking/layout.
> Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents.
Bingo. I’ve made mistakes that have taken down systems or caused them to silently fail. The worst mistake I’ve made took down basically my entire company for about 20 minutes. This turned out not to be critical, because our site was still operating, and it was just external data feeds that weren’t getting updated, but I freaked out about it for a minute. After that minute, I went and got help, and we fixed it. Had I not, I probably could have fixed it myself, but it would have taken much longer and cost much more than it did.
If you’re in an environment that doesn’t recognize that, you aren’t in a place that actually values and understands engineering work.
I've also found that being forthright about your mistakes gives credence to your _disavowal_ of responsibility. When it's not your fault and everybody's looking for a root cause, if you're known as someone who is quick to admit fault, then when you deny it everybody accepts it.
The CEO blaming an intern is the big thing that looks bad, but it's also very telling of their culture and how they got in this situation in the first place.
The CEO saying this, they are inadvertently signaling that they want a certain class of customer, a customer who agrees with and would make similar statements.
Any customer that can see through the BS will immediately turn around on their heels. And SolarWinds will be happy that they just lost that problem customer.
SolarWinds is looking for an Equifax not a Netflix.
At well managed big tech companies, stepping on a landmine like that is not only not detrimental to your career, but it can actually help you (unless you didn’t do that by mistake, but by being reckless and on purpose skipping all the protection layers). You can own your accident and, depending on the complexity, create whole teams to properly fix it, so no one else can cause an outrage like that.
My colleague typically asks this question in interviews, and I've occasionally borrowed it from him: tell me about the last problem you caused and how it got resolved. It's one of those questions designed to just get them talking and there aren't too many wrong answers, except maybe, you know, not being able to think of anything.
I was on the other side of that Netflix outage, as the main contact point between Netflix and AWS at the time. Amazon did give us a detailed rundown of what happened, but was very specific to not name names, nor did we ask. We all agreed it was an excellent learning opportunity for both AWS and Netflix.
That outage is what drove us to rearchitect all of Netflix to be multi-region.
I am deeply ambivalent about my tenure at Amazon overall, but I unconditionally applaud the level of psychological safety in their dev culture: there is a shared understanding that any system failure caused by a single person is necessarily a failure of the process that governs changes to that system.
If there were any singular behavior I wish every tech company would adopt, it is that level of perspective on what makes a system reliable.
My experience there was overall very negative. My comment above, though, was actually a very bright spot, and I want to call out the positives that exist.
At the time, I had several years in the industry under my belt but was new to big tech companies, and especially the tech that Amazon had built. I lacked confidence in myself but I was super conscious of actively seeking out the help of others. And it failed spectacularly - one SDE3 in my team with whom I was sort of paired would quite literally back away slowly when I was showing him an issue I needed his advice on.
Big companies like that have such variation in teams, and I think I just got rather unlucky in my experience.
Absolutely right, and further, we in tech know better as we could walk into so many big name shops and not be surprised to find huge, simple, obvious security holes. We know it’s standard operating procedure too often, so this blaming an intern rings hollow.
Agreed. All this statement does is reinforce the speculation that there was a failure of process but adds to that that there is a culture of blame rather than a culture of improvement.
If a mistake happens, don't blame the individual, blame the process then find a way to fix that process. If a company has a blame culture people spend more time covering their own arse instead of building safer processes.
They said in the article they don't know if this password was even related to the breach (meaning all of these points are not really describing reality, especially #3). It's not very clear what the password was for from the article. Sounds like the CEO doesn't have a very good understanding of what Github is though unless this intern was POCing github enterprise or something though.
The same is true of passwords now since the invention of rainbow tables. The best way, aside from voluntary disclosure, to compromise any password is brute force. The best way to eliminate brute force is to require a long minimum character count and to hash passwords using a 512bit hash algorithm.
Imagine the size of a rainbow table of SHA512 or SHA3-512 hashes for a 60 character password. A 60 character password could be as simple as:
* I enjoy driving my tiny white car with a standard transmission.
* My big cat, Ace, really sleeps a lot during the day while I work!
* Growing up my favorite song was Time by Pink Floyd about regret :(
For most passwords, you should be using a password manager, which means long, high entropy passwords (which will not be memorable as a result).
For the few that you need to manually enter, something long is good, but it should ideally use characters or different classes, and ideally not be comprised solely of dictionary words (which your first example is), otherwise the search space is greatly reduced.
Also, manually entering a 60 character password is not going to be fun :) I think the longest "manual enter" password I have is 25 characters, and it's a PITA to enter in a password field!
You only need that to impose a greater character width against brute force attacks. That is the only value in high entropy.
The actual reason people think they need this is because it was written into a NIST publication a very long time ago and it just became common practice. As a proof of concept what is the published standard that imposes that practice? I bet you think you need this but cannot find the written standard guidance suggesting it.
The guy who originally wrote that standard later came out and said it was a mistake. Bad advise that he wishes he could take back, but it’s too late everybody thinks they need it and they don’t know why or where that guidance even comes from.
> and it's a PITA to enter in a password field!
Only on a touch screen.
My first example also contains uppercase and punctuation. Think of it like IPv6. When the key space is large enough you don’t need a bunch of bullshit and gimmicks to ensure uniqueness.
> You only need that to impose a greater character width against brute force attacks. That is the only value in high entropy.
That's exactly the reason I meant - to make brute forcing take much longer.
> Only on a touch screen
Yep, certainly worse on a touch screen, but I hate entering long passwords even on a keyboard - it's OK if it's a field that let's you see what you type, but if it's just asterisks, then I find it hard not seeing that visual feedback.
> Counter: Why did an intern have a level of access such that their account being breached could lead to this level of compromise/exfiltration?
I've worker with interns whose major redeeming quality was that their internship was fixed length and would be over soon. I've also worker with interns who demonstrated ability and responsibility sufficient to get the same access that I had (and, clearly, an offer letter). That it was an intern doesn't mean the level of access was inappropriate; of course, if it were my intern, I would take blame for them leaving their password on github.
It shouldn't be usual. Temps don't get unsupervised access to sensitive systems. Too much risk of abuse and misuse from short-term less mature employees.
Are you a college student? Or an independent contractor? I don’t know how anyone who has worked in the real world can have these misconceptions.
1) Every password policy allows for dumb passwords in certain places. Because password policies are only enforced on systems that integrate with the password policy enforcement mechanism. Which never covers everything. Even with a password policy, it’s easy to make dumb passwords.
2) It doesn’t say that the intern chose this policy or that it belonged to something critical. There has been no link established between that password and the breach. A random researcher said they found the password a year ago and reported it. It could have been used, but there’s no reason to believe it’s relevant.
3) Nothing suggests that they did.
Conclusion: It’s better to not be an armchair quarterback after a breach, especially when it’s still under active investigation by actual professionals with access to actual data, and they aren’t even making the claims that folks here are making.
It shouldn't be possible for an intern to leak a password, or if it is, the blast radius should be limited. In other words, it's the job of senior people to make this sort of thing hard or at least not super damaging when it occurs.
222 comments
[ 3.5 ms ] story [ 196 ms ] threadIt is the fault of both of you. ¯\_(シ)_/¯
checks notes
... a single intern! then you're doing it wrong.
Whatever that means.
This would never have happened in the first place had they used an encrypted complex password and a simple password manager.
The whole company takes the hit with blunders like this. It's everyone's fault responsible for the infrastructure allowing this to happen in the first place. Those who pass the blame on others very quickly are equally to blame which means the CEO is just as to blame as the 'intern'.
Clearly the whole company doesn't train their interns.
The git repository just happened to be public.
It's entirely reasonable to think that the person in question possibly didn't even stop to think that Solarwinds123 was an actual secret that needed to be kept, as it is the equivalent of common passwords that are published publicly in manufacturer documentation.
I’d say in all the ways that matters this was basically everybody BUT the interns fault.
SolarWinds: “the intern revealed our ‘solarwinds123’ password.”
How do we solve this though? Who is the CEO responsible to - who do they care, if anyone, about how they appear to? How do we shift this behaviour for the examples of CEOs and politicians who maybe haven't been outed yet by such a security breach but are vulnerable to such lacking of a rigid structure of command/responsibility?
Edit to add: I'm beginning to think HN is full of really lazy people.
Because of downvotes? That's probably just because you brought partisan politics into the discussion.
Wut? Mentioning a politician’s name isn’t partisan politics. He said nothing about the politics of Cruz, he compared the Senator shifting blame onto his kids with this CEO blaming an intern. Don’t be like that.
(Edited to correct typo)
A much simpler explanation is that people who read and comment on HN do not want to see every conversation degenerate into yet another political fistfight, since it seems to have infected every other part of our lives. One of the core values of this community is that we mostly manage to avoid that.
It's interesting how I can reference, with no personal opinion attached to it, a factual, recent situation that's comparable to the thread - about a well-known person who happens to be a politician - happens to be a politician, and as you said, because you're exhausted of hearing about them, you'd prefer they be censored/avoided in conversation. It's interesting to say the least.
P.S. Triggered is a valid word, I used it in the correct context - and you were in fact triggered by using it and are trying to gate keep its use.
Of course it is, why would you think otherwise? By calling out a particular politician by name, especially a controversial one, OP introduced a political slant to the discussion. Could just as easily have said "this is a problem we commonly see with politicians, or other people in leadership positions" without calling a particular one out.
> Don’t be like that.
Like what, exactly? OP made a comment, and when it started to go grey he maligned the HN commentariat as a whole as being full of lazy people. All I did was help him understand what aspect of his comment was the likely trigger for the downvotes. HN by and large does not like political discussions, it's even right there in the moderation rules.
> Ted Cruz claimed it was his children's fault for pressuring him into their trip...
I'll bite. (Stick with me; I'll get to the point after some analysis that may be objectionable or seem unrelated.)
He's got responsibility to two parties: the citizens of his state and his children. Both are important in different ways, and the needs of each conflict in certain circumstances.
Did he want to get somewhere warm? Probably.
Did his children want their father with them? Probably.
Was he unable to advise while in Cancun? Doubtful.
Was he unable to legislate while in Cancun. He was not.
So, despite the inappropriate optics of his trip (and reflection of poor character for a leader of the state), it appears as though he made a judgement that favored his family and his self, presumably because he judged no apparent harm to the state by his actions.
I bring all this up because it's relevant to the parallel you bring: he blames his children for the choice. The problem is, neither you nor anyone else I'm aware of have demonstrated harm by his choice. So his "blaming" his children appears to reflect his choice between two conflicting priorities.
The intern, on the other hand, caused demonstrable harm and the CEO blames his own decisions on said intern.
Cruz did not blame lack of preparedness on his children. He blamed bad optics on his choice to acquiesce to his children. Far as I can tell, you're bringing a straw man to a knife fight.
I don't think your assertion is incorrect, that incompetence and lack of integrity are responsible for the CEO's choice, but I don't think you demonstrated that properly with your parallel. I find Ted Cruz to be pretty unpalatable, but I believe the hype around his trip is exaggerated, and your use of it to affirm your assertion is misplaced.
However, in an effort to make this more interesting, let's say the parallel is appropriate. I would, for the sake of argument, challenge that the fault, instead, lies in the hands of the people of Texas. If "the buck stops here", then let us remember that there is an entity above that of Ted Cruz, and that is the people of Texas.
If we truly believe that mistakes are not the fault of the individual who made the mistake, but rather the person at the top of the chain, then it stands to reason that politicians are not at fault, but rather the citizens that voted that politician in.
Since voters have ultimate responsibility to decide the way in which their community is governed—just as a CEO makes choices that bubble down—then it seems to me the people are ultimately responsible.
And in this context, your parallel seems even more misplaced, because the people of Texas (and the rest of the country) are blaming the "little guy" when those in charge of actually deciding who's hired and who's fired are the responsible parties.
Personally, I find the ultimate lack of integrity in "the people", because everyone blames "government" when "government" goes wrong, but fail to acknowledge that they are, in fact, said government.
I charge you to challenge my assertion and justify your parallel.
Great! where do we sign up to vote on legislation?
I'd like to clarify one thing that I think was nebulous in my post: while I think "the people" are the responsible party, I meant to suggest that choosing who is the responsible party at all is close to arbitrary.
That is to say, arguments over who's held responsible more reflects who we want to be responsible than who may actually be responsible, which is in fact shared among several people. (The CEO, the intern, the person who gave the intern the password, the person who configured the system to require a password that could be leaked, etc.)
We choose the CEO (or Ted Cruz, or whomever) because of our own biases, in spite of the fact that responsibility is shared, and in spite of the fact that who's "in charge" is nebulous at best.
We do the same when blaming the 12 US billionaires (or however many there are) for income inequality in the country, or blame all white people for racial injustice, and so on.
That doesn't mean there aren't responsible parties, or that people and groups should be free from criticism. The truth of the matter is that—no matter how justified it is to assign blame—it's exceedingly rare for anyone to take responsibility for their own failings.
This shouldn't necessarily contort to whom and how we assign blame. But, rather, it should inform us of the breadth of the problem, our own contributions to it, and help us become better citizens, instead of shift the blame as the CEO or Cruz did.
If we find that unpalatable, but don't acknowledge our own complicity, then all we've achieved is a pitchfork-laden witch hunt, and not actual direction towards resolution.
He chose 1.
1) He's a pushover and his children, that we can assume he loves, he doesn't have control or authority over them for whatever reason; and in fact they'd have control over him.
2) He's using his children as an excuse to avoid taking accountability, and the problem down the whole line is a lack of accountability.
I agree the necessity of being in-person is debatable, however regardless, the lack of common sense to understand how it would be easily perceived is quite astonishing.
You're blaming the intern here - in fact it was the policies and protocol that allowed that whole shenanigans to unfold how it did: it shouldn't have been the interns responsibility to begin with - protocol should have prevented it, protocol and process that in a chain of command/accountability should reach the CEO if the protocol is adequate.
Of course the hype about his trip is exaggerated, that's to be expected - perhaps we've all been attempted to be gaslighted by Trump for so long that this doesn't seem like a big deal, voted in representatives not being actively engaged in military fashion to protect and help "his" people. We're so used to unqualified, inadequate, incompetent politicians being voted into power - that this seems okay - how many people died so far, how much damage was caused leading to economic damage and reducing productivity of those harmed - especially if they have to fight insurance companies for damage; it speaks to Eric Weinstein's commentary that it's only during wartime that the true leaders appear out of necessity.
Indeed, the people of Texas have been manipulated, arguably controlled subtly for many decades, directed by the duopoly - crafted heavily by industrial complexes - to put relatively bad people into the spotlight, for the mainstream media complex, controlled by a handful of conglomerates, to then amplify the two main narratives of either side of the duopoly. So is it really the people's fault? Yes and no, of course with the right policy proposals all of this can be countered strongly, and the change will cascade quickly, exponentially - policy proposals such as Andrew Yang's core policies; it's all the yin-yang dance of the universe, people need to suffer to learn - to pay attention, and hopefully we're at a part of a cycle where healing and creation is rapidly approaching, hopefully this isn't just a foreshock - where Trump's wave is still building to a tsunami - and we're yet awaiting the major earthquake before everything destabilizes and perhaps we're going to lead into a civil war - and perhaps even global war will start in parallel, if democracies can't be stabilized enough to create multi-lateral trade agreements to economically funnel resources away from the known bad actors who aren't behaving or falling in line to peace and freedom.
It's complex, there are systems that most don't know or understand or consider - and these systems self-perpetuate like a virus to keep counter attacks at bay; the autonomous immune system of America and the world is failing, inadequate. Luckily the purpose and value of distributing power, control - separating Federal government from the States - and further diversifying risk via having many States with their own varying rules - allows neighbouring States to shine light/the truth onto the States that have been darkened, influenced, lead too far astray by bad actors (domestic and foreign).
It's a lack of integrity in the systems - these flows that lead to multi-generational trauma, karma - action, consequences - passed down from one generation to the next. It's why my focus for all my projects is health: there are still ripples passed through stress-trauma of parents of war onto their children, and onto their children, and so on. We are generating new active trauma while not doing enough to support new people born or migrating to our more/rel...
Looking at your other posts, it also seems that you easily show a pessimistic attitude, similar to how old people will complain about younger generations. No one responds positively to such generalizations.
Mentioning a political figure doesn't necessitate baiting one into a conversation - however yes, it's clear that people are tired, exhausted - and no longer actually read to comprehend, stunting critical thinking throughout what they're reading - and then making an assumption as to what they're reading, and then doing a lazy downvote for a dopamine hit to feel fulfilled - as if they were diligent, reaffirming their assumptions.
And I disagree that "no one responds positively to such generalizations" - it doesn't make them untrue, and usually I also reference the solution, policy proposals, to solve the problems.
When will IT companies learn not to use telnet, ftp, and easy to guess passwords on their intranet.
It is often the case after these breaches that HNers start assigning blame in ways that indicate their lack of practical experience in the security world. After 20 years working in security, from startups to fortune 500s, including the most well funded security teams in the world, EVERYONE has stupid password problems in some corners of their company. The password leak was absolutely not the CEOs mistake.
So in terms of specific people that's Bill Bock, the chairman of that board at the time.
They did indeed replace their CEO. I would assume after agreeing to pay off the old one as this is the usual practice and, unlike some no-name intern, a handsomely compensated CEO can afford expensive lawyers if you try to kick them out uncompensated for incompetence based merely on the evidence that their inadequate oversight cost you billions of dollars.
Are we all this blind?
Failure to realize this is shameful.
More recently, our politicians simply “don’t recall”, or worse directly lie to us.
For C-level folks, its simpler. Take no responsibility ever, unless forced to by the courts. Even then, taking actual responsibility is so rare that I have no examples.
That this is common in most organizations shows that most larger organizations end up being selection systems for filtering sociopathic to the top.
It is not because their are deliberately designed this way, but because this is what sociopaths seek, and the organization fails to actively filter against it.
The sociopaths are the ones designing the organization and creating the legal and bureaucratic frameworks ostensibly meant to filter them out. That the end result nurtures and rewards them and allows them to use their subordinates as a bullet sponge seems entirely deliberate.
The management team should be the first to be blamed when such incidents arise.
Throwing an intern under the bus is silly and disgusting.
Still there's some interesting things that could help a (much smaller, less critical) software vendor decide where to focus their security efforts. Perhaps near the top of the list should be: 1. Who in your organisation has access to your build and distribution toolchain, and how secure are their credentials? 2. How good is your record keeping? Are all your builds traceable back to a specific revision in your source control, and are you keeping build logs somewhere they can't be tampered with?
[1] https://www.bloomberg.com/opinion/articles/2021-02-26/deepen...
Rep. Katie Porter: "You and your company were supposed to be preventing the Russians from reading DoD emails!"
Microsoft President Brad Smith: "There is no indication, to my knowledge, that the DoD was attacked"
That's accountability! dilligence! Microsoft at its best.
I wouldn't. I'd blame senior technical leadership for putting processes into place that allow failures like that. Most especially, the CEO.
The last time an intern made such a disproportionate impact was in 1996.
As a CEO you've got nobody to blame after your first 90 days of employment.
In small companies I can understand that. But I've had it also happen in government jobs, in big companies and startups that collect a lot of sensitive personal data.
In one case I was with a startup that had names, addresses, DOBs, phone numbers and debit/card details in their DB. When they hired me they didn't ask for references or ID. I was given full admin rights to their Azure account on day one.
If I were in charge I would at the very least want to validate a new employees ID before giving them any form of access to IT systems, and only elevate access privileges once they had an established track record within the company.
Amazing.
The SolarWinds execs have provided us with keen insight for the root cause of the SolarWinds attack, but the insight they conveyed is probably not the insight they intended to convey.
So let's analyze the various scenarios under which the "intern" might have been responsible, and why each one is bullshit.
1) The intern exposed a company password on their own account.
Counter: What kind of "password policies" allowed such a weak password in the first place?
2) The intern came up with the weak password themselves, thus violating "password policies" not just for secrecy but strength/security. This password was then used for several critical, production applications.
Counter: Why was an intern in charge of deciding a password used for anything critical?
3) The intern came up with the weak password and exposed it, but it was only used for the intern's own corporate accounts (e.g. their Windows workstation).
Counter: Why did an intern have a level of access such that their account being breached could lead to this level of compromise/exfiltration?
Conclusion: There is no conceivable scenario under which this makes sense.
I have an employee who I thought I could give more responsibilities, but he keeps not locking his computer when he walks away. He has very limited access to everything and it would impede his career if he didn’t also have the same attitude about other issues. (My question is, how do I make him diligent — It’s real potential wasted).
It might be that in your organisation this is a cargo-cult security practice that he's not bothered by because he knows it's not an effective practice.
Or it could be that he knows he doesn't have enough permissions to do any damage, so he doesn't bother locking his computer. Trusting him with a little responsibility might change that.
I wish I had an answer for the second part; it's hard to see someone with talent be the one to get laid off after months of asking them to be more diligent in their work.
Oof, this hits close to home. I don't have an answer either.
https://news.ycombinator.com/item?id=26286183
Edit: Also singing over the top praises of employees using the victim's account. "MonkeyButton is truly the best coworker I have ever had "
All this fun has gone away now with Covid and remote working.
We had a similar thing at one gamedev place that I worked at where and email would go out to the team if you left your computer unlocked(I forget the exact phrase but it was fairly silly).
We had shared offices and one of the programmers had the office right next to the kitchen. One day we all heard the senior programmer shout "WHAT THE FUCK!" and all ran over to see what had happened.
It turns out one of our engineers had walked into the kitchen and left his computer unlocked. The senior developer seeing this had opened up outlook, started a new message and began typing in the subject. What he didn't know is the developer had hand-rolled a keylogger with a match pattern for the message that everyone would send and dispatched Windows+L via key injection to the main window loop.
The trap was sprung and the machine locked right in front of him as he typed the last letter unable to send the email.
There was all sorts of other shenanigans at that place(like a fake "April 2nd" firing, they got the person who did that back with an annoy-a-tron over a 6 month period) but that was one of the more memorable ones.
If nothing else at least it promotes awareness (and cake!).
Physical access is considered game over, no?
For example, posted yesterday: https://tailscale.com/blog/rotate-ssh-keys/ (https://news.ycombinator.com/item?id=26249380)
> One company had an employee they fired for bad behaviour. A few months later all their production servers got wiped out. ... A month later, it happened again! Everything gone.
> What happened was that the rogue employee, months before they were fired, had waited until everyone went to lunch, walked around from one PC to the next, and collected the ssh private keys from any of them that weren’t locked. ... Then, a while after getting fired, he used a co-worker’s ssh key to login and destroy everything.
Of course, screen locking would not prevent a determined, disgruntled (or desperate) employee from pulling off something like this, but it would stop the casual collection of secrets while everyone is at lunch. Add full disk encryption to PCs and you have a reasonable chance at defending.
Oh, and if you're doing anything with credit cards, PCI DSS requires you to enforce a workstation locking policy. Failing to do so opens you up to massive liability in the event of a breach.
I think that there's sometimes a tendency to overthink management topics - there's no need to overthink this one.
It's reasonably believable that they leaked their password because. Interns.
And the second factor?
1) access restrictions such that even malicious interns, and certainly careless interns can do little damage *when* the inevitable leak happens
2) Actively scan everyone's online presence, and let them know that this is a requirement of employment.
3) Require 2FA
4) Much better training so it is reduced
5) Internally firewalled and airgapped systems
I could go on... but none of these were done
The fact that they blame the intern shows that they are insanely unqualified for any job related to any sort of security. These CxOs are active hazards in the industry.
this should be something that is implemented in any organisation beyond a very small scale, mainly because even if not malicious, people should not be able to make critical mistakes in systems they have no know how off.
The intern leaked the password, but how was he able to know this was critical information? Not to mention he should never have been put in that position in the first place.
One morning when I came in and sat down at my desk, all of the old-timers were having coffee and discussing the fiasco. I was very happy to hear all of them talk about how mistakes happen, and the last person to be blamed for such an outage is the poor guy or gal that hit the ENTER button. Rather, blame falls (to various degrees) on: the engineers in their orbit who should be backing them up; the managers helping to onboard them; the chain of command; the entire system that is in place to prevent inappropriate access.
One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way.
[0] https://netflixtechblog.com/a-closer-look-at-the-christmas-e...
owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers.
Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents.
Something we tell employees during our onboarding in a technical comes down to this.
- reason about a problem by yourself first - think about impact before you do a change, if in doubt, ask and doublecheck. - admit mistakes when you realise them, explain your reasoning and why you did the action. - learn from your mistakes, but accept that being error-free is simply not possible.
Third option: they don't realise when they make a mistake, either because they are not smart enough or too full of themselves.
The only way to guaranteed never have mistakes in your code, is to not have any code at all.
If you are in the right company. I made a big mistake at one point. We had all of the people responsible for one of the payment methods away on holiday (this one payment method was managed by another team in another country, as it wasn't using our normal payment gateway and provider that my team was maintaining).
Huge panic, someone needs to fix this, you can't use X any more for completing your order. I'm in the right team that's handling payments and fulfilment, only one at work at that point so I'm told to fix it. I do, I fix it, send it over to testing, get the green light, fix is deployed, everyone is happy.
2 hours later, we figure out the payments are working, but the orders are not being finalised and are still in unpaid. We realised that that single payment method done by this other team in another country was not using our standard payment processing workflow and it has a different way of actually getting the confirmation from the payment provider. This was quite a big company, we had around tens of thousands of Euros blocked in those 2 hours. I own up to it, I admin I made a mistake, go deeper (we did not have anything about this documented) fix it again, we unblock everything, all good. Until 1 month later I got fired (there were layoffs because of larger financial issues, but I was on the list because of the incident), it was the only time in my career this happened to me.
In the same time, someone else in my team made a mistake, hid it from management, even though we knew about it internally, fixed it and bragged about fixing the issue (no mention of him being the one who caused it) and got somehow (didn't even know we had such a thing) employee of the month and big praise in the next department meeting from management.
My lesson from this? Screw these companies and the people running them. I was asked to help, I did it, I made a mistake, fully aware of that, but then I'm the only one thrown under the bus for it.
To give you a counterexample.
The same company my prior example came from, also had some other "silly mistakes" made by an intern. He had to do inventory of a couple of old servers and remove hard disks from these servers. The servers where to be sold.
Sadly, no one told him we had additional servers in the back of the storage room which he forgot to check because they where not on the same pallet as the batch he was told to check.
Result, an couple of servers got sold with disks still in them. Luckely the company we sold to was friendly enough to give us a headsup about it and it resulted in no further issues, but still. Our company director personally took this as a reason to spearhead a plan about improving operational security and change processes (Aka, remove the hard disks when the machines are put out of service instead of half a decade later when their sold).
The intern felt pretty bummed and thought he was responsible for the mistake, but in my opinion he done the job that was asked of him, he just got incomplete instructions. This was also explicitly communicated with him by his direct supervisor.
In my experience, not throwing people under the bus to hide organizational or process failure, but simply admitting the processes could be better and striving for improvement does absolute wonders for morale and team building.
Being perfect is impossible, organizations should keep people to impossible standards, especially to hide incompetence.
Bingo. I’ve made mistakes that have taken down systems or caused them to silently fail. The worst mistake I’ve made took down basically my entire company for about 20 minutes. This turned out not to be critical, because our site was still operating, and it was just external data feeds that weren’t getting updated, but I freaked out about it for a minute. After that minute, I went and got help, and we fixed it. Had I not, I probably could have fixed it myself, but it would have taken much longer and cost much more than it did.
If you’re in an environment that doesn’t recognize that, you aren’t in a place that actually values and understands engineering work.
Any customer that can see through the BS will immediately turn around on their heels. And SolarWinds will be happy that they just lost that problem customer.
SolarWinds is looking for an Equifax not a Netflix.
Every individual contributor in that company has just learned that they need to cover their ass for any action that could possibly go wrong.
The cultural outcome is that accountabilities will be spread across managers so that blame can’t be assigned to an individual.
That outage is what drove us to rearchitect all of Netflix to be multi-region.
If there were any singular behavior I wish every tech company would adopt, it is that level of perspective on what makes a system reliable.
At the time, I had several years in the industry under my belt but was new to big tech companies, and especially the tech that Amazon had built. I lacked confidence in myself but I was super conscious of actively seeking out the help of others. And it failed spectacularly - one SDE3 in my team with whom I was sort of paired would quite literally back away slowly when I was showing him an issue I needed his advice on.
Big companies like that have such variation in teams, and I think I just got rather unlucky in my experience.
If a mistake happens, don't blame the individual, blame the process then find a way to fix that process. If a company has a blame culture people spend more time covering their own arse instead of building safer processes.
Imagine the size of a rainbow table of SHA512 or SHA3-512 hashes for a 60 character password. A 60 character password could be as simple as:
* I enjoy driving my tiny white car with a standard transmission.
* My big cat, Ace, really sleeps a lot during the day while I work!
* Growing up my favorite song was Time by Pink Floyd about regret :(
For the few that you need to manually enter, something long is good, but it should ideally use characters or different classes, and ideally not be comprised solely of dictionary words (which your first example is), otherwise the search space is greatly reduced.
Also, manually entering a 60 character password is not going to be fun :) I think the longest "manual enter" password I have is 25 characters, and it's a PITA to enter in a password field!
You only need that to impose a greater character width against brute force attacks. That is the only value in high entropy.
The actual reason people think they need this is because it was written into a NIST publication a very long time ago and it just became common practice. As a proof of concept what is the published standard that imposes that practice? I bet you think you need this but cannot find the written standard guidance suggesting it.
The guy who originally wrote that standard later came out and said it was a mistake. Bad advise that he wishes he could take back, but it’s too late everybody thinks they need it and they don’t know why or where that guidance even comes from.
> and it's a PITA to enter in a password field!
Only on a touch screen.
My first example also contains uppercase and punctuation. Think of it like IPv6. When the key space is large enough you don’t need a bunch of bullshit and gimmicks to ensure uniqueness.
That's exactly the reason I meant - to make brute forcing take much longer.
> Only on a touch screen
Yep, certainly worse on a touch screen, but I hate entering long passwords even on a keyboard - it's OK if it's a field that let's you see what you type, but if it's just asterisks, then I find it hard not seeing that visual feedback.
"you can delegate authority, but you cannot delegate responsibility." [1]
This CEO is a very poor leader.
1. https://www.theleadermaker.com/you-cant-delegate-responsibil...
I've worker with interns whose major redeeming quality was that their internship was fixed length and would be over soon. I've also worker with interns who demonstrated ability and responsibility sufficient to get the same access that I had (and, clearly, an offer letter). That it was an intern doesn't mean the level of access was inappropriate; of course, if it were my intern, I would take blame for them leaving their password on github.
1) Every password policy allows for dumb passwords in certain places. Because password policies are only enforced on systems that integrate with the password policy enforcement mechanism. Which never covers everything. Even with a password policy, it’s easy to make dumb passwords.
2) It doesn’t say that the intern chose this policy or that it belonged to something critical. There has been no link established between that password and the breach. A random researcher said they found the password a year ago and reported it. It could have been used, but there’s no reason to believe it’s relevant.
3) Nothing suggests that they did.
Conclusion: It’s better to not be an armchair quarterback after a breach, especially when it’s still under active investigation by actual professionals with access to actual data, and they aren’t even making the claims that folks here are making.