Not specific to this post but...I'm sick of management (in general) for taking the lions share of profit for a the "responsibility" they carry and yet, all that responsibility comes with no blame when things go south.
It’s such a perverse filter: take accountability/blame and you get relegated. Pass the blame and market yourself as the one heroically fighting the dysfunction and you get promoted.
I wonder if there are any management/organizational systems which have accountability and graceful failure as positive requirements for promotion.
Somebody in my company calculated that 70% of the incidents at my company happened due to pushy, penalizing management forcing engineers to rush things out.
Incident reviews literally pointed out "being rushed".
To be fair, do "incident reviews" mean anything? Most people blame someone else when things go south. How many people would complete a survey with, "Honestly, I'm not a very good programmer, I had no idea this piece of code was exploit-able."
Yes, they do – if done well. These are supposed to be blameless postmortems, focusing on what we can do better in the system to prevent similar issues from happening in the future. The answer is never “get better programmers”. Typical action items from the couple of recent postmortems I led/were a part of were along the lines of “decrease this alert's threshold, so we know about it earlier” ; “we need to invest in a proper local testing environment” ; “let's make our service independent of that other service” etc.
A particular bit that I appreciate about my workplaces "incident review" process is that it is anonymous and non-punitave. They exist solely to understand the root causes that lead to an incident.
They also don't stop asking why, an employee did something stupid? Why? Were they not properly trained? Why not? Were they pressured? By whom and why? Why did they pressure the first? Etc, etc.
They're can of course be a parallel disciplinary investigation of the situation so requires, but they don't share notes and you're not obligated to tell them both the same story.
It's a model I wish more organizations would adopt. People make mistakes or deliberately do things that are wrong for all kinds of reasons. Pretending they don't or won't is a poor plan.
I have written my fair share of post-mortems with a conclusion of "my knowledge of the topic/domain was limited at the time of implementation and hence created issue X due to Y".
It all depends on the organisation's culture, if management is always looking for blame you'll get meaningless incident reviews caused by fear, if there's trust you'll get a larger share of honest reviews with actionable points to minimising the risk of it happening again.
> What is the share of newly created profits and seized market opportunities due to rushing things out?
If the latter is greater then the former, in some quantifiable sense, the company might be doing alright.
The problem is not what is the cost of the incidents themselves to the business. The problem is how management deals with it. In my company, they usually spend an enormous amount of time in incident reviews, trying to find the blame (usually engineers), adding more processes, firing someone, spending 6 months and $$ backfilling the role.
The true cost of poor management is often hidden in bureaucracy.
Literally just happened. We advised that a change had systemic impact and we needed a proper regression. Mgmt said the build addresses an urgent issue (there was a bug that caught the attention of a CEO from partner and they were squeezing the vice on our balls) and it’s their call and responsibility, but get it out there. Unfortunately, the release did cause problems, causing much (rightful) anger from our customers. We rolled it back.
Now engineering is subject to a witch hunt from senior management about quality concerns. Management who authorized is currently wearing a hot dog costume demanding to know who crashed the hot dog themed car through the wall.
> Now engineering is subject to a witch hunt from senior management about quality concerns. Management who authorized is currently wearing a hot dog costume demanding to know who crashed the hot dog themed car through the wall.
Management: "Roll-out fast, fast, fast so that I can look good. But if something fails in this fast roll out, we will find a way to beat it out of engineers."
> “During our investigation into suspicious use of this account, Okta Security identified that an employee had signed-in to their personal Google profile on the Chrome browser of their Okta-managed laptop,” Bradbury wrote. “The username and password of the service account had been saved into the employee’s personal Google account. The most likely avenue for exposure of this credential is the compromise of the employee’s personal Google account or personal device.”
Isn't the problem the fact that Google forced on people their "sign into chrome" (which is basically "sign into google") strategy? The attack wouldn't work on Firefox.
I don't get how a security monoculture with a single point of failure across the economy -- i.e., Okta -- is supposed to be a good thing. Whenever you create something like this, you create a giant target that gets hacked. LastPass got hacked. Okta got hacked. Equifax got hacked. The OPM got hacked. I get that security software is hard to get right, so it's not great if everyone is rolling their own. But there has to be a way for things to be more decentralized, and for any one target not to be so juicy.
17 comments
[ 3.0 ms ] story [ 51.8 ms ] threadI wonder if there are any management/organizational systems which have accountability and graceful failure as positive requirements for promotion.
Incident reviews literally pointed out "being rushed".
They also don't stop asking why, an employee did something stupid? Why? Were they not properly trained? Why not? Were they pressured? By whom and why? Why did they pressure the first? Etc, etc.
They're can of course be a parallel disciplinary investigation of the situation so requires, but they don't share notes and you're not obligated to tell them both the same story.
It's a model I wish more organizations would adopt. People make mistakes or deliberately do things that are wrong for all kinds of reasons. Pretending they don't or won't is a poor plan.
It all depends on the organisation's culture, if management is always looking for blame you'll get meaningless incident reviews caused by fear, if there's trust you'll get a larger share of honest reviews with actionable points to minimising the risk of it happening again.
What is the share of newly created profits and seized market opportunities due to rushing things out?
If the latter is greater then the former, in some quantifiable sense, the company might be doing alright.
If the latter is greater then the former, in some quantifiable sense, the company might be doing alright.
The problem is not what is the cost of the incidents themselves to the business. The problem is how management deals with it. In my company, they usually spend an enormous amount of time in incident reviews, trying to find the blame (usually engineers), adding more processes, firing someone, spending 6 months and $$ backfilling the role.
The true cost of poor management is often hidden in bureaucracy.
Now engineering is subject to a witch hunt from senior management about quality concerns. Management who authorized is currently wearing a hot dog costume demanding to know who crashed the hot dog themed car through the wall.
It’s always the same shit.
Management: "Roll-out fast, fast, fast so that I can look good. But if something fails in this fast roll out, we will find a way to beat it out of engineers."
Isn't the problem the fact that Google forced on people their "sign into chrome" (which is basically "sign into google") strategy? The attack wouldn't work on Firefox.