A friend of mine started writing his predictions and putting them in sealed, dated envelopes. He said the 3rd time he pulled one out Carnac[1] style, management actually started listening to him. Nobody really got "I predicted this 18 months ago" but the theatrics apparently drove the point home.
Only if the employee decides they were harassed. This was a "loophole" in a harassment training package a former employer used. Basically if you were of an ilk that will always be believed by HR you get a blank check on what to consider harassment. A classic some are more equal than others scenario.
Seems like you can read this 2 ways, the additional way being allowing workplace relationships that have a touching component to it. Regardless of how appropriate that is at work.
Is the lesson for native-chinese-speakers about speaking English? If so, interesting, I haven't actually seen this (presumably common?) oopsie before. It also hadn't occurred to me how crushing of an insult it is, but damn yeah it sure is.
A "thing" or "stuff" in Chinese is 东西 (dōngxi). That's literally "east-west" if you pick the individual characters apart. That's what the footnote refers to.
Calling somebody 不是个东西 (bùshì ge dōngxī) means something along the lines of being good for nothing, i.e. an insult. Translating it literally, it would be calling somebody "not a thing".
Sounds like a wonderful niche for a startup to innovate in and ensure up-to-date communication. Though I'm surprised such a service isn't already offered by Zendesk et al.
Log every intra-company communication, and if some communication was meant to go to department X, Y, Z, and it only went to X, Y, then a flag would be immediately raised to department Z's attention and whoever sent it.
Of course the personnel in department Z might review it but ignore it anyways, but at least now there's a paper trail of who's at fault.
An Exchange system already gets you 80% of the way there if you force all on-the-record communications via email.
> Of course the personnel in department Z might review it but ignore it
That's the problem right there. One of my clients had a large IT organization of over 1000+ employees, with strict change control rules, and procedures that were tracked in SN. Every time there was a change management meeting everyone who could possibly be effected would get an email from Service Now notifying them of the upcoming changes.
Pretty much engineer ignored those emails, because there was so much going on in the org you'd get dozens of emails in a week and most of them you'd only be tangentially effected by, meanwhile you had your work to do.
So the problem isn't getting the notifications out it's getting people to pay attention to them.
I don't see how that's a problem for the organization.
Individual preferences do vary, one ignores 90%, another 95%, another 100%. And the one who's ignoring 100% of them will likely eventually make a mistake that otherwise wouldn't have happened.
But it will be fairly straightforward to resolve, after all there's an extensive paper trail as the chain of custody seems clear. Assuming the "change management meeting" emails were the approved means of communication.
> I don't see how that's a problem for the organization.
IMO, one of the lessons that came out of Chernobyl is that it absolutely is a problem for the organization. Exposing people to too many "alarms" that are constantly going off will cause people to start ignoring them.
Part of good design is figuring out which things are truly important, and how to communicate that to the people who are supposed to be paying attention.
Alarm or not doesn't really matter. If a person is receiving a signal that does not affect them most of the time they WILL start to ignore that signal. Many will attempt to combat this with policies and consequences, "Make sure your reading these e-mails, or else!" but it's a fruitless endeavor. Humans will human. Better to recognize that and build your systems around it.
If someone makes the wrong decisions because they start ignoring signals then don't promote them or give them important coordinating responsibilities. Those who are capable of filtering out a larger fraction of noise do exist.
Of course there will always be folks whose preference is to read near 0% of their emails, but that doesn't imply organizations must be designed around them.
It's not about playing the blame-game, it's simply to make sure the personnel in responsible positions are those who can handle it, in a verifiable on-the-record way.
> If someone makes the wrong decisions because they start ignoring signals then don't promote them or give them important coordinating responsibilities. Those who are capable of filtering out a larger fraction of noise do exist.
This is simply wishful thinking. Outliers certainly exist, but the idea that there are sufficient number of them that you can just ignore human nature is a path to disaster. You'd have to somehow accurately measure not just who is opening these noisey e-mails, but what they are retaining from them, and measure it over a large period of time, knowing that the vast majority or going to fail. It's far cheaper and more reliable to fix your noisey system than to try to outwit human nature.
Yes. Make sure you are sending people notifications that are relevant to them, instead of sending all your employees the firehose and relying on them to pay close attention on the off chance something they need to know actually slips in.
Sure! The people administering the system define these types of rules according to the shape of your organization. It may surprise you to know this is a core feature of most change management software. It's not like "target your notifications to a specific group of users" is a novel idea.
How would you envision these 'types of rules according to the shape of your organization' be set in a way that doesn't cause endless politicking and horse-trading?
It's really weird that you think you can't decide who is responsible for dealing with a type of notification ahead of time without "endless politicking and horse-trading", but think that blasting everyone with every notification, then attempting to sort out responsibility after something has gone wrong will somehow not cause "endless politicking".
Again, this is something many businesses do already. They don't blast the whole company when a bank account balance is low, when a server's disk is full, etc, notifications are targeted to appropriate groups. The specifics about who gets what is going to vary based on your organization. I can't give you hard and fast rules that will work for every organization, but that doesn't mean it's somehow impossible or not worth doing.
The reason for implementing restrictive organizational procedures in the first place IS because nobody behaves like a 'rational human being' in such an environment for any significant duration.
'Everything' may be labelled or described as 'highly important and #urgent#' but they may not correspond to reality.
In a sufficiently large and complex organization there will always be some percentage, neither 0% nor 100%, of actually 'highly important and #urgent#' hidden among the pretenders.
The point of any such tracking system is to discover, via positive or negative selection, what is actual instead of what various flawed humans pretend.
It sounds like what you're suggesting is that, so long as you know who to blame for the problem, it doesn't really matter how bad the problem is when it hits you? Even if the company goes insolvent because of the problem, if you've got someone to point to and say "their fault", it's not a problem for the organisation?
That... doesn't sound like a great approach to me.
Well, there's supposed to be a group keeping track of the alerts about anyone who is doing badly, but one of them is ignoring 90% of their alerts, another 95%, and another 100%. It's OK though, because the system is built to handle that kind of problem - by keeping a paper trail of who is to blame. Marvellous.
In an ideal situation, you'd get all "planned maintenance" emails for things you care about and no emails for the rest of them.
That (probably) means that the system for dealing with planning maintenances (well, usually, "approving them") needs to have a sufficiently good understanding of what humans care about what changes.
At a previous job, the planned change tracking system was REALLY good at tracking what specific compute facility was going to be impacted by any specific change taking place in that facility. And had a really good way of allowing you to filter for "only places I have stuff running" (and I think, even some breakdown of general change types as well).
It was, however, not easy to get notification of "there will be maintenance on submarine cable C, taking it off-line for 4 hours" or "there will be maintenance at cable station CS, taking cables C1, C2, and C3 down for 3h". And as one of the things "we" (the team i worked in then) was doing was world-wide low latency replication of data, we did actually care that cable C was going to be down. But, the only way we could find out was "read all upcoming changes" and stick them in the team calendar.
Was it good? Eh, it worked. Was it the best process I've seen? Probably ,yes.
Let's suppose there was a magic software app that had notified the Service Desk ahead of time. It's magic because only they only got the notifications that mattered.
So what? How would that change anything? Service Desk would send strongly worded emails to... somebody? The Organizational Overmind was already getting warning messages. Daily. There's already a trail of emails and status reports. It wasn't a knowledge issue. It was a "there's no effective direction" problem.
I know, the funny part is it is still leagues better than what the client had before it HP Service Manager. Imagine something so bad that SN makes you feel happy in contrast.
I wonder what a good ServiceNow implementation looks like. I’ve been at a few enterprise orgs now and all their SNs are.. beyond terrible. The hosted SAAS performance is agonizingly slow. If this is ITIL personified… I’m aghast.
I've used one that was at least mediocre. Access requests were signed off (optionally) by your manager and then (optionally) by an authoriser of the service required, and then delivered automatically 99% of the time.
So you'd select "System Z Update Access", write a one liner of "I need update system Z to do my job supporting The Alpha Process", your manager would approve it (providing they agreed that your role did involve supporting the Alpha Process) and then someone from the System Z team would also approve it (providing they agreed that you needed that access to support that process), and then a few minutes later you'd be automatically added to the right role.
Traceable, secure, and fairly painless. Required a lot of setup for all of the roles and automation though, I believe.
Yes, and making changes to those processes over time will probably be dependent on consultants/staff who are no longer there, and their understanding of the Rube Goldberg configuration will take months/quarters to unravel. I've found that even simple requests -- add one field -- is considered a high risk change, usually because "the business". It's a weird world where the IS and business sides of the company can hold each other hostage.
It seems like there may be some differences in how we understand the meaning of 'logging', 'apathy', and 'finger pointing games'.
I am making some assumptions when discussing this topic, namely:
1. The CEO and executive leadership are actually competent, have at least average middle aged adult levels of patience, and don't have their daggers out for each other.
2. There exist more than one method of communication between each layer in the hierarchy.
3. The organization is engaged in normal business, not in the middle of an M&A deal, outside investigation, or bankruptcy proceedings.
It's considered rude when someone attempts to speak for the entire reader-base on HN for a topic that obviously attracts a wide range of views.
In reality, most large organizations do have at least a semi-competent leadership team most of the time. Unlike what is commonly supposed by junior level staff expressing their frustrations.
2 people constitutes "we", and there's at least one person other than me that has commented on you lack of awareness. So perhaps quit lecturing others on what is 'rude' when in reality you just don't like people telling you you aren't as insightful as your smug attitude conveys. A state which is regrettably common for junior level staff.
I don't usually try to be nice with semi-trolling replies but since this doesn't seem like a troll account I will put in an effort.
Everyone on HN can read the comment chain and see where you joined in, and who said what... so the claims are difficult to take seriously.
Even if you focused on my alleged lack of qualifications to comment it would have not been as self-discrediting.
Though I don't get riled up by oblique insults, that might not be the case for others, especially for a topic that might be linked to negative emotions in the reader-base.
My sentence on 'junior level staff' wasn't directed towards you, it was a reflection on why it can be difficult to see the grand scheme of things when the environment might restrict that.
I somehow often interpret "Put me in, Coach." as "Put me in coach." for moment before the feeling of "O, god no!" and memories of hours discomfort are replaced^W^H overlain with understanding it's meant as an appeal for being allowed to do the thing.
I would go with an incremental progression of something like 100 random devices a week for 17 weeks, so that people see the tidal wave eating others and suddenly "get it". Less overwhelming for the service/support desk folk too.
Or start with the "most critical" devices: those belonging to the highest-ranking users. They're the high-value hacking targets so it's only reasonable. Might unstick a few wallets better than hitting mostly rank-and-file for something they have no control over.
If you are this close to a deadline, it doesnt really matter anymore. There is no avoiding the iceberg. You can fix your organization for next time, which is what the org believe it has initiated.
Well, yes you don't do this two hours out, you'd need to do it some time in the 13 months they knew about it for it to work.
But at the end of the day, they did the job they were paid to do and were clear about the looming impact. It's not their job to also wipe their clients' metaphorical bottoms when they were ignored.
Based on the thread I think it was Legal and Compliance forcing it. It may have been regulatory required. Also they say they are managing on behalf of their client, so their position as mercenaries is just to follow orders.
TFA explains that this was self imposed a hard cutoff:
> For anyone wondering why we don’t just lift the compliance restrictions, we don’t specify it. Their Compliance department does
after a year-long grace period:
> The machines came to end of life about 12 months ago, and the company being a multi-billion dollar operation managed to eke out another year of manufacturer support. Mostly symbolic as they’re not exactly going to release custom firmware for a handful of devices. They then put a set-in-stone tombstone date on support. 12pm today.
This was imposed internally by the company's compliance and legal departments, TFAA is the executioner but the execution would be contractually mandated:
> They require, and have specified, a zero-tolerance for device non-compliance.
This means an unapproved batched and drawn-out phaseout would be a breach of contract.
Which might breach an SLA or itself be deliberate malfeasance. If I were in this position, I would want to be absolutely squeaky clean.
What one could try is to call the CEO directly. Or maybe try the legal backdoor: contact the general counsel, tell them that the contract says such-and-such, that you think the contract is well written and you intend to do what it says, but that the organization should be aware that it may cause a problem. If legal doesn’t know how to get the CEO’s attention, then something is very wrong.
The very first post of TFA lays out that they've been sending emergency-level alerts for a while (aside from that being a year-long issue):
> 4 meetings, 124 emails, and two phone calls a day for the last 14 days have warned them of this.
There's only so much you can do.
> If legal doesn’t know how to get the CEO’s attention, then something is very wrong.
From the thread legal (and / or compliance) is the setter of the issue, and was well aware that it would cause issues (for a minority), but they were not in charge of resolution. And from downthread posts, they likely extensively documented their warnings:
> oh I’m absolutely backing the horse with the 3 miles of email threads proving this
And methinks legal and compliance had very much planned for the issue coming to a head, because they were getting fed up with being blow off, and having to shoulder the legal or regulatory risk.
I disagree. In conservative companies, it would be common to meet in person and give a firm handshake, before taking out the required documents from a briefcase. It's sad to see this tradition evaporate.
I'm not sorry to see it go. Half the time the people in the meeting forget what was decided on because it's one of a thousand other things they had a meeting about and no one we wrote anything down.
There's value in face to face, but not when it's a wide reaching announcement like this.
Various people seem to think I'm blaming the person doing the work - nope, I didn't. Blame isn't the point and won't fix anything. But strategically the best option is to work with stakeholders towards incremental force, squeezing the trash compactor slowly. If stakeholders insist on backing everyone into the worst possible corner, then so be it; next time they'll probably listen.
But you keep implying that the onus is on the poster, who is a third-party service provider, to resolve this, or at least get everyone around the table. It's not -- the issue is between the company's compliance department and operations. All the stakeholders for this issue are inside the company, and the poster is not.
But that would involve actual planning. When do you do that?
Do it for 17 weeks before the out of support deadline? The users start screaming, we still have 17 weeks left before the deadline, how dare you disable us early! And they get enabled again.
Do it for 17 weeks after the deadline? The compliance people start screaming, you have 1600 devices out of compliance, we need them shut down now!
Github once did brownouts on features they were removing. For 12 hours, then later for 24 hours, they turn off the feature temporarily. The idea was to cause alarm bells to start loudly ringing for anyone still using it.
Why do you hate the concept? I've seen it done a few times in different areas and I think it's a neat way of notifying people in large organisations/ecosystems. As long as the brownout is done after the support period, i.e. it's "contractually" ok to do so, it seems like a good idea.
Brownouts can be very useful in finding impacted systems that may have been overlooked as well. Last month, I had to do some updates because a customer's API had moved to a new URL on a new server. My team and I identified (what we thought was) everything using the API and did the updates.
A week later, the customer notified us that they were still seeing some traffic on the old URL, but all they could give us was the IP address it was coming from. Unfortunately this IP address belonged to a server that hosts a lot of our smaller applications, so it didn't really help locate the offender. So I just added a firewall rule to block access to the IP address of the customer's old server, and sure enough I heard the scream 15 minutes later. Removed the rule to get that application back up and running, got it updated to the new URL, and all was good.
The CEO will have a convo with two of his managers that will resemble me when I talk to my two sons when they get into a fight and start pulling out all of their excuses.
Mostly unrelated, but I hate websites like this that think they're way of handling arrow keys for scrolling should be implemented over how every other web page does it. I lost my place so many times when I mindlessly tried to scroll again with the arrow keys.
And even though this website only contains his posts and nothing else, individual posts are minimised and have to be clicked on to unfold, which scrolls the entire page for me on mobile chrome and I have to find it again. It's a usability nightmare.
This website is a well known infosec Mastodon host. The linked site is to a specific person's feed but the site local feed, with many other individuals' posts is at https://infosec.exchange/public/local
Top right of the content bar has an eyeball icon with "show more for all", which expands them all at once, but agreed, this isn't great UI or UX (still better than twitter though!)
At a particular point the author specifically indicates that for the sake of reducing stuff that many readers might find uninteresting or too frequent, they will be condensing larger posts.
> Alright, things are moving faster now, so I might condense-toot to avoid pollution.
But I guess the chronically online need something to complain about.
It's not the condensation of content I'm complaining about, it's the fact that when you try to unfold a post the entire website scrolls and puts you somewhere else. That's just bad UI.
My reaction to this tweet was surprisingly intense. It's like the plot to a horror movie, or the 5 minute opening credit montage of a post-apocalyptic film.
I'm guessing upper management prioritized the update of these devices with downstream management rather than overburden them with other stuff.
So in the end, this is just one piss poor managed division abusing another piss poor managed division. Who gets the heat? Probably the lowest level people.
Why "wipe" them? That seems unnecessarily punative.
You can see the "don't give a shit I work with a predatory organization" oozing from everywhere.
The security guy is trying to claim that they've sent out many many notices, but really this is just an excuse to abuse other people in a machiavellian abusive organization.
"Service Desk is now aware that everyone else except them was aware, and now IT is absolutely incandescent." Whoops, missed an email and a meeting in there bucko.
And it's the SECOND company where this was "implemented" or "specced"? This sounds like someone checked a box or compliance or ass-covering upper management slid this under the table, but all the people it ACTUALLY AFFECTS didn't get any input or opinion on the matter. And when push came to shove over funding it that person had probably moved on to bigger and better things.
So since you get to do it, you seem to be gleefully doing it. Great job.
What exactly are you suggesting this person does? The policy to wipe clearly came from the company's compliance department. They warned them over and over what was about to happen, and went above and beyond doing it with multiple meetings and phone calls.
A gleeful feeling does come across, although the poster does claim it's not schadenfreude. They also mention they think plenty of notice was given out to various middle managers.
There are better ways to handle this, when sending the messages out. If the deadline for compliance was 31 Jan, then when sending comms out say the deadline is 31 Oct, and machines would be wiped after that. Then start wiping them, 10% of machines on 1st November, another 10% on 8th November, etc.
I think we can charitably call it watching a trainwreck unfold.
There is not necessarily any Schadenfreude in watching and reporting it. No one really needs to be taking pleasure, it's just hard to not pay attention to a train crash occuring in slow motion.
It's very natural to want to talk about something this stupid/bad. Rubber necking is extremely human.
I agree. Not a single concern for boots on the ground that are likely already squeezed, and now has the apparently abusive compliance and security departments fucking them over.
Now everyone in the affected chain gets a black mark on their "permanent records" and gets exposed at a time when likely layoffs are coming.
What I don't hear is "why can't they upgrade, and how can we help them upgrade", it's WE TOLD YOU, NOW YOU SUFFER.
Here's the kicker: it's 1600 devices. Ok, so they've been told for 13 months to do this. Well, let's do some math. That's 260 working days. Oh look, about 1600 working hours. So if you guys had simply upgraded a device an hour over the last year, this wouldn't have been the problem. Yes, that's not fair, but neither is what the person doing.
Security is the military arm of compliance. Finger pointing at compliance is a bit mendacious. Saying LOL it's not my fault, it was compliance. NOW WATCH ME DROP THE HAMMER BOOM.
I mean, I guess the guy is saying LOL I'm outsourced and not even in the company HAHAHA. Still, eff this guy for taking a bit of glee in this.
If I were uncharitable and cynical I would claim that it looks like somebody has a goal to implement central device procurement and management and have built a system to enable that to happen.
> Here's the kicker: it's 1600 devices. Ok, so they've been told for 13 months to do this
That is not really how I read it; the devices got an year extension because people had already failed to refresh them within the standard cycle. From the sounds of it these are typical workstations etc, their support cycles are very predictable and if you bought some crappy ones without predictable lifecycle that is on you too. That extension should have been wakeup call, the process had already failed then.
Not sure if the devices are laptops or phones, so assume $400/device and $50/hour time, that's about a $700,000 dollars.
"If you had simply spent nearly three quarters of a million dollars of your own money, done 40 weeks of volunteer overtime on top of your normal job, without any purchase approval, without the authority to do that, and no guarantee of seeing that money back, this wouldn't have been a problem, so fuck you"
is a terrible take all around.
> "What I don't hear is "why can't they upgrade, and how can we help them upgrade""
We know why they can't upgrade, because the departments responsible for purchasing the upgrades won't agree to spend the money. This isn't something which can be helped by more technical input.
> "Finger pointing at compliance is a bit mendacious"
"mendacious: not telling the truth; lying." - nope, wrong. Legal and Compliance say it must be done and you must do it, and have the authority to do that. Pointing fingers at them is honest and appropriate, that is where the instruction is coming from (legal) and the reason why the instruction exists (compliance with internal or external regulations).
I'm not saying the guy is the second coming of Hitler. I mean is it his job to care? Not really. Is the absurdity humorous? Maybe he communicated it wrong? It's twitter.
It's more that security teams tend to have uncooperative, aggressive, authoritatian, and punative dispositions. I think ye old security industry had its roots in three letter government agencies which are used to conformance, policy hammers, and enemies of the state.
But when you add that to an organization already rife with infighting, dissatisfaction, and frustration, it will just lead to more resentment and your employees become your enemies.
The biggest security threats these days aren't leet hackers exploiting 0days, or even the county password inspector conning his way in. It's overworked angry pissed off employees leaving the door open. It's like Princess Leia said: the tighter you squeeze, the more people you lose.
If compliance and legal say to wipe the laptops, and everyone with a budget was aware of it for a year, it's not reasonable to put the disaster on whoever was in charge of implementing policy.
This is not a Petrov situation, you're not saving the world by going out of your way to be the person that will defy Compliance today, just because the policy is really dumb.
The people locked out would be shortsighted to blame the random security guy. They joined a big company with a very strict compliance machine, not a startup where you move fast and break things, then ask legal for forgiveness.
Big organizations are dysfunctional, news at 11. Don't blame a random IC for executing policy after considerable warnings. If communication is so thoroughly broken internally, and no one wants to take responsability for necessary spending, it's not the job of some random security guy to fix that internal dysfunction.
It seems to me that compliance was fully aware of what was going to happen and wanted to set an example. An expensive example, but apparently still spare change for the company, so it was well calibrated.
> Why "wipe" them? That seems unnecessarily punative.
You're proposing to remove the devices from the cloud management service, probably including admin lockouts and anti-malware software, but leaving all the data on them?
Perhaps I've misunderstood, but if you're warned repeatedly that you'll lose access to your device(s) and haven't taken any action, I have to think you don't find it very important.
That’s not how it works. Your manager’s boss’s boss was warned a year ago that a dozen laptops used by people in his department were going to go end–of–life at the end of this month. Nobody warned you about it at all; you were just plugging away at whatever tasks were assigned to you by your manager. Your manager might have known about it but was probably only told that you would be getting a new laptop “soon”. Someone was supposed to be taking care of it, but nobody really knew who, or when, etc. So when your laptop didn’t work right this morning, you called the tech support department, who ironically were the only department who didn’t know this problem was coming.
I tend to think that was a misplay on the part of the original author. If they had notified the 1647 users that their machines would be wiped a month in advance, then a bottom up pressure to get it resolved would have occurred. Few folks are as invested in their daily work as the people who will be blocked.
> "So for a whole year, they knew this was coming.
But nobody wants all that additional spend, so close to year end. Departments bickering over who’s responsibility it was, who’s budget it came out of, and so on. So everyone dug their heels in, and we continued to shout “iceberg!” from the sidelines."
I've seen this play out multiple times over the years. What's the solution?
Correct answer. You do not win by playing a defensive game, and it’s essential to realise that the organisation’s goals are not your goals.
The correct strategy here is to go on the offensive. Make friends with your manager’s manager’s manager. Just go full on social bribery mode. Invite them for Christmas dinner. Even if they decline, it will make them remember you, and next time, ask them to a barbecue or a picnic instead, and they’ll say yes. If you can’t throw that high, shoot for your manager’s manager. Once you’re in, get the dagger in your manager’s back, but only once you’ve made them a pariah, take their role, and repeat until you retire wealthy. Remember to take every opportunity to accrue political capital.
Eventually you will be in a position to fire people who make poor decisions, but you won’t, because the salary is good, and retirement is only a few years off.
You literally cannot prevent this behaviour in any human-operated organisation of any scale beyond a fistful of people - you can only co-opt it.
Leadership. Unfortunately most companies with bloated middle management layers spread the responsibility so thin that the level of consensus required to take action would stump the reincarnation of George Washington, Agustus, Ghandi and Genghis Khan combined (granted Genghis would probably just murder anyone who opposed him, which sadly would probably improve a lot of companies).
Even in my program of just north of 100 people broken infrastructure follows that same pattern. Something moderately breaks, the devs complain, they are sympathetically told to make do. Rinse and repeat until the devs realize their complaints never get addressed and stop complaining past a quick email. Then something REALLY breaks with no workaround, devs mention it's been heading this way for a while, and management, all aghast, exclaims "well why didn't you say so earlier if this was such a problem?". It's to the point where we (the devs) have started keeping locally archived email records just for the "told you so". Which of course makes us no friends because we point the blame where it belongs, so we're officially covered but our complaints get listened to even less. And the infrastructure is fixed just enough to limp along until the next catastrophic explosion.
You need someone who gives a shit with the power to crack the whip. In short, you need to give someone enough power that they can potentially abuse it, something modern business is allergic to.
I think they should focus on fixing the "gives a shit" part of it.
There is very little reason for an employee to care about preventing a failure they will not be directly held accountable for. I don't care if we lose clients. I don't care if we get hacked.
My life is not impacted one way or another by whether the divisions I work in succeed or fail, unless they utterly fail.
So if this Intune thing landed on my desk, I would do nothing about it. Give me some incentive to care and I would.
Your post describes the problem more succinctly than you might think.
You are trying to do a good job which might lead to a promotion. You see the rules and operations of the organisation as something to work within.
The people who ignore you are trying to get promoted. They see the rules and operations of the organisation as a irritating backdrop to their personal goals. The job could be anything. Ascent is all that matters.
They will get promoted. You will retire one day, your nerves fried.
The incentives are all wrong, and playing the social metagame rather than playing the game by the rules always results in an advantage, and thus this behaviour is inherently embedded in any human structure.
The solution isn’t “better management” - it’s in fundamental societal change, which ain’t coming any time soon.
It’s tractable, sure, but it doesn’t solve the inherent problem of human nature and tribal dynamics.
Put an impartial AI in charge and it will make the right decisions — and people will riot over the injustice. Ultimately, you’d have to go machine the whole way and the humans can all go bake bread and have wars over that or something.
I remember asking the facilities person responsible about an office move 3 times. I was ignored. I emailed a manager about moving my office. I was polite. I was ignored again. A coworker suggested emailing the department head.
Within a minute the deptartment head forwarded my request to the manager who was ignoring me and told him to take care of it. It was promptly. I got the email chain when it was done, there was some comment from the manager who ignored me to the facilites person who ignored me, that it "should never have been allowed to escalate.."
The company I worked for got bought out, and new management took over (of course). I kept raising issues about how broken the new "Agile development system" was. My complaints hit a VP level, who was "looking into it." Until said VP was "let go" and replaced. New VP said, "How unfortunate" to my complaints.
So much for escalation. I no longer work there (I left; I wasn't fired).
Most managers aren't leaders. Yes, leadership is the solution here, and it's not cringeworthy. Manageship is.
In dysfunctional organizations, the management structure exists to rein in true leaders. In a healthy organization, leaders are recognized and supported by management, whether they're in management positions themselves or not.
Success and dysfunction are often bedfellows. Look at some of the most successful companies, and internally, they are an outright shit-show. But - for many different reasons - they make a shit-ton of money and defeat all competitors.
Like, with these 2,000 people locked out of devices? Assuming they're contractors? It probably won't affect the bottom line at all. A minor nuisance, monetarily. Easy to route around; just work people harder until the problem is solved, or take a minor write-off and play Tetris with the books to make it look like you actually made money rather than lost it.
But if you want to have a company where people are happy to work, you absolutely must have non-asshole leadership.
Not always. Said by people who worked at tesla/spacex, many engineers and tech people have to dance around, present correct decisions such a way that the asshat at top realises it and does it. Lots of theatre.
If they don't do that, the guy just does whatever he want, often leading horrible outcomes. It's only working because there are enough people who are passionate enough for the project and want it to succeed. Such people would leave if that was just a software company.
You need someone who gives a shit with the power to spend money. That's pretty much it. Every single year, a company costs more to operate than it did the year before. Every piece of software you buy costs more over time. Same thing with hardware. Same thing with leases. Same thing with desks, chairs, trash cans, pencils, paper towels, and on and on. Yet I've watched many companies try to implement a "flat" budget, meaning no additional spend compared to the year before. Works fine if you've got room to cut, but do that over a long enough period of time (really not that long, two or three years max), and you'll end up with a situation where critical infrastructure is running on hardware that's so old it wouldn't survive a reboot or a gentle move from one rack to another. You can have great leaders up and down the chain, all of them saying, "Hey! This thing is probably going to break soon and it's going to ruin the whole year's sales forecast 'cause we won't be able to ship a damn thing!" but if that messages gets to the CFO, and they'd rather roll the dice on one more year of "lean" operation before they jump ship well...that's pretty much that.
Also, no company I've ever worked for makes their budget after asking people what they need. Instead, the Finance team just copy-pastes last year's budget, usually with a little haircut off the top for "efficiency." So let's say you run a team that depends on a handful of servers. You bought the current ones five years ago, and it cost $500k. Now it's time to replace them. But since you didn't buy $500k in servers last year, it's not in the budget for this year, so now your entirely reasonable request is being scrutinized as "unplanned spend". Several times I've seen teams try to build a plan for the upcoming year, only to find that there's nobody to give it to, because the "budgeting process" is done in about a week's time, usually two or three weeks after the fiscal year starts and with commensurate effort.
Some low level guy using an opportunity to alert upper / upper-middle management, more or less backed by his or her middle manager.
Seen some years ago. A mail was then sent from CEO to CxO along the lines "it seems there's something hiding under a rock there, please check it out". The guy who talked about the thing was a recognized expert in his own domain, while the CEO was on a kind of "thumbs-up tour". The manoeuver had been briefly discussed with the expert's hierarchical chain and pitched as an opportunity for action rather than "those guys don't do their jobs".
A small shitstorm followed in middle management, at the end the problem was quickly solved, deemed "not that important in the end", and since no one was at fault and no one innocent, everyone quickly went quiet again.
This reminds me of a particular project that I was tangentially involved in. It had a large capital expenditure at the start of it. After working there for a few years, I eventually realized the project was scheduled for next quarter. Literally. As in, no one ever wanted to take the budget hit this quarter so it was always just included in next quarter's budget.
People parrot "leadership" as if saying that people need to do better makes it happen. More constructive suggestion is to make sure these sort of things get good post-mortems, even better if publicized and made case studies in mba curriculums. Thats would have at least a chance of people learning how to be better leaders.
Employees need incentives to prevent problems, even if they aren't responsible for them.
In this case, the smart thing for every person who could have done something was to take no action, as they would get no credit for preventing a problem but would take a budget or resource hit for doing so.
I am a fire fighter, not a fire marshall. Fire fighters are heros. The fire marshall is a pest.
The author goes on to say centralized IT procurement. I.e. IT should have been the one supply devices, in which case they'd have replaced all the relevant devices rather than it being the responsibility of every dept.
Set aside a percentage of the cost each year in a pot so that when you need to buy you can just use those savings instead of draining your entire budget.
Can someone who knows more than me tell me whether "negotiate with compliance to disable those devices for a single day two months in advance so the ensuing shitstorm only costs $10 million" is a working solution?
I know our company lets you BYOD if you don't want to deal with also carrying around the company supplied devices. They have the opposite problem, as the company lets you pick between the top end iPhone and Samsung with a 2 year replacement window, so people are tempted to use their company supplied device as their personal device also.
I once worked at a company that paid "device rental" money to employees that used their own devices. It needed MDM, but you could rescind it at any time. Some co-workers would finance laptops and use the "rental" money to pay the installments. I did the math and it wasn't worth it for me, though.
The thread talks about getting extended manufacturer support, which suggests they are all the same device, which suggests this was company provided devices.
How were the users informed? Did they even understand what's going on? If I receive an email saying my device is out of compliance, I'd ask, out of compliance with what? How do I check? How do I get in compliance?
The way this is communicated to the users and what actions they had available to them makes all the difference here.
I would also have thought there should be alerts for devices going out of compliance soon. I'd set that for months back to account for lead times and deal with it as it comes. CC finance / procurement on the alerts if necessary.
Skimming the thread it appears to be middle management being informed, not users. The company devolved IT purchasing out to individual non-IT departments. Many of the purchased devices were past end of support life. Legal and Compliance set a hard cutoff when they could not be connected anymore and would not budge. This was known at least 12 months ago as the company bought extended support for some of the devices. IT told the department managers these devices needed updating/replacing over hundreds of emails and dozens of calls and meetings. Department managers took no action. Somehow the CTO was unaware.
"For anyone wondering why we don’t just lift the compliance restrictions, we don’t specify it. Their Compliance department does, and as it’s a large company and the affected users are less than 25% of overall workforce… no exception will be made. One side of the org is going b-a-n-a-n-a-s and the other is taking a very parental “well you should have thought about that” tone.
You kinda have to admire their commitment to the cause."
It's a picture of several org charts, each within a balloon by themselfes but connect together by a line. However, they are all pointing guns at each other.
No, they are the ones responsible for enforcing them. They are not the ones responsible for deciding what they look like and if exceptions are granted.
The cost should be billed to the department with the users that were affected. The laptops are assigned to those employees, so normally any kind of compliance/spend should be associated with them as well. It doesn't matter if those costs are mandated by compliance, it's up to each department to keep up.
FYI for those non-corporate readers, if there's an actual compliance department that means that the cost of non-compliance is really, really high. That either means financial or government/DoD.
it might also be healthcare, or nuclear/aviation, no?
> The cost should be billed to the department with the users that were affected.
It doesn't really matter. These are arbitrary slicings and dicings of one big monolithic blob anyway, internally connected by interdependent causal links.
It's a "simple" management decision how compliant the whole operation gets to be, also weighing the cost of compliance versus consequences of non-compliance (from simple things like "have to massage the scope during the next audit" to "might impact a potential lawsuit" to regulator fines us to personal criminal responsibility for the CEO), similarly it's a management decision how much spending each org/department can do on getting compliant without requesting budget for it, etc, etc.
In this case the priorities are clear and it just happens that there is this big discontinuity (crisis!) that will probably look like a bad overall trade off, something that should have been prevented.
Will this even result in some change to the whole decision-making hierarchy? Unlikely. If this hierarchy was able to completely impervious to all the input from below (emails, calls, meetings, 1 year grace period, etc) then it likely does not matter that these devices will get wiped.
Given that the yearly extension was supposedly long enough that there should have been more than one budgeting cycle where the department could request money to get the budget increase...
All the management there fucked up. Possibly with the exception of Compliance/Legal and IT.
I actually upvoted the comment because it's true (and it's positive now anyway), but it was probably negative because people complaining about the platform on which a given link is posted is pretty boring, and happens pretty regularly. Surprisingly regularly especially when you consider that approximately 0% of regular web users have JS disabled so there's not exactly a strong incentive to build for that crowd.
>Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.
Nah. Its accessibility at this point and its time we start talking about it, yes, even here. The internet is being ruined by js and screenshotted or otherwise uncopyable, inaccessible text.
Not necessarily. Without JS some sites would be horrible to use. I don't even know if it is possible to submit a form without reloading the page. It probably is by now but a lot of basic functionality either needs JS, very complex server-side rendering or url resource structure.
And server side rendering can be as ugly, it is just less visible. We can be really lucky that there is a universal language on the client side for web browsers. Even if it is often abused, it is very much worth it.
Maybe not the most elegant language, but probably one of the fasted to develop with.
Do you genuinely believe this is still an open question?
I won't dispute the argument that maybe it was a mistake, but to me it seems indisputable the ship has sailed.
I might buy the argument that any "mandatory" websites - government, library, academic - should be operational without Javascript.
But in the casual or entertainment domain, noone is obligated to provide their users an operational Javascript-free website. If you can't read something without Javascript, that's a you problem.
Most websites are far better with JS disabled. They're responsive, they have fewer ads, they don't assault me with autoplaying video/audio, they don't track me (as much), they don't make my laptop fan spin up and waste my battery. Just absolutely better in every way.
Some websites don't function without JS. I either enable it on a case-by-case basis, or avoid those websites.
It's rumoured someone could patch it and run their own. Haven't tried for myself. Seems like a lot of work and JavaScript is pretty convenient as compared to full page reloads or frames.
Lmao, the never amount of times a page felt faster or more responsive than a page refresh. Oh joy, i clicked and nothing is happening and i get no progress notification. Really great.
The entire premise is absurd. What application spends more time on SSR than data fetch, or other? Exactly freaking zero that ive ever operated.
Try https://dro.pm and tell me if I should make every click a full page reload. That bigcorps fuck it up doesn't mean it's the method itself that's harmful
Fwiw, I'd like this site to have fallbacks for non-JS users, but it's a heck of a lot of work to make everything twice and nobody asked yet or contributed a patch for even basic functionality
There is at least one open issue on the Mastodon github project requesting basic functionality without requiring Javascript. Which notes that this is clearly already possible:
"Show post content at standard post URLs when JS is disabled instead of just 'enable JavaScript' message, since this is already done for /embed URLs #23153"
I'm glad they are too. Let's take another example: what if the client was a healthcare provider and instead of us merely chuckling at the inconvenience of losses we witnessed deaths from management's incompetence. Would you still want the event to stay confidential if someone you knew died? I'm glad someone is discreetly share details of the situation to signal to the world "Hey if you fuck up compliance people can die, please don't do it like this" instead of keeping it confidential.
>I'm sure it's not hard to identify the customer either.
OP seems rather sure of the opposite:
>But I felt I needed to address one particular concern that has been repeatedly raised. That of the identity of the company in question.
>I’m a professional, and I’ve been doing this a long ol’ time. There is no way I’m going to risk the identity of the company, or my reputation, or the potential legal consequences for some interaction on social media.
>So to clarify, enough details of the incident and those involved have been changed to protect their identity and everyone else involved. I am confident that you could work at the company involved and not even be aware this happened, even after reading this Partly due to scale and partly due to managerial secrecy.
I found the writer to be downright empathetic. People make these decisions on purpose to cause their employees this much pain and you're worried about defending their feelings from a writer who is actually deploying empathy? I am so confused by this position.
If the writing was nasty and exposing specifics, sure, but it very much is not.
I'm not defeding the "feelings" of the company. I'm pointing out unprofessional behavior encouraged by a culture of oversharing.
Journalists and other outside observers can and should write about corporate incompetence wherever they find it. When you're in a paid position of trust, though, talking about your client's failings is tacky.
I agree that the writing isn't nasty. The specificity is the key. I guess to some people this came across as sufficiently anonymized. To me, it seems like anybody working at this place knows that the auther is talking about their company, which is a problem in and of itself. But it also means we're just one equally-indiscreet reply away from knowing exactly who this is (something like "yeah, I work here and..."). Though I really don't think that even that much additional info is necessary to deanonymize this. Just a hunch; obviously you disagree.
I can see how the ambiguity makes it seem like I'm saying that I think the people made this decision to directly cause pain. That's not what I meant and no I don't believe they made the decision trying to hurt people.
What I do think happened is that they saw how much pain it would cause people, then they looked at some fucking dollar amount and some spreadsheet and said "that's fine". I think that's fucked up.
The author's account is about as anonymous as Lemony Snicket is pseudonymous. Technically, but not practically.
Don't get me wrong, I don't dislike this person. It's not a "boycott". I'd just prefer not to be their customer, because this story goes a bit over where I'd draw the line of oversharing. I don't imagine that it will every actually come up, and I hope it never does.
Because we all learn from it can see what went wrong and what didn't and overall become better practioners, whereas in the old days everyone would be making the same mistakes and no one would learn from them.
Also he made pretty clear that he was anonymzing it to the point it would be incredibly difficult to tell.
I will admit that it caught my eye too. No names were mentioned, but how difficult would it be to determine who the client was/is? Then again, most companies these days have a 'social media' clause in their contracts so that you don't do around saying things you shouldn't about your work.
and it sounds like in this case, anything that was out of compliance (in any regard) was acted on by wiping the device and deregistering it on the deadline day -- read this as 1700 laptops or desktops getting wiped in one day.
I've since left my gig with MS systems, but I seem to recall seeing some sort of InTune client on my laptop. Is my memory failing me or is the client just weirdly named?
intune is the cloud replacement for Configuration Manager. It's been renamed a bunch of times over the years. I'm pretty sure they call it Endpoint Manager right now.
I concluded one of the last year deals in 8 months total from the first Mail I sent. Fortunately, 1 day before deadline (31st Dec) there were 4 different departments heads (each at least 2 level above my rank but still below C-level) involved with extended working hours on 30th December…… Ha ha.
So when the next renewal comes up, I am gonna kick-start the procedure 12 months in advance. :D
I like his avatar image. I've just finished restoring and pumping some steroids on an Amiga 500. It's still open next to me and it's nice to have that logo pop up on an unrelated context.
This sounds a lot like federal government contractor / FISMA compliance. I was in a similar situation with VPN remote access device non-compliance, but we ended up ignoring the compliance requirements since Important People were using the VPN.
That happened when XP was decommissioned. The project's due date was set by a congressional order. The calendar ticks over whether you are ready for it or not!
It's like a dog owner whose dog has been shitting on the sidewalk for so long that the owner eventually can't walk on the sidewalk without stepping in their own dog's shit.
349 comments
[ 3.1 ms ] story [ 268 ms ] thread"Ow! Why did this happen?"
well don’t do $that then
Me: If we do A, B will happen
Client: Do A anyway, we'll deal with B later.
(time passes)
Client: OH MY GOD B HAS HAPPENED
1: https://en.wikipedia.org/wiki/Carnac_the_Magnificent
2. We did it.
3. It was done.
I see that life as enterprise service desk hasn’t changed much. “Nobody tells me nuthin!”
Shout out to the ever under appreciated service desk folks out there.
"Why didn't you tell us you hired something new?"
"They work here for 2 weeks now"
"Tauch' deinen Füller nie in Firmentinte."
That's a very important advice!
Poetic
Chinese lesson two: really don't call them "not a thing".
¹: Also it's still my favourite, uh, thing that the word for a "thing" is, literally, an "east-west".
A "thing" or "stuff" in Chinese is 东西 (dōngxi). That's literally "east-west" if you pick the individual characters apart. That's what the footnote refers to.
Calling somebody 不是个东西 (bùshì ge dōngxī) means something along the lines of being good for nothing, i.e. an insult. Translating it literally, it would be calling somebody "not a thing".
Saying someone is irrelevant is likely an insult, also in English.
Not the corp in question but here is another one: https://www.thewealthmosaic.com/vendors/geissbuhler-weber-pa...
You may have heard of some of them, ServiceNow is one, SAP is another, some small companies like that.
It turns out there is not a technological solution to a management problem.
I can imagine the brute-force approach:
Log every intra-company communication, and if some communication was meant to go to department X, Y, Z, and it only went to X, Y, then a flag would be immediately raised to department Z's attention and whoever sent it.
Of course the personnel in department Z might review it but ignore it anyways, but at least now there's a paper trail of who's at fault.
An Exchange system already gets you 80% of the way there if you force all on-the-record communications via email.
That's the problem right there. One of my clients had a large IT organization of over 1000+ employees, with strict change control rules, and procedures that were tracked in SN. Every time there was a change management meeting everyone who could possibly be effected would get an email from Service Now notifying them of the upcoming changes.
Pretty much engineer ignored those emails, because there was so much going on in the org you'd get dozens of emails in a week and most of them you'd only be tangentially effected by, meanwhile you had your work to do.
So the problem isn't getting the notifications out it's getting people to pay attention to them.
Individual preferences do vary, one ignores 90%, another 95%, another 100%. And the one who's ignoring 100% of them will likely eventually make a mistake that otherwise wouldn't have happened.
But it will be fairly straightforward to resolve, after all there's an extensive paper trail as the chain of custody seems clear. Assuming the "change management meeting" emails were the approved means of communication.
IMO, one of the lessons that came out of Chernobyl is that it absolutely is a problem for the organization. Exposing people to too many "alarms" that are constantly going off will cause people to start ignoring them.
Part of good design is figuring out which things are truly important, and how to communicate that to the people who are supposed to be paying attention.
The emails mentioned by the parent don't sound like alarms. Because an alarm is usually for 'drop everything and focus on this' situations.
The equivalent in email terms would be a receiving an email with a subject in ALL CAPS bolded and underlined.
Or in general intra-company communication terms, a phone call from your boss without any pleasantries and a serious voice.
If someone makes the wrong decisions because they start ignoring signals then don't promote them or give them important coordinating responsibilities. Those who are capable of filtering out a larger fraction of noise do exist.
Of course there will always be folks whose preference is to read near 0% of their emails, but that doesn't imply organizations must be designed around them.
This is simply wishful thinking. Outliers certainly exist, but the idea that there are sufficient number of them that you can just ignore human nature is a path to disaster. You'd have to somehow accurately measure not just who is opening these noisey e-mails, but what they are retaining from them, and measure it over a large period of time, knowing that the vast majority or going to fail. It's far cheaper and more reliable to fix your noisey system than to try to outwit human nature.
Can you describe this 'cheaper and more reliable fix'?
It's really weird that you think you can't decide who is responsible for dealing with a type of notification ahead of time without "endless politicking and horse-trading", but think that blasting everyone with every notification, then attempting to sort out responsibility after something has gone wrong will somehow not cause "endless politicking".
Again, this is something many businesses do already. They don't blast the whole company when a bank account balance is low, when a server's disk is full, etc, notifications are targeted to appropriate groups. The specifics about who gets what is going to vary based on your organization. I can't give you hard and fast rules that will work for every organization, but that doesn't mean it's somehow impossible or not worth doing.
It's hard to tell if your joking.
The reason for implementing restrictive organizational procedures in the first place IS because nobody behaves like a 'rational human being' in such an environment for any significant duration.
When __everything__ is highly important and #urgent#, nothing is important and urgent.
In a sufficiently large and complex organization there will always be some percentage, neither 0% nor 100%, of actually 'highly important and #urgent#' hidden among the pretenders.
The point of any such tracking system is to discover, via positive or negative selection, what is actual instead of what various flawed humans pretend.
That... doesn't sound like a great approach to me.
If no one is keeping track of who is doing well and who to replace, then fixing that first is more important.
:-)
That (probably) means that the system for dealing with planning maintenances (well, usually, "approving them") needs to have a sufficiently good understanding of what humans care about what changes.
At a previous job, the planned change tracking system was REALLY good at tracking what specific compute facility was going to be impacted by any specific change taking place in that facility. And had a really good way of allowing you to filter for "only places I have stuff running" (and I think, even some breakdown of general change types as well).
It was, however, not easy to get notification of "there will be maintenance on submarine cable C, taking it off-line for 4 hours" or "there will be maintenance at cable station CS, taking cables C1, C2, and C3 down for 3h". And as one of the things "we" (the team i worked in then) was doing was world-wide low latency replication of data, we did actually care that cable C was going to be down. But, the only way we could find out was "read all upcoming changes" and stick them in the team calendar.
Was it good? Eh, it worked. Was it the best process I've seen? Probably ,yes.
So what? How would that change anything? Service Desk would send strongly worded emails to... somebody? The Organizational Overmind was already getting warning messages. Daily. There's already a trail of emails and status reports. It wasn't a knowledge issue. It was a "there's no effective direction" problem.
Which is why I consider it a 'partial-solution' as technology cannot do everything by itself yet.
So you'd select "System Z Update Access", write a one liner of "I need update system Z to do my job supporting The Alpha Process", your manager would approve it (providing they agreed that your role did involve supporting the Alpha Process) and then someone from the System Z team would also approve it (providing they agreed that you needed that access to support that process), and then a few minutes later you'd be automatically added to the right role.
Traceable, secure, and fairly painless. Required a lot of setup for all of the roles and automation though, I believe.
Human problems require human solutions. Tooling problems require tooling solutions.
Been my favorite saying at my small business for years now when people propose technological solutions to HR issues. That isn't gonna cut it.
Or did just some clueless start-up beg to get sued into oblivion?
https://en.wikipedia.org/wiki/SAP
I am making some assumptions when discussing this topic, namely:
1. The CEO and executive leadership are actually competent, have at least average middle aged adult levels of patience, and don't have their daggers out for each other.
2. There exist more than one method of communication between each layer in the hierarchy.
3. The organization is engaged in normal business, not in the middle of an M&A deal, outside investigation, or bankruptcy proceedings.
In reality, most large organizations do have at least a semi-competent leadership team most of the time. Unlike what is commonly supposed by junior level staff expressing their frustrations.
Everyone on HN can read the comment chain and see where you joined in, and who said what... so the claims are difficult to take seriously.
Even if you focused on my alleged lack of qualifications to comment it would have not been as self-discrediting.
Though I don't get riled up by oblique insults, that might not be the case for others, especially for a topic that might be linked to negative emotions in the reader-base.
My sentence on 'junior level staff' wasn't directed towards you, it was a reflection on why it can be difficult to see the grand scheme of things when the environment might restrict that.
Nice Hot Fuzz reference.
But at the end of the day, they did the job they were paid to do and were clear about the looming impact. It's not their job to also wipe their clients' metaphorical bottoms when they were ignored.
> For anyone wondering why we don’t just lift the compliance restrictions, we don’t specify it. Their Compliance department does
after a year-long grace period:
> The machines came to end of life about 12 months ago, and the company being a multi-billion dollar operation managed to eke out another year of manufacturer support. Mostly symbolic as they’re not exactly going to release custom firmware for a handful of devices. They then put a set-in-stone tombstone date on support. 12pm today.
This was imposed internally by the company's compliance and legal departments, TFAA is the executioner but the execution would be contractually mandated:
> They require, and have specified, a zero-tolerance for device non-compliance.
This means an unapproved batched and drawn-out phaseout would be a breach of contract.
You could brownout or kill a few a few days before the real issue, though, potentially.
What one could try is to call the CEO directly. Or maybe try the legal backdoor: contact the general counsel, tell them that the contract says such-and-such, that you think the contract is well written and you intend to do what it says, but that the organization should be aware that it may cause a problem. If legal doesn’t know how to get the CEO’s attention, then something is very wrong.
> 4 meetings, 124 emails, and two phone calls a day for the last 14 days have warned them of this.
There's only so much you can do.
> If legal doesn’t know how to get the CEO’s attention, then something is very wrong.
From the thread legal (and / or compliance) is the setter of the issue, and was well aware that it would cause issues (for a minority), but they were not in charge of resolution. And from downthread posts, they likely extensively documented their warnings:
> oh I’m absolutely backing the horse with the 3 miles of email threads proving this
And methinks legal and compliance had very much planned for the issue coming to a head, because they were getting fed up with being blow off, and having to shoulder the legal or regulatory risk.
A year to make fixes and nothing was done?
Legal is like, "We warned you every way we possibly could have."
"two phone calls a day for 14 days" is far from insufficient notice, to say nothing of the rest.
There's value in face to face, but not when it's a wide reaching announcement like this.
You can always start early.
People here blaming management but discussing how to do things at the last day. You don't
You give them a new device before locking out their old one
Do it for 17 weeks before the out of support deadline? The users start screaming, we still have 17 weeks left before the deadline, how dare you disable us early! And they get enabled again.
Do it for 17 weeks after the deadline? The compliance people start screaming, you have 1600 devices out of compliance, we need them shut down now!
https://github.blog/changelog/2021-04-19-sunsetting-api-auth...
I don't know if that concept would work in this case, compliance is it's own beast, but I love that idea in general.
A week later, the customer notified us that they were still seeing some traffic on the old URL, but all they could give us was the IP address it was coming from. Unfortunately this IP address belonged to a server that hosts a lot of our smaller applications, so it didn't really help locate the offender. So I just added a firewall rule to block access to the IP address of the customer's old server, and sure enough I heard the scream 15 minutes later. Removed the rule to get that application back up and running, got it updated to the new URL, and all was good.
> Alright, things are moving faster now, so I might condense-toot to avoid pollution.
But I guess the chronically online need something to complain about.
So in the end, this is just one piss poor managed division abusing another piss poor managed division. Who gets the heat? Probably the lowest level people.
Why "wipe" them? That seems unnecessarily punative.
You can see the "don't give a shit I work with a predatory organization" oozing from everywhere.
The security guy is trying to claim that they've sent out many many notices, but really this is just an excuse to abuse other people in a machiavellian abusive organization.
"Service Desk is now aware that everyone else except them was aware, and now IT is absolutely incandescent." Whoops, missed an email and a meeting in there bucko.
And it's the SECOND company where this was "implemented" or "specced"? This sounds like someone checked a box or compliance or ass-covering upper management slid this under the table, but all the people it ACTUALLY AFFECTS didn't get any input or opinion on the matter. And when push came to shove over funding it that person had probably moved on to bigger and better things.
So since you get to do it, you seem to be gleefully doing it. Great job.
There was something in the thread about the devices coming out of support by manufacturer, which was already extended by a year.
There are better ways to handle this, when sending the messages out. If the deadline for compliance was 31 Jan, then when sending comms out say the deadline is 31 Oct, and machines would be wiped after that. Then start wiping them, 10% of machines on 1st November, another 10% on 8th November, etc.
There is not necessarily any Schadenfreude in watching and reporting it. No one really needs to be taking pleasure, it's just hard to not pay attention to a train crash occuring in slow motion.
It's very natural to want to talk about something this stupid/bad. Rubber necking is extremely human.
Now everyone in the affected chain gets a black mark on their "permanent records" and gets exposed at a time when likely layoffs are coming.
What I don't hear is "why can't they upgrade, and how can we help them upgrade", it's WE TOLD YOU, NOW YOU SUFFER.
Here's the kicker: it's 1600 devices. Ok, so they've been told for 13 months to do this. Well, let's do some math. That's 260 working days. Oh look, about 1600 working hours. So if you guys had simply upgraded a device an hour over the last year, this wouldn't have been the problem. Yes, that's not fair, but neither is what the person doing.
Security is the military arm of compliance. Finger pointing at compliance is a bit mendacious. Saying LOL it's not my fault, it was compliance. NOW WATCH ME DROP THE HAMMER BOOM.
I mean, I guess the guy is saying LOL I'm outsourced and not even in the company HAHAHA. Still, eff this guy for taking a bit of glee in this.
That is not really how I read it; the devices got an year extension because people had already failed to refresh them within the standard cycle. From the sounds of it these are typical workstations etc, their support cycles are very predictable and if you bought some crappy ones without predictable lifecycle that is on you too. That extension should have been wakeup call, the process had already failed then.
"If you had simply spent nearly three quarters of a million dollars of your own money, done 40 weeks of volunteer overtime on top of your normal job, without any purchase approval, without the authority to do that, and no guarantee of seeing that money back, this wouldn't have been a problem, so fuck you"
is a terrible take all around.
> "What I don't hear is "why can't they upgrade, and how can we help them upgrade""
We know why they can't upgrade, because the departments responsible for purchasing the upgrades won't agree to spend the money. This isn't something which can be helped by more technical input.
> "Finger pointing at compliance is a bit mendacious"
"mendacious: not telling the truth; lying." - nope, wrong. Legal and Compliance say it must be done and you must do it, and have the authority to do that. Pointing fingers at them is honest and appropriate, that is where the instruction is coming from (legal) and the reason why the instruction exists (compliance with internal or external regulations).
Clearly the compliance team (or whoever is implementing it) has failed in its communication
It's more that security teams tend to have uncooperative, aggressive, authoritatian, and punative dispositions. I think ye old security industry had its roots in three letter government agencies which are used to conformance, policy hammers, and enemies of the state.
But when you add that to an organization already rife with infighting, dissatisfaction, and frustration, it will just lead to more resentment and your employees become your enemies.
The biggest security threats these days aren't leet hackers exploiting 0days, or even the county password inspector conning his way in. It's overworked angry pissed off employees leaving the door open. It's like Princess Leia said: the tighter you squeeze, the more people you lose.
If compliance and legal say to wipe the laptops, and everyone with a budget was aware of it for a year, it's not reasonable to put the disaster on whoever was in charge of implementing policy.
This is not a Petrov situation, you're not saving the world by going out of your way to be the person that will defy Compliance today, just because the policy is really dumb.
The people locked out would be shortsighted to blame the random security guy. They joined a big company with a very strict compliance machine, not a startup where you move fast and break things, then ask legal for forgiveness.
Big organizations are dysfunctional, news at 11. Don't blame a random IC for executing policy after considerable warnings. If communication is so thoroughly broken internally, and no one wants to take responsability for necessary spending, it's not the job of some random security guy to fix that internal dysfunction.
Potentially leaving company confidential material on non-compliant devices is not something Compliance department would want to allow
Don't fuck with compliance I guess?
You're proposing to remove the devices from the cloud management service, probably including admin lockouts and anti-malware software, but leaving all the data on them?
* Seems I've misunderstood; I was corrected downthread.
20% can probably handle some legacy stuff that requires an old computer. Or a migration.
In addition some can be management, so they wont be making decisions for some time.
But nobody wants all that additional spend, so close to year end. Departments bickering over who’s responsibility it was, who’s budget it came out of, and so on. So everyone dug their heels in, and we continued to shout “iceberg!” from the sidelines."
I've seen this play out multiple times over the years. What's the solution?
The correct strategy here is to go on the offensive. Make friends with your manager’s manager’s manager. Just go full on social bribery mode. Invite them for Christmas dinner. Even if they decline, it will make them remember you, and next time, ask them to a barbecue or a picnic instead, and they’ll say yes. If you can’t throw that high, shoot for your manager’s manager. Once you’re in, get the dagger in your manager’s back, but only once you’ve made them a pariah, take their role, and repeat until you retire wealthy. Remember to take every opportunity to accrue political capital.
Eventually you will be in a position to fire people who make poor decisions, but you won’t, because the salary is good, and retirement is only a few years off.
You literally cannot prevent this behaviour in any human-operated organisation of any scale beyond a fistful of people - you can only co-opt it.
Even in my program of just north of 100 people broken infrastructure follows that same pattern. Something moderately breaks, the devs complain, they are sympathetically told to make do. Rinse and repeat until the devs realize their complaints never get addressed and stop complaining past a quick email. Then something REALLY breaks with no workaround, devs mention it's been heading this way for a while, and management, all aghast, exclaims "well why didn't you say so earlier if this was such a problem?". It's to the point where we (the devs) have started keeping locally archived email records just for the "told you so". Which of course makes us no friends because we point the blame where it belongs, so we're officially covered but our complaints get listened to even less. And the infrastructure is fixed just enough to limp along until the next catastrophic explosion.
You need someone who gives a shit with the power to crack the whip. In short, you need to give someone enough power that they can potentially abuse it, something modern business is allergic to.
There is very little reason for an employee to care about preventing a failure they will not be directly held accountable for. I don't care if we lose clients. I don't care if we get hacked.
My life is not impacted one way or another by whether the divisions I work in succeed or fail, unless they utterly fail.
So if this Intune thing landed on my desk, I would do nothing about it. Give me some incentive to care and I would.
You are trying to do a good job which might lead to a promotion. You see the rules and operations of the organisation as something to work within.
The people who ignore you are trying to get promoted. They see the rules and operations of the organisation as a irritating backdrop to their personal goals. The job could be anything. Ascent is all that matters.
They will get promoted. You will retire one day, your nerves fried.
The incentives are all wrong, and playing the social metagame rather than playing the game by the rules always results in an advantage, and thus this behaviour is inherently embedded in any human structure.
The solution isn’t “better management” - it’s in fundamental societal change, which ain’t coming any time soon.
Put an impartial AI in charge and it will make the right decisions — and people will riot over the injustice. Ultimately, you’d have to go machine the whole way and the humans can all go bake bread and have wars over that or something.
Within a minute the deptartment head forwarded my request to the manager who was ignoring me and told him to take care of it. It was promptly. I got the email chain when it was done, there was some comment from the manager who ignored me to the facilites person who ignored me, that it "should never have been allowed to escalate.."
I don't miss big companies sometimes.
Don't want the issue to escalate? Then maybe do the job you're paid to do
So much for escalation. I no longer work there (I left; I wasn't fired).
Cringe.
Apparently most managers can’t even manage properly. Most of the time this “ship” word is tantamount to stolen valor in the workplace.
In dysfunctional organizations, the management structure exists to rein in true leaders. In a healthy organization, leaders are recognized and supported by management, whether they're in management positions themselves or not.
Like, with these 2,000 people locked out of devices? Assuming they're contractors? It probably won't affect the bottom line at all. A minor nuisance, monetarily. Easy to route around; just work people harder until the problem is solved, or take a minor write-off and play Tetris with the books to make it look like you actually made money rather than lost it.
But if you want to have a company where people are happy to work, you absolutely must have non-asshole leadership.
If they don't do that, the guy just does whatever he want, often leading horrible outcomes. It's only working because there are enough people who are passionate enough for the project and want it to succeed. Such people would leave if that was just a software company.
Also, no company I've ever worked for makes their budget after asking people what they need. Instead, the Finance team just copy-pastes last year's budget, usually with a little haircut off the top for "efficiency." So let's say you run a team that depends on a handful of servers. You bought the current ones five years ago, and it cost $500k. Now it's time to replace them. But since you didn't buy $500k in servers last year, it's not in the budget for this year, so now your entirely reasonable request is being scrutinized as "unplanned spend". Several times I've seen teams try to build a plan for the upcoming year, only to find that there's nobody to give it to, because the "budgeting process" is done in about a week's time, usually two or three weeks after the fiscal year starts and with commensurate effort.
Seen some years ago. A mail was then sent from CEO to CxO along the lines "it seems there's something hiding under a rock there, please check it out". The guy who talked about the thing was a recognized expert in his own domain, while the CEO was on a kind of "thumbs-up tour". The manoeuver had been briefly discussed with the expert's hierarchical chain and pitched as an opportunity for action rather than "those guys don't do their jobs".
A small shitstorm followed in middle management, at the end the problem was quickly solved, deemed "not that important in the end", and since no one was at fault and no one innocent, everyone quickly went quiet again.
In this case, the smart thing for every person who could have done something was to take no action, as they would get no credit for preventing a problem but would take a budget or resource hit for doing so.
I am a fire fighter, not a fire marshall. Fire fighters are heros. The fire marshall is a pest.
Archive those CYA emails and enjoy the popcorn as you watch management sink on the Titanic.
The way this is communicated to the users and what actions they had available to them makes all the difference here.
You kinda have to admire their commitment to the cause."
I want to know what their org chart looks like.
https://images.app.goo.gl/9bRVF4EeZW4SJbqC9
https://goomics.net/62/
It was linking to the Microsoft one, but I think the Oracle one is more relevant.
What means "we don't specify it" in this sentence?
FYI for those non-corporate readers, if there's an actual compliance department that means that the cost of non-compliance is really, really high. That either means financial or government/DoD.
> The cost should be billed to the department with the users that were affected.
It doesn't really matter. These are arbitrary slicings and dicings of one big monolithic blob anyway, internally connected by interdependent causal links.
It's a "simple" management decision how compliant the whole operation gets to be, also weighing the cost of compliance versus consequences of non-compliance (from simple things like "have to massage the scope during the next audit" to "might impact a potential lawsuit" to regulator fines us to personal criminal responsibility for the CEO), similarly it's a management decision how much spending each org/department can do on getting compliant without requesting budget for it, etc, etc.
In this case the priorities are clear and it just happens that there is this big discontinuity (crisis!) that will probably look like a bad overall trade off, something that should have been prevented.
Will this even result in some change to the whole decision-making hierarchy? Unlikely. If this hierarchy was able to completely impervious to all the input from below (emails, calls, meetings, 1 year grace period, etc) then it likely does not matter that these devices will get wiped.
All the management there fucked up. Possibly with the exception of Compliance/Legal and IT.
I don't want to "use the Mastodon web application", whatever that means. I just want to read a web page like a normal person.
Sigh Does anyone have the article text handy?
https://archive.is/fxqui
A hammer can be an important part of building breathtaking architecture. It can also be used to gouge somebody's eyes out.
Just like JavaScript.
EDIT: That "web app" was very nice to use.
From https://news.ycombinator.com/newsguidelines.html
And server side rendering can be as ugly, it is just less visible. We can be really lucky that there is a universal language on the client side for web browsers. Even if it is often abused, it is very much worth it.
Maybe not the most elegant language, but probably one of the fasted to develop with.
Do you genuinely believe this is still an open question?
I won't dispute the argument that maybe it was a mistake, but to me it seems indisputable the ship has sailed.
I might buy the argument that any "mandatory" websites - government, library, academic - should be operational without Javascript.
But in the casual or entertainment domain, noone is obligated to provide their users an operational Javascript-free website. If you can't read something without Javascript, that's a you problem.
Some websites don't function without JS. I either enable it on a case-by-case basis, or avoid those websites.
the 2000s and early 2010s era internet ("Web 2.0") was richer with Javascript WITHOUT all the gargantuan trackers and auto-playing videos.
It's banned in the rules on the site. No-one wants to hear about your off topic journey of self discovery. Maybe Reddit does?
The entire premise is absurd. What application spends more time on SSR than data fetch, or other? Exactly freaking zero that ive ever operated.
Fwiw, I'd like this site to have fallbacks for non-JS users, but it's a heck of a lot of work to make everything twice and nobody asked yet or contributed a patch for even basic functionality
"Show post content at standard post URLs when JS is disabled instead of just 'enable JavaScript' message, since this is already done for /embed URLs #23153"
<https://github.com/mastodon/mastodon/issues/23153>
Why are we live-tooting (ugh) our client's private disasters? The indiscretion is staggering.
This account casts its author in as bad a light as it does their client. I wouldn't want to work with either.
Indeed. I'm glad they are, because it is fascinating but very odd. I'm sure it's not hard to identify the customer either.
OP seems rather sure of the opposite:
>But I felt I needed to address one particular concern that has been repeatedly raised. That of the identity of the company in question.
>I’m a professional, and I’ve been doing this a long ol’ time. There is no way I’m going to risk the identity of the company, or my reputation, or the potential legal consequences for some interaction on social media.
>So to clarify, enough details of the incident and those involved have been changed to protect their identity and everyone else involved. I am confident that you could work at the company involved and not even be aware this happened, even after reading this Partly due to scale and partly due to managerial secrecy.
If the writing was nasty and exposing specifics, sure, but it very much is not.
Journalists and other outside observers can and should write about corporate incompetence wherever they find it. When you're in a paid position of trust, though, talking about your client's failings is tacky.
I agree that the writing isn't nasty. The specificity is the key. I guess to some people this came across as sufficiently anonymized. To me, it seems like anybody working at this place knows that the auther is talking about their company, which is a problem in and of itself. But it also means we're just one equally-indiscreet reply away from knowing exactly who this is (something like "yeah, I work here and..."). Though I really don't think that even that much additional info is necessary to deanonymize this. Just a hunch; obviously you disagree.
Do you actually truly believe this?
What I do think happened is that they saw how much pain it would cause people, then they looked at some fucking dollar amount and some spreadsheet and said "that's fine". I think that's fucked up.
Don't get me wrong, I don't dislike this person. It's not a "boycott". I'd just prefer not to be their customer, because this story goes a bit over where I'd draw the line of oversharing. I don't imagine that it will every actually come up, and I hope it never does.
Also he made pretty clear that he was anonymzing it to the point it would be incredibly difficult to tell.
I don't think the reporting being 'live' makes it much better or worse, though it probably wastes more of the readers' time.
IT admins can set a policy like "must have 6 number PIN for login", if it does not then it is out of compliance.
This can mean nothing at all but if the company wants to act on it, it can.
I concluded one of the last year deals in 8 months total from the first Mail I sent. Fortunately, 1 day before deadline (31st Dec) there were 4 different departments heads (each at least 2 level above my rank but still below C-level) involved with extended working hours on 30th December…… Ha ha.
So when the next renewal comes up, I am gonna kick-start the procedure 12 months in advance. :D
For my mental peace.
I'm not sure I could tell if this was truth or fiction.
The depiction is so close to IT / compliance-office revenge fantasies, so fiction seems plausible.
But they will absolutely blame the dog.