It's a balance. If 10 firefighters don't see high utilization, you don't want to increase the staff to 20, just in case. That's just a waste of money.
The rule of thumb is that you want utilization to be where there is an acceptable latency depending on some percentile of cases. For a firefighter, you'd probably look at p99 latency. For a hamburger joint, p50 on order time would be good enough.
Seems like the original article/book and your comment tie back to Little's Law from queuing theory which is used as justification in Kanban for minimizing WIP.
I doubt you will ever see even firefighters with low utilization. They can use extra time to do inspections to make sure there aren't any fires, or put on presentations at neighborhood association meetings teaching the public about fire hazards. If they aren't doing those things and there aren't any fires, they're just lazy.
Absolutely not. Not that those things aren't great, and hopefully already being done, but there is zero need to ask firefighters to be at 100% utilization at all times. There is value in having some one perform busy waiting in the event of an emergency. Did you know thats a training drill they perform? How fast they can be out and on their way from the moment of getting a call.
>you don't want to increase the staff to 20, just in case. That's just a waste of money.
I believe OP's point is that you do want the staff of 20, for the once-in-a-lifetime fire that requires 20 people. See also: the recent Texas power grid debacle, or the saturation of ERs due to COVID.
No, you don't. Resources are not infinite, and at some point the budget has to be taken from other services which are more useful than a once-in-a-lifetime event.
Or you do want the staff of 20, but then it must be a volunteer and/or on-call duty system. Otherwise it is not sustainable.
In my country, the shift/switch to professionalisation that started 30 years ago has gradually become a big problem. Despite the fact that they still represent only 20% of the firemen, they are killing the budgets, and they always want more (lots of strikes); apart from 'standard' raises, the most common thing they ask for, is that on-call hours should be paid full-rate, as active hours. Which, beside being extremely costly, is absurd when they are 'working' 24h shifts! It contains a few hours of training and, depending on location, a few hours of duty; the rest is on-call (at home or on premises depending on the type of station), the number of service calls is limited, and 1 in 4 shifts happen without a single call (even more for 12h shifts).
There are plenty of other problems which surround this professionalisation, but they are not directly related to this subject.
On-call at home should definitely pay less than active duty. That way, when a call comes in, the annoyance of being interrupted is offset by knowing you're getting paid more for responding.
If you get paid the same, responding to a call just becomes an annoying interruption ("I could be at home making the same money if this person I'm helping had just been more careful!"), leading to a worse experience for everybody.
I think we’re getting hung up on specific numbers.
The general point, I think, could be summarized as: you want some idle resources to handle surge capacity. Whether that’s 5 or 20 firefighters is of course the next question.
TL;DR: Author has discovered another author who has written about a topic that's been known since queuing theory and statistical modeling were explicitly applied to businesses some time last century which merely revealed what many people already understood intuitively (but probably couldn't quantify or articulate).
100% utilization is moronic, you need slack in your system or you will introduce the risk of a severe backlog of work that may never finish (or will finish because clients cancel the requests and go elsewhere).
Iff you consider your work a net-negative or just a means to an end. There's certainly a case to be made for optimizing for work that is an end in itself where more work may increases positive utility in some areas (joy, fulfillment, whatever) and possibly decreasing it in others (money, status, whatever).
I hope work isn't a net-netagive, but there are always things that must be done that you don't want to do. Sometimes you can hire someone else, but often you cannot.
I was probably too sloppy in my wording. I interpreted the OP to mean work may have negative utility for the individual person, but not in the aggregate. I don't know that the idea that work is essential individual sacrifice for some end goal is particularly healthy.
I understand where you're coming from, but I think it's built on assumptions that I'm saying don't necessarily need to be true. Namely, that cleaning your dishes is "distasteful" which tracks with the originally assertion that you do some jobs just to get them done to get to what you really want. I.e., they are just a means to an end.
My claim is this can be unhealthy because you're not really living in the present and constantly stuck in the mindset of "once this distasteful thing is done, then I can get to the stuff that makes me happy." Or to steal from The Good Place, "Humans only live for 80 years and they spend so much of it waiting for things to be over."
My idea is that "work", or maybe more precisely, a "job", is something that you only do because you have to, because you need something it gives in exchange, and otherwise won't do.
If you do something because you enjoy it, it's a "hobby". If you are paid for that, too, you are just lucky to have the best of both worlds :)
I agree that's a goal, but I think it's overly concerned with the product (job) rather than the process (how the job is done). I think focusing on the latter is more sustainable for enjoying something because the job itself will undoubtedly get stale at some point and can lead to the hedonic treadmill of constantly looking to the next thing to make you happy. I've known people who have had jobs that people would traditionally consider unenjoyable, but they had the mindsets to truly enjoy the work by focusing on the process, constantly bettering their performance, etc.
I did some refactoring work a few months ago, replacing the old way we did something with the new. I didn’t have to do it, I could have punted like the creator did. But I was used to the new thing and I wasn’t about to write new code that was already deprecated.
Nobody called me out on it but it wouldn’t have been the first time in my career.
But now I find out belatedly that we’re changing our auth system, and now that work is going to save me from having to drop everything to get it done on time.
If there's something you feel you can do, it'd be good, you absolutely should investigate that path. If it turns out it didn't work out, someone stomped on it or you get pulled elsewhere, it wasn't meant to be. You feeling that goal within your grasp, is anyways valuable as learning exercise.
No matter your efforts, there's a time for everything.
I was about to write that. This is very visible in operational teams.
Depending on what is going on, an operational team will spend 20 - 40% of their time firefighting or at tightening screws and oiling wheels - maintaining systems. Sometimes it's a good week and it's just 10%. Sometimes you launched a new product, and it's 60% because everything is failing.
As a conclusion from there, it's not a good idea to schedule more than 50% - 60% of deliverables with deadlines, because the right outage is going to toss those estimates really quickly.
That's in itself the definition of sufficient slack. If you don't have that, prod fails and no one is around to fix it. If you do, someone can usually start poking at it quickly.
Yup. I always liked to have my team (been in the infra/ops/sre/devops/sysadmin areas) focussed on latency rather than throughput by having slack.
Luckily we've usually been able to avoid Scrum etc, and work in a way closer to Kanban.
There have been times though (like right now sigh) where we're forced into a throughput oriented mode by commitments made elsewhere out of our control. It sucks, and we end up ignoring too much of the little stuff for long enough that they end up becoming fires you need to put out (is ops debt a thing?), your tooling and automation suffer, knowledge silos build up within the team, and the throughput will end up tanking anyway.
I like to use 50% allocation on "project" work as a nice rule of thumb for reactive ops oriented teams. Any higher can only be sustained for short periods without negative effects.
I am worried most about how our system pushes efficiency to the detriment of resiliency.
Businesses use just in time inventory to increase efficiency, but that means a small disruption anywhere in the supply chain can grind everything to a halt.
Also, instead of maintaining cash savings, businesses will rely on cheap credit. However, if enough businesses do that, any type of credit crunch results in widespread economic disaster.
Often business efficiency is a way to privatize the gains from efficiency, but but have the public bear the losses from lack of resilience.
Just like banks are required to keep a reserve (which is inefficient in some sense), I think some type of regulation or increasing the personal moral hazard for business leaders is required to make sure businesses don’t sacrifice resiliency for too much efficiency (and the profits of efficiency).
Agreed. Businesses optimize profit generation. Providing good products or services are merely the means to such an end. Increases in efficiency often means sacrificing some quality they view as superfluous but is likely valuable.
I've only been working professionally for 5 years and this is something I am only recently beginning to appreciate. Optimizing for efficiency makes you get a lot of work done but it reduces your ability to think creatively which can potentially impact the quality of ones work.
Your job is to make their wishful thinking the thundering success they deserve.
To counter the original point, I find removing obstacles and latency-inducing loops helpful, to start seeing what the work really should be. Gaining efficiency through simplifying is a good thing, and can be creative too. The goal is not efficiency though.
I think you're on to something important here - the word "efficiency" is used to describe optimization in two different mental regimes: one, "how to meet a given quality of work with the minimum of friction/wastage" vs two, "how to perform the maximal work within a fixed resource allocation."
They sound similar, which is probably why we use the word "efficiency" to describe improvements in both regimes, but the fundamental constraint is different: in the first case, it's the standard of work that must be achieved; in the second, it's the resources allocated to the work. I'd summarize the first "do enough with enough" and the second as "do more with less."
What you describe sounds like "doing enough with enough:" given the work to be done, how can we remove resource-draining obstacles, idle loops, etc. and identify "what the work really should be?" - is that a fair assessment or am I off the mark?
Say more about XOps? From where I sit, DevOps is "about" treating software product more like modern manufacturing products && tightening the feedback loops between development and operations... How does that translate to, say, Graphic design, Finance, X where X==$operational_unit?
This might even be rooted in human psychology. "The Master and His Emissary" and other books about neuroscience argue that we have brain functions specialized for focus and other brain functions specialized for context. Activating focus reduces contextual thinking.
Even shorter: there's no single absolute optimum; if you optimize for efficiency, you lose in other areas. But if you optimize in other areas, you lose in efficiency, of course. Everything in real life is a compromise.
Doesn't this just imply that the cost function is over simplistic rather than there is "no single absolute optimum"? E.g., maybe a more appropriate cost function factors in both resilience and efficiency.
It reminds me of working a scheduling problem that failed to factor in union concerns. It was a bad solution because it didn't factor in all the dimensions of the actual problem and only originally concerned itself with "management's" cost concerns, not the "union's" cost concerns.
Tbf, I never finished "Antifragile" as I felt like it just kept going over the same concept from different angles without introducing anything new after the first 50 or so pages.
Indeed, I meant that a simplistic approach which wants to have every dimension of utility turned up to 11 simultaneously does not work. You not just need to factor in more dimensions than efficiency, you have to trade gains in one dimension for losses in another. The scalar utility function you can reasonably optimize here is some weighted combination.
In retrospect, yes, this is absolutely true. One is much more effective with slack. The question is how to set one up for that. Ironically, slack (the app) is really counterproductive here. It takes some effort not to react immediately to every request. While this might look like the setup of Gloria, the secretary, often the sheer amount of requests kills any slack.
You get slack by not delaying on the things that can be done immediately (with no or minimal planning), and then planning the rest so that you tackle it in an appropriate fashion. If you don't plan it, you'll end up creating additional work on top of the desired work to fix the issues produced by skipping planning.
I work in public sector digitalisation and have for a decade, so this article sort of rings home with me. Especially now, having passed a year of thousands of office workers working from home and having seen a rise in efficiency and quality across all our sectors. I’m not saying working from home is an all-good sort of thing, we have also seen an increase in stress and depression related sickness, but in terms of getting shit done, things have been never been better.
Which is sort of ironic from my department, because this has also been a year where our process optimisers and MBAs have been almost completely unable of performing their usual efficiency and benefit realisation consulting in our different departments, as that’s a hands on sort of thing. Not that they’ve done nothing, they’ve been to really good work helping managers coordinate remote work and teaching both the CEO and Political layers how to use Microsoft teams efficiently.
Anyway, if we’ve increased efficiency and quality more in a year or not trying to, it sort of begs the question what good trying really does. You obviously can’t really conclude anything scientific on our anecdotal measurements as we’ve seen the major change of going remote on top of it, but it is something to think about.
Not that we will, we’re already trying to figure out how to go back to the way things were, as the majority of our managers still seem to think people work better if they spend 7 and a half hours in an open office 5 days a week.
It reminds me a little of a story I read about Alcoa, the Aluminium manufacturer.
They had been have issues with improving productivity. One of the issues was the workforce and unions were reluctant to accept any change.
New management came in and rather than pressing on productivity issues they decided to double down on safety. They hadn't been particularly bad but not great either.
Obviously staff and unions are never going to complain about making their workplace safer. What also happened was as soon as process were open to change to improve safety they were also open to change for productivity.
I wonder in cases like yours whether the pandemic just unstuck a lot of things because all the previous excuses floated away as soon as someone says "because covid".
Anecdotal but I know at least one megacorp that is doing layoffs of manager level positions after noticing that their services/industrial processes worked well while them being on furlough during the lockdowns.
We've replaced secretaries with software, and now we have people making $150k+ a year busy working on things that they should be paying someone $40k a year to handle.
It's easier to prove that one "saved" the company money by removing a support position than to evaluate the amount of money wasted yearly by engineers having to buy paperclips and figuring out how to expense them in the horrible expensing software.
yes and there is also often departmental budget arb going on here
Devs in engineering org now have to spend more time on self-service portals & chasing tickets because the manager of the infrastructure org laid off a bunch of sysadmins.
I once worked at a bank where even replacing a physical disk in a US datacenter involved a ticketing system which dispatched tickets to India.
The remote guys would then, presumably, raise some sort of internal ticket so the guy physically in US could you know.. replace a bad disk.
Turnaround on bad disk swaps went from hours to weeks. As the hardware aged, we started to have enough disk failures pile up on RAID arrays that data losses occurred.
I never thought open offices were about cost savings, I always thought it was about someone reading that we spend too much time isolated and need more interaction or some such BS.
Open offices, especially with flexible seating, are a huge cost saving on real-estate. You need much less floor space, and much less interior walls.
I think at my company, we are sticking with it despite arguments against, because it is just cheaper.
The advent of more people working from home is going to mkae this even more attractive probably. :(
Systems are better too though. When I was an intern all meeting scheduling was done on some convoluted mainframe system. Most of my co-workers had forgot their login to the system, they either grabbed a room that was empty and left if someone showed up, or they had the secretary schedule it (these were computer programmers Sun workstations on their desk, not computer haters who refused to learn). One day we rolled out a new system that was easy to use and suddenly all meetings got scheduled by whoever wanted to have one (then that got replaced by exchange/outlook which we could figure out but wasn't anywhere near as easy).
So some of the savings is good. It is faster for me to schedule a meeting in outlook (not the same company) than to find a secretary to schedule my meetings. However the secretary might be worth going back to just because they always knew important gossip that was worth knowing.
Your example definitely is something that should have been automated and not good use of an administrative assistant's time, but humans are good at navigating unclear processes and organizations.
For example, in one of my internships, it turns out that someone mistyped my address so my paychecks were sent to the wrong building; after a few weeks of that not getting resolved through HR, the administrative assistant took it upon herself to fix it and figured out whom to go yell at to get it resolved within a few days.
They're also good for things where having specialized knowledge of a process that's not done often can be done by someone who does it more often.
For example, when it comes to corporate travel, our company has a self service portal, and every time I need to book business travel I have waste an hour to figure out the right combination of flights and hotels to use, and another hour after returning to enter all of the expenses in the expensing system; I'd much rather send an email like "need to go to office X between Y and Z, no red eye flights" and "here's the receipts from our last trip, we took client W for a business dinner on May nth" and have it all happen.
Someone who does it several times per week would be much more efficient at doing it than me doing it a few times per year. But maybe in a few years we'll get some AI assistant that figures out that I like seat 3A, departures that are not too early, and figures out how to determine the expense types from various receipts.
For expense types I'm told it used to be eaiser. Then some scammers figured out they could send invoices and they would be paid now everyone needs to waste time tracking little expenses because if you don't even more money is wasted.
So there is a trade off which means some of the tedious jobs can't be automated. Though i agree I shouldn't have to separate my hotel room from meals at the hotel.
Software can only replace some roles that administrative staff filled. Most often, like in the case of email and word processors, it distributes the work and eats up everyone's time by moving it from specialists to non-specialists.
The optimum should be, of course, software empowering the specialists, so they can do more with less, providing better service to more people. But hey, a specialist costs $X in salary; a specialist + software that empowers them cost $X + $Y for the expensive license. Meanwhile, a SaaS that allows everyone to do the task lets us save $X on the specialist, and costs peanuts... plus a good fraction of everyone's salary, but nobody notices that.
The secretaries were specialists: in language. How much better would my posts be if someone who was good at writing and so would correct my grammar (grammar checkers are horrible - or were last time I tried, mostly I don't even look anymore). Often what I saw should be worded a little differently. Spell check doesn't notice when I spelled a different word than what I wanted.
I'm a faster typist than I am at talking, so I don't need a typist. I could really use someone to proofread for me. We have lost both.
I’m saying those programmers should have secretaries to help them with all of the admin etc. They shouldn’t be booking their own flights, or making dinner reservations, or running expenses (or a lot of other manual work)
Circa 2000 (and continuing through the 00s) there was a massive cutback in administrative personnel. By the time I got there (circa 2010) there were essentially no administrative support personnel except for those at the very top. During the 10s they realized that they were spending around $10k/year/person on travel related stuffs not because it was necessary, but because of the time lost to deal with the software that was supposed to remove the need for the full-time administrative staff.
By the end of the 10s, they'd restored the administrative teams and were spending much less per year on "overhead" (non-billable hour) even if you counted the admin teams as only overhead. Down from around $10 million to less than $1 million by just having a dedicated team that dealt with travel and finance stuffs.
The problem was that most people only traveled once a year, at best, and so they had no real experience with the unintuitive software. The average traveler was spending a week extra per trip, which was not billed to customers, dealing with reservations (1-2 days total pre-trip) and finances (2-3 days total post-trip).
Mythical Man Month describes secretaries in a software context. What spoke to me about it was none of things, but instead: scheduling meetings, taking and distributing agendas and notes for those meetings, being the curator and librarian of project documentation. Just because these things are digital now, doesn’t mean any of the engineers on the team will step up to do them consistently or well. Many projects could run more smoothly with someone in that role.
Every minute an engineer spends doing those things is a minute they aren't coding, debugging, or architecting. So what happens? We have to hire another engineer to pick up the slack. When we could hire an administrator for much less to handle those things.
I continue to be amused that the many of the activities and artifacts promotion committees want to see when they consider engineers for the highest levels, are exactly this kind of "admin" work.
And the same happened to everyone else too, just with smaller dollar amounts.
During my high school and early university years, I was in love with the concept of being able to run errands over the Internet. Why go to the bank when I can order a transfer on-line? Why make orders over the phone when I can choose what I want on a webpage with few clicks? Why ask anyone to do anything, when I could just click or type my way through?
As an adult with a bit more years behind me, I now feel the exact opposite way. Why on Earth am I doing these errands, when I could ask or pay someone else to do them? Why do I waste my time clicking on this bloated, user-hostile page full of upselling garbage, when I could just phone the company and tell them what I need? Alas, companies jumped at the opportunity to outsource the effort to customers, so increasingly I can't phone anyone. Self-service becomes the only option.
I suppose the shift in perspective comes from the fact that back then, I had no money and a lot of time; these days, I have some disposable income, but very little time to spare.
Phoning for anything is my favorite thing. Absolutely I want to talk to customer service. No I don’t want this automated (especially when I’m trying to get myself a slightly better deal). And I hate phone trees with a passion, just get me the operator now.
Phoning is my least favorite thing (well, except for in-person visits), but I also want a human on the other end of the line when I call. If I'm calling, it means I couldn't get the website to do the right thing for me.
To me, the web page nearly is always faster than asking a human. Even setting aside the phone trees designed to slow me down, I'm more comfortable with the bloated, user-hostile page than trying to understand a human voice through a 4000 Hz telephone channel. I like not having to try to explain to them what I want. With the web I can do it whenever I want, at my own pace.
It feels like a voice call is an admission of failure: sometimes of their web interface design, sometimes of my ability to read. If I am calling on the phone it's only because I want to talk to a human being, and I want that disgusting process over with as soon as possible.
There is never anything I want from the phone tree except a human operator. If you could automate my request then you should have done it over the far clearer interface of the web page. Maybe there are some people who have a phone but not a computer, but I am not one of them. I'm only talking to you because the easier (for me) ways have failed.
The idea was that all the jobs of someone handling errands for you were automated away and it became a self service thing. Yes, sometimes it is easier to buy something online, no doubt about that but when problems creep in it takes too much of our precious time and from our mental context dealing with things that would be dealt with if those jobs existed.
If those jobs existed in higher numbers you could just call a number and a human responds and assists on the other end of the line.
> There is never anything I want from the phone tree except a human operator. If you could automate my request then you should have done it over the far clearer interface of the web page.
This. It's particularly infuriating that now many phone trees don't even have a built-in "I want to talk to a human, now" option. Hitting "0" used to be a fairly common way to do that, but doesn't appear to be any more. Often I have to wade through three or four levels of phone menus just to get to something that will take me to a human.
I absolutely agree, and I also hate the automated chat systems which are even _worse_ than calling on the phone. It's trying to get the 'call tree' scripts out to chat. When if I'm online, a program script should be able to cover 80% of the issues, and the other 20% need a human touch for collecting the correct data or providing features missing; often intentionally from the website.
The local power company does this very well for reporting outages, and takes data that's hard to fit through a human that can recognize what the data means and get it to the correct departments and people.
Cable is the WORST in that respect. With zero transparency on what the root cause or investigation status. Also, insufficient and critically lacking detail (Is it JUST me, or is it my block, or is it the whole neighborhood or city, etc).
My favorite is when companies (looking at you, UPS) provide a phone menu with a very narrow list of choices, and if none of them matches what you want, you're just out of luck. There's no "other" option.
(In reality, you lie to get through to a human, but come on.)
Also, the difference between talking to somebody reading a script in Bangalore and someone on the inside who actually knows what they are doing (if you can manage to convince someone to connect you to the latter) is crazy. The drones handling customer support info often have essentially zero information on top of what's given to the customer.
One time UPS lost my package (bona fide lost in a warehouse, maybe stolen or something) and the phone CS drones assured me that it was on the way, just running late, was on the truck now, etc. etc.
I managed to berate them into connecting me with a US-based operator who, after asking how the hell I actually got her number, gave me the real tracking info, which is completely hidden from consumers and made it very clear that my package was gone.
Same with self checkout at the grocery store: I know a lot of people like it, but whenever I do it I’m thinking “this is someone else’s job! Why the hell am I having to do it now? ?”
2 employees assisting with 6-12 self-checkout stations is still a net-win for the employer if the objective is cutting "overhead" of personnel. It's only a problem if the self-checkout system is flawed in a way that requires frequent and lengthy intervention by the employees, versus the occasional ID check for alcohol purchases and item check for something not scanning correctly.
For me checkout with a cashier doesn't feel like less work though. I bring them my bag, take out each item, they scan them, and then I put them back in my bag. All they're doing is swiping! (Growing up we had disposable bags and human baggers, so maybe it saved a tiny bit more time.)
Good lord, you do all that with cashier? I load the stuff onto the belt (or don't even do that now thanks to COVID), and get a cart full of bagged groceries on the other side.
Have you noticed that change with COVID? Some places now seem to want you to be further away (sometimes with plexiglass) so some places do more bagging (actually putting the bags in your cart and pushing you a completed cart.) There are some places that seem to have gone the other way and gotten rid of the bagger so you can do your own bagging and placing 6 ft away from the cashier. But pre-COVID it seemed like most places had bags that you pick up at and put in the cart yourself.
Not really COVID related where I live. More or less the smaller the shop, the larger the chance they will bag your groceries. Large ones always have a conveyor belt from which they slide items on the other side where you can pick it up and bag yourself. I think the change mostly came when it has become illegal to give away bags. Since now you have to pay for them, a lot of people bring their own.
I buy a lot of fresh produce, which is way faster with a good cashier who knows all of the codes. I also tend to buy multiples of an item, which is also way faster when the cashier can swipe one and then hit 6x or whatevs.
And that's assuming the self-checkout system is working perfectly, which is rare. They often have some janky anti-theft sensors that freak out if you remove a bag or item from the bagging area. Self-checkout is fine or maybe even better if you have a few items, but for a cart full of groceries, it is inarguably way slower than a decent human cashier.
Around here we have two self-service checkout systems: one is as described, where you have to take each item individually out of your card and present it to a scanner. I detest those, as it makes me feel like a fool fumbling around with the scanners while the cashiers are 10x faster than I am.
The other one is where you pick up a handheld scanner at the store entrance, and you build up your item list while you're in the store loading your cart. Those ones I like, as they feel much faster and much more reliable to me. Moreover, I can look at the display of the scanner to see if I have all the items I came for, I don't have to search the cart.
Tangent about self checkout -- the versions I've found in supermarkets are uniformly terrible. But Home Depot, has a version I don't detest, for two reasons.
1. You get a hand-held scanner (at the checkout register, not walking around the store) instead of having to run everything over the counter.
2. You don't have to put the items on a certain platform (scale) after you scan; you can stick them back on your cart.
3. Although there is a webcam mounted (boo!), there is no screen showing yourself.
They're really three variations on the same reason: it doesn't treat me like a thief who's going to shoplift at the first opportunity.
I don't really mind doing self-checkout, as I'm nearly as fast at it as the cashiers (unless I have a lot of produce), and it lets me bag my food in exactly the order I want (bag for the fridge, bag for the pantry, etc). But the anti-theft measures (especially when the thing scans wrong and seizes up instead of letting me abort and try again) make it an absolute train wreck, and so I petty much only do self-checkout when I have under 5 items or the line is incredibly shorter.
One of the quotes suggests that DeMarco (the author of the book under discussion in this post) himself views 'slack' somewhat negatively:
> “Slack represents operational capacity sacrificed in the interests of long-term health.” [emphasis mine]
Quibble it might be, but nevertheless: surely if the goal is to appreciate the value of 'slack,' it would be better to describe it as an investment in long-term health, rather than a sacrifice for it?
To be fair, I suspect he was trying to implicitly refer to "opportunity cost" - every investment implies a "sacrifice" of the alternatives. But it seems to lead the blog post author into a similar dissonant frame of thinking:
> Slack consists of excess resources... Slack is vital...
Here I quibble with the use of "excess." Excess compared to what? Presumably, excess compared to the resources one might assume were necessary to guarantee a healthy operation - but as the author then notes, these resources are not "excess" at all, but indeed "vital!" In other words, they are fundamental to the continued existence of the operation. Why imply anything else?
I was reading this in the operations research sense of optimality. From this standpoint, they are excess in the current set of constraints but may be vital (an non-excessive) if those constraints change in the future.
Some good food for thought in the article. You want enough slack time and energy to be able to accomplish the high priority items. You definitely want to avoid low ROI busy work.
But also beware the easy seduction of the idea that you don't have to work hard to have a good chance of success.
> Saving 4 minutes a day on a process, over the course of your lifetime you will save 1200 hours. That’s a half year of vacation.
If I had to micromanage my personal life that way, I'd need a lot more than half a year of extra vacation in my lifetime to stay sane.
I value not having to think too hard about running my personal life higher than saving a few minutes here and there. Let alone wondering if those saved minutes are really fungible enough to translate into bulk time or money savings elsewhere.
In other words 99th percentile latency is what matters, not utilization. Anyone who's tried to get things done with batch-class resources has probably noticed this as well.
There's a similarity to the overall economy as well. Just in time inventory is certainly efficient but it's incredibly fragile. Just look at how much "damage" is being claimed for a boat that made other boats two weeks late.
Just to add, 99th percentile latency to the institution processes is an ok metric, but 99th percentile latency to personal or departmental processes is a naive and bad one.
If you are not making an effort to be complete on what you are measuring, you'll probably want to put more 9s there.
The most famous book to discuss this idea is The Goal by Eliyahu Goldratt, published in the days when American manufacturing was trying to compete with Japanese methods. That book inspired The Phoenix Project by Gene Kim and others, which applied the ideas to DevOps.
It's been several years, so I don't remember how TPP tied slack to concrete DevOps practices. But in Google's SRE book (published by O'Reilly), they talk about how if more than half of an SRE's time is consumed by incident response, they push maintenance back to the developers. Reserving 50+% time for project work is a way to maintain slack. (See Time Management for System Administrators by Thomas Limoncelli for more techniques.)
(EDIT: I'm starting to remember more details from TPP now: One way to add slack is to find & remove bottlenecks, e.g. the sysadmin who was "too good" at solving everyone's problems. This is also why you may want to mix more generalists into your teams than is strictly efficient. Likewise with having "cross-functional teams". They can share work so there are fewer bottlenecks.)
In other software development, you can achieve slack by filling each sprint with a mix of high- and low-urgency work. (And btw, we should replace the word "sprint".) Or leaving 20% of your time for refactoring. Or practicing the Boy Scout method (and factoring it into your estimates). Or when the graybeards double any estimate before sharing it with the customer. Google's 20% time is another form of slack.
Webdev shops struggle with this since utilization is a major driver for profitability. I've seen many start in-house products to fill the time between client work. You'd think these would turn out great, since they are (or ought to be) experts at building and launching new tech ventures. But I've only seen a couple work out. In practice they get neglected as soon as more billable work arrives.
What works better is a focus on internal tooling. This is much like Toyota's continuous improvement (a connection also made in The Goal). You don't get continuous improvement unless you have slack, and it's a good way to "use" your slack. If you don't have any ideas for internal tooling (ha), maybe encourage your devs to make some open-source contributions.
It's notable that you don't achieve slack by sleeping in. The secretaries still had to show up to work, even if they didn't have too much to do. So you still need a good work ethic. This makes me skeptical of the author's idea that you can motivate yourself with tight deadlines. I often wait until the last minute to do things, but that's not really buying me slack.
On the other hand playing Counterstrike may be genuine slack, since you can always turn it off if something comes up. :-)
For developers, another way to use slack time, besides building tools and playing games, is personal development: read a book, do a course, write a blog post, etc. Your greatest asset is your mind, and you must invest in it! Or engage in a community and meet new people. That is a kind of investment too.
I suspect slack is better managed in the small than in the large. I'm thinking of Hayek's critique of central planning in Road to Serfdom, or Michael Polanyi's objection to centrally-planned scientific research. I knew a company once that devoted one in four sprints to refactoring. While that would be an improvement in many places, it feels a bit too centrally-planned to me. Give people slack, but let them use it how they like.
It's worth noting that part of how TPS (Toyota Production System) achieves continuous improvement is by creating/exacerbating pain points. By reducing inventory, in particular, it reveals issues in the system that might get papered over elsewhere.
Things that high inventory can cover up:
- Low yield. If half your produced product is low quality and unsellable, but you can still make enough and have enough inventory you can still hit your targets. But you're not motivated (or insufficiently motivated) to improve the yield.
- Maintenance/repair downtime. If you can run flat out 24 hours a day for 3 months and get enough inventory to cover that one month of repair work, you're not encouraged to improve the maintenance program or maintainability of the system.
- Tooling changeover (a big one in TPS). Related to the preceding one. If tooling changeover introduces a week of downtime, but your inventory is high enough to absorb that loss, then you're not motivated to improve the changeover.
By reducing inventory (it's not the only way, but it is a way) these issues become much more apparent and can then be addressed. Continuing with this, in TPS/Lean as you improve one area you can either produce more (which means selling more) or produce enough (but less than before, because there's less need to paper over issues with high inventory) and free up capital and resources for the next area with issues. Which also introduces slack into the system so you can handle surges in demand more effectively.
And this is why you schedule flights in the morning: there's so damn little slack in the system that if things go wrong at 10:00 AM at O'Hare, flights for the whole rest of the day are screwed up owing to the cascading delays.
At least in the morning the airlines have had the overnight to unsnarl the previous day's mess because of the reduced revenue traffic overnight and the corresponding slack that accrues as a happy side effect.
This is also why things like the healthcare system, transportation network, and postal system shouldn't be run for maximum utilization/efficiency under normal load: if there isn't any slack in the system it gets real ugly when things get squirrelly. In cases of localized disturbance, we get by on mutual aid: linemen and bucket trucks from far away respond in the aftermath of e.g. a hurricane or tornado. Likewise, fire departments from all over The greater Boston area responded to the gas explosions in the Merrimack Valley[0] and companies from even further away repositioned trucks to cover the departments that responded directly.
When it's national or global scale event, you're left leaning on whatever slack was in the system. As we're all painfully aware, there hasn't been enough. The public health and healthcare systems have been doing heroic work, but if they weren't stretched so tight in the name of efficiency beforehand, there'd be less need for the heroic efforts.
> This is also why things like the healthcare system, transportation network, and postal system shouldn't be run for maximum utilization/efficiency under normal load: if there isn't any slack in the system it gets real ugly when things get squirrelly.
A couple years before the pandemic, the counties neighboring mine (where I lived at the time) cut out most public medical services (mind you, these aren't free, just publicly funded, people still paid for them). There wasn't enough money for a private hospital to bother so this was a critical piece of infrastructure. They kept emergency and urgent care clinics in each county, but directed people to the other (more populated, higher income) counties like mine for many services. Last year was not a good year for those counties as people were now being shuttled 40+ miles if they needed to be treated (at a hospital) for COVID.
Lean means cutting the fat, not the meat. They didn't just cut the fat and meat, they cut to the bone. This is when organizations find themselves in trouble and fail their clients/customers: eliminating things they don't think they need because they're "underutilized", only to discover later there was a sound reason to have that capability to begin with.
Yep, I started my programming career optimizing stuff (traveling salesmen, roster, schedules) and the optimizing criteria in academia is very far from reality. The effects of not having slack are disastrous.
Although, the effects start showing after the optimization makes people believe they can squeeze even more work. The initial optimized schedules create more slack than handcrafted ones.
Been seeing this with UPS after the Texas snow storm. Packages were canceled or delivered late, often missing parts of it, for over a month. It's an obvious problem with homebuilders, too. I keep seeing all these constructions just halt all work, sometimes for a few weeks or months, sometimes for a couple years. They seem to operate with absolutely no margin for error and run out of money really easily.
"The best time to fly is between 6 and 7 in the morning. Flights scheduled to depart in that window arrived just 8.6 minutes late on average. Flights leaving before 6, or between 7 and 8, are nearly as good.
But delay times build from there. Through the rest of the morning and the afternoon, for every hour later you depart you can expect an extra minute of delays. Delay times peak at 20.7 minutes — more than twice as long as for early-morning flights — in the block between 6 and 7 p.m. They remain at 20-plus minutes through the 9 p.m. hour."
It's been several years since I looked at the data, but if I remember correctly most flights actually arrive ahead of time. The median if I remember correctly is actually negative (ie. at least 50% of flights arrive ahead of time).
However, this means that there's a long tail of flights that get delayed an hour or more, which brings the average up. Not a good situation to be in if one has to make a late connection during the day.
It's a little disappointing that 538 didn't talk about the distribution on this.
I'm looking to avoid the one flight that's delayed an hour that means I miss my connection because my layover was an hour. Only 5 flights have to arrive on time to hit an average arrival delay of 10 minutes. Those are not favorable odds, as I see them. Especially if it's cutting into my vacation time.
Yes, missing a friend's wedding or an on site interview or ..., because no more connecting flights until same time tomorrow, with a probability 1 in 6 is pretty bad odds
OK, but the corollary to this is that there are a fair few people sitting around doing not a lot on the taxpayer's dime. That's the visible side-effect of organisational slack (as TFA points out: the secretary spends most of her day doing nothing).
There's been a lack of tolerance for that. We tend to feel that if our tax dollars are paying for someone, that someone had better be busy all day long. Politicians have made political capital from "cutting slack" in public services.
We need a cultural recognition that slack is good. And I doubt that's going to happen any time soon.
Shortly before the Covid-19 pandemic, a highly-influential think tank (the Bertelsmann-Stiftung) suggested closing half of Germany's hospitals because of the low bed utilization and simple inefficiencies that pop up when smaller hospitals deal with less-common cases.
That plan has now been effectively scrapped. The large number of ICU beds per capita has been one of the main reasons why Germany got so well through the first wave. The plan also overlooked that the main bottleneck has always been staff, who were already running with very low slack.
334 comments
[ 3.1 ms ] story [ 207 ms ] threadThe rule of thumb is that you want utilization to be where there is an acceptable latency depending on some percentile of cases. For a firefighter, you'd probably look at p99 latency. For a hamburger joint, p50 on order time would be good enough.
That's why I didn't call for it.
I believe OP's point is that you do want the staff of 20, for the once-in-a-lifetime fire that requires 20 people. See also: the recent Texas power grid debacle, or the saturation of ERs due to COVID.
No, you don't. Resources are not infinite, and at some point the budget has to be taken from other services which are more useful than a once-in-a-lifetime event.
Or you do want the staff of 20, but then it must be a volunteer and/or on-call duty system. Otherwise it is not sustainable.
In my country, the shift/switch to professionalisation that started 30 years ago has gradually become a big problem. Despite the fact that they still represent only 20% of the firemen, they are killing the budgets, and they always want more (lots of strikes); apart from 'standard' raises, the most common thing they ask for, is that on-call hours should be paid full-rate, as active hours. Which, beside being extremely costly, is absurd when they are 'working' 24h shifts! It contains a few hours of training and, depending on location, a few hours of duty; the rest is on-call (at home or on premises depending on the type of station), the number of service calls is limited, and 1 in 4 shifts happen without a single call (even more for 12h shifts).
There are plenty of other problems which surround this professionalisation, but they are not directly related to this subject.
If you get paid the same, responding to a call just becomes an annoying interruption ("I could be at home making the same money if this person I'm helping had just been more careful!"), leading to a worse experience for everybody.
The general point, I think, could be summarized as: you want some idle resources to handle surge capacity. Whether that’s 5 or 20 firefighters is of course the next question.
"Creativity is the residue of time wasted" - probably not Albert Einstein but it's attributed to him
100% utilization is moronic, you need slack in your system or you will introduce the risk of a severe backlog of work that may never finish (or will finish because clients cancel the requests and go elsewhere).
See also The Goal.
I like being efficient in doing work to get things done. I also like slack because I want to enjoy life.
I also recognize increasing workload doesn't mean being efficient, just that you do more work.
Increased efficiency means more money (or joy, or other things with positive utility), or less work :)
My claim is this can be unhealthy because you're not really living in the present and constantly stuck in the mindset of "once this distasteful thing is done, then I can get to the stuff that makes me happy." Or to steal from The Good Place, "Humans only live for 80 years and they spend so much of it waiting for things to be over."
If you do something because you enjoy it, it's a "hobby". If you are paid for that, too, you are just lucky to have the best of both worlds :)
As a sysadmin / DevOps / SRE / whatever, I also realized at some point that being constantly busy is actually a state of extreme fragility.
Nowadays I try spending a significant part of my day just trying new stuff and reading, not being micromanaged helps a lot.
Nobody called me out on it but it wouldn’t have been the first time in my career.
But now I find out belatedly that we’re changing our auth system, and now that work is going to save me from having to drop everything to get it done on time.
No matter your efforts, there's a time for everything.
Depending on what is going on, an operational team will spend 20 - 40% of their time firefighting or at tightening screws and oiling wheels - maintaining systems. Sometimes it's a good week and it's just 10%. Sometimes you launched a new product, and it's 60% because everything is failing.
As a conclusion from there, it's not a good idea to schedule more than 50% - 60% of deliverables with deadlines, because the right outage is going to toss those estimates really quickly.
That's in itself the definition of sufficient slack. If you don't have that, prod fails and no one is around to fix it. If you do, someone can usually start poking at it quickly.
Luckily we've usually been able to avoid Scrum etc, and work in a way closer to Kanban.
There have been times though (like right now sigh) where we're forced into a throughput oriented mode by commitments made elsewhere out of our control. It sucks, and we end up ignoring too much of the little stuff for long enough that they end up becoming fires you need to put out (is ops debt a thing?), your tooling and automation suffer, knowledge silos build up within the team, and the throughput will end up tanking anyway.
I like to use 50% allocation on "project" work as a nice rule of thumb for reactive ops oriented teams. Any higher can only be sustained for short periods without negative effects.
This article resonated with me pretty deeply.
Businesses use just in time inventory to increase efficiency, but that means a small disruption anywhere in the supply chain can grind everything to a halt.
Also, instead of maintaining cash savings, businesses will rely on cheap credit. However, if enough businesses do that, any type of credit crunch results in widespread economic disaster.
Often business efficiency is a way to privatize the gains from efficiency, but but have the public bear the losses from lack of resilience.
Just like banks are required to keep a reserve (which is inefficient in some sense), I think some type of regulation or increasing the personal moral hazard for business leaders is required to make sure businesses don’t sacrifice resiliency for too much efficiency (and the profits of efficiency).
To counter the original point, I find removing obstacles and latency-inducing loops helpful, to start seeing what the work really should be. Gaining efficiency through simplifying is a good thing, and can be creative too. The goal is not efficiency though.
They sound similar, which is probably why we use the word "efficiency" to describe improvements in both regimes, but the fundamental constraint is different: in the first case, it's the standard of work that must be achieved; in the second, it's the resources allocated to the work. I'd summarize the first "do enough with enough" and the second as "do more with less."
What you describe sounds like "doing enough with enough:" given the work to be done, how can we remove resource-draining obstacles, idle loops, etc. and identify "what the work really should be?" - is that a fair assessment or am I off the mark?
I'd rather people come up with their own ideas.
Should MBA's be studying DevOps?
It helps to ask around.
> It helps to ask around
It helps whom to ask whom about what?
Even shorter: there's no single absolute optimum; if you optimize for efficiency, you lose in other areas. But if you optimize in other areas, you lose in efficiency, of course. Everything in real life is a compromise.
It reminds me of working a scheduling problem that failed to factor in union concerns. It was a bad solution because it didn't factor in all the dimensions of the actual problem and only originally concerned itself with "management's" cost concerns, not the "union's" cost concerns.
Tbf, I never finished "Antifragile" as I felt like it just kept going over the same concept from different angles without introducing anything new after the first 50 or so pages.
Which is sort of ironic from my department, because this has also been a year where our process optimisers and MBAs have been almost completely unable of performing their usual efficiency and benefit realisation consulting in our different departments, as that’s a hands on sort of thing. Not that they’ve done nothing, they’ve been to really good work helping managers coordinate remote work and teaching both the CEO and Political layers how to use Microsoft teams efficiently.
Anyway, if we’ve increased efficiency and quality more in a year or not trying to, it sort of begs the question what good trying really does. You obviously can’t really conclude anything scientific on our anecdotal measurements as we’ve seen the major change of going remote on top of it, but it is something to think about.
Not that we will, we’re already trying to figure out how to go back to the way things were, as the majority of our managers still seem to think people work better if they spend 7 and a half hours in an open office 5 days a week.
They had been have issues with improving productivity. One of the issues was the workforce and unions were reluctant to accept any change.
New management came in and rather than pressing on productivity issues they decided to double down on safety. They hadn't been particularly bad but not great either.
Obviously staff and unions are never going to complain about making their workplace safer. What also happened was as soon as process were open to change to improve safety they were also open to change for productivity.
I wonder in cases like yours whether the pandemic just unstuck a lot of things because all the previous excuses floated away as soon as someone says "because covid".
Devs in engineering org now have to spend more time on self-service portals & chasing tickets because the manager of the infrastructure org laid off a bunch of sysadmins.
I once worked at a bank where even replacing a physical disk in a US datacenter involved a ticketing system which dispatched tickets to India. The remote guys would then, presumably, raise some sort of internal ticket so the guy physically in US could you know.. replace a bad disk.
Turnaround on bad disk swaps went from hours to weeks. As the hardware aged, we started to have enough disk failures pile up on RAID arrays that data losses occurred.
Somewhere someone in infra cut his budget though!
Open offices: another amazing example of enormous value destruction in the name of saving a little bit of money.
So some of the savings is good. It is faster for me to schedule a meeting in outlook (not the same company) than to find a secretary to schedule my meetings. However the secretary might be worth going back to just because they always knew important gossip that was worth knowing.
For example, in one of my internships, it turns out that someone mistyped my address so my paychecks were sent to the wrong building; after a few weeks of that not getting resolved through HR, the administrative assistant took it upon herself to fix it and figured out whom to go yell at to get it resolved within a few days.
They're also good for things where having specialized knowledge of a process that's not done often can be done by someone who does it more often.
For example, when it comes to corporate travel, our company has a self service portal, and every time I need to book business travel I have waste an hour to figure out the right combination of flights and hotels to use, and another hour after returning to enter all of the expenses in the expensing system; I'd much rather send an email like "need to go to office X between Y and Z, no red eye flights" and "here's the receipts from our last trip, we took client W for a business dinner on May nth" and have it all happen.
Someone who does it several times per week would be much more efficient at doing it than me doing it a few times per year. But maybe in a few years we'll get some AI assistant that figures out that I like seat 3A, departures that are not too early, and figures out how to determine the expense types from various receipts.
So there is a trade off which means some of the tedious jobs can't be automated. Though i agree I shouldn't have to separate my hotel room from meals at the hotel.
Software scales - that's why programmers have such high salaries (which are usually only a fraction of the value that they're delivering anyway).
The optimum should be, of course, software empowering the specialists, so they can do more with less, providing better service to more people. But hey, a specialist costs $X in salary; a specialist + software that empowers them cost $X + $Y for the expensive license. Meanwhile, a SaaS that allows everyone to do the task lets us save $X on the specialist, and costs peanuts... plus a good fraction of everyone's salary, but nobody notices that.
I'm a faster typist than I am at talking, so I don't need a typist. I could really use someone to proofread for me. We have lost both.
Circa 2000 (and continuing through the 00s) there was a massive cutback in administrative personnel. By the time I got there (circa 2010) there were essentially no administrative support personnel except for those at the very top. During the 10s they realized that they were spending around $10k/year/person on travel related stuffs not because it was necessary, but because of the time lost to deal with the software that was supposed to remove the need for the full-time administrative staff.
By the end of the 10s, they'd restored the administrative teams and were spending much less per year on "overhead" (non-billable hour) even if you counted the admin teams as only overhead. Down from around $10 million to less than $1 million by just having a dedicated team that dealt with travel and finance stuffs.
The problem was that most people only traveled once a year, at best, and so they had no real experience with the unintuitive software. The average traveler was spending a week extra per trip, which was not billed to customers, dealing with reservations (1-2 days total pre-trip) and finances (2-3 days total post-trip).
During my high school and early university years, I was in love with the concept of being able to run errands over the Internet. Why go to the bank when I can order a transfer on-line? Why make orders over the phone when I can choose what I want on a webpage with few clicks? Why ask anyone to do anything, when I could just click or type my way through?
As an adult with a bit more years behind me, I now feel the exact opposite way. Why on Earth am I doing these errands, when I could ask or pay someone else to do them? Why do I waste my time clicking on this bloated, user-hostile page full of upselling garbage, when I could just phone the company and tell them what I need? Alas, companies jumped at the opportunity to outsource the effort to customers, so increasingly I can't phone anyone. Self-service becomes the only option.
I suppose the shift in perspective comes from the fact that back then, I had no money and a lot of time; these days, I have some disposable income, but very little time to spare.
It feels like a voice call is an admission of failure: sometimes of their web interface design, sometimes of my ability to read. If I am calling on the phone it's only because I want to talk to a human being, and I want that disgusting process over with as soon as possible.
There is never anything I want from the phone tree except a human operator. If you could automate my request then you should have done it over the far clearer interface of the web page. Maybe there are some people who have a phone but not a computer, but I am not one of them. I'm only talking to you because the easier (for me) ways have failed.
This. It's particularly infuriating that now many phone trees don't even have a built-in "I want to talk to a human, now" option. Hitting "0" used to be a fairly common way to do that, but doesn't appear to be any more. Often I have to wade through three or four levels of phone menus just to get to something that will take me to a human.
The local power company does this very well for reporting outages, and takes data that's hard to fit through a human that can recognize what the data means and get it to the correct departments and people.
Cable is the WORST in that respect. With zero transparency on what the root cause or investigation status. Also, insufficient and critically lacking detail (Is it JUST me, or is it my block, or is it the whole neighborhood or city, etc).
(In reality, you lie to get through to a human, but come on.)
Also, the difference between talking to somebody reading a script in Bangalore and someone on the inside who actually knows what they are doing (if you can manage to convince someone to connect you to the latter) is crazy. The drones handling customer support info often have essentially zero information on top of what's given to the customer.
One time UPS lost my package (bona fide lost in a warehouse, maybe stolen or something) and the phone CS drones assured me that it was on the way, just running late, was on the truck now, etc. etc.
I managed to berate them into connecting me with a US-based operator who, after asking how the hell I actually got her number, gave me the real tracking info, which is completely hidden from consumers and made it very clear that my package was gone.
And that's assuming the self-checkout system is working perfectly, which is rare. They often have some janky anti-theft sensors that freak out if you remove a bag or item from the bagging area. Self-checkout is fine or maybe even better if you have a few items, but for a cart full of groceries, it is inarguably way slower than a decent human cashier.
The other one is where you pick up a handheld scanner at the store entrance, and you build up your item list while you're in the store loading your cart. Those ones I like, as they feel much faster and much more reliable to me. Moreover, I can look at the display of the scanner to see if I have all the items I came for, I don't have to search the cart.
1. You get a hand-held scanner (at the checkout register, not walking around the store) instead of having to run everything over the counter.
2. You don't have to put the items on a certain platform (scale) after you scan; you can stick them back on your cart.
3. Although there is a webcam mounted (boo!), there is no screen showing yourself.
They're really three variations on the same reason: it doesn't treat me like a thief who's going to shoplift at the first opportunity.
I don't really mind doing self-checkout, as I'm nearly as fast at it as the cashiers (unless I have a lot of produce), and it lets me bag my food in exactly the order I want (bag for the fridge, bag for the pantry, etc). But the anti-theft measures (especially when the thing scans wrong and seizes up instead of letting me abort and try again) make it an absolute train wreck, and so I petty much only do self-checkout when I have under 5 items or the line is incredibly shorter.
> “Slack represents operational capacity sacrificed in the interests of long-term health.” [emphasis mine]
Quibble it might be, but nevertheless: surely if the goal is to appreciate the value of 'slack,' it would be better to describe it as an investment in long-term health, rather than a sacrifice for it?
To be fair, I suspect he was trying to implicitly refer to "opportunity cost" - every investment implies a "sacrifice" of the alternatives. But it seems to lead the blog post author into a similar dissonant frame of thinking:
> Slack consists of excess resources... Slack is vital...
Here I quibble with the use of "excess." Excess compared to what? Presumably, excess compared to the resources one might assume were necessary to guarantee a healthy operation - but as the author then notes, these resources are not "excess" at all, but indeed "vital!" In other words, they are fundamental to the continued existence of the operation. Why imply anything else?
I was reading this in the operations research sense of optimality. From this standpoint, they are excess in the current set of constraints but may be vital (an non-excessive) if those constraints change in the future.
But also beware the easy seduction of the idea that you don't have to work hard to have a good chance of success.
> Saving 4 minutes a day on a process, over the course of your lifetime you will save 1200 hours. That’s a half year of vacation.
If I had to micromanage my personal life that way, I'd need a lot more than half a year of extra vacation in my lifetime to stay sane.
I value not having to think too hard about running my personal life higher than saving a few minutes here and there. Let alone wondering if those saved minutes are really fungible enough to translate into bulk time or money savings elsewhere.
There's a similarity to the overall economy as well. Just in time inventory is certainly efficient but it's incredibly fragile. Just look at how much "damage" is being claimed for a boat that made other boats two weeks late.
If you are not making an effort to be complete on what you are measuring, you'll probably want to put more 9s there.
It's been several years, so I don't remember how TPP tied slack to concrete DevOps practices. But in Google's SRE book (published by O'Reilly), they talk about how if more than half of an SRE's time is consumed by incident response, they push maintenance back to the developers. Reserving 50+% time for project work is a way to maintain slack. (See Time Management for System Administrators by Thomas Limoncelli for more techniques.)
(EDIT: I'm starting to remember more details from TPP now: One way to add slack is to find & remove bottlenecks, e.g. the sysadmin who was "too good" at solving everyone's problems. This is also why you may want to mix more generalists into your teams than is strictly efficient. Likewise with having "cross-functional teams". They can share work so there are fewer bottlenecks.)
In other software development, you can achieve slack by filling each sprint with a mix of high- and low-urgency work. (And btw, we should replace the word "sprint".) Or leaving 20% of your time for refactoring. Or practicing the Boy Scout method (and factoring it into your estimates). Or when the graybeards double any estimate before sharing it with the customer. Google's 20% time is another form of slack.
Webdev shops struggle with this since utilization is a major driver for profitability. I've seen many start in-house products to fill the time between client work. You'd think these would turn out great, since they are (or ought to be) experts at building and launching new tech ventures. But I've only seen a couple work out. In practice they get neglected as soon as more billable work arrives.
What works better is a focus on internal tooling. This is much like Toyota's continuous improvement (a connection also made in The Goal). You don't get continuous improvement unless you have slack, and it's a good way to "use" your slack. If you don't have any ideas for internal tooling (ha), maybe encourage your devs to make some open-source contributions.
It's notable that you don't achieve slack by sleeping in. The secretaries still had to show up to work, even if they didn't have too much to do. So you still need a good work ethic. This makes me skeptical of the author's idea that you can motivate yourself with tight deadlines. I often wait until the last minute to do things, but that's not really buying me slack.
On the other hand playing Counterstrike may be genuine slack, since you can always turn it off if something comes up. :-)
For developers, another way to use slack time, besides building tools and playing games, is personal development: read a book, do a course, write a blog post, etc. Your greatest asset is your mind, and you must invest in it! Or engage in a community and meet new people. That is a kind of investment too.
I suspect slack is better managed in the small than in the large. I'm thinking of Hayek's critique of central planning in Road to Serfdom, or Michael Polanyi's objection to centrally-planned scientific research. I knew a company once that devoted one in four sprints to refactoring. While that would be an improvement in many places, it feels a bit too centrally-planned to me. Give people slack, but let them use it how they like.
Things that high inventory can cover up:
- Low yield. If half your produced product is low quality and unsellable, but you can still make enough and have enough inventory you can still hit your targets. But you're not motivated (or insufficiently motivated) to improve the yield.
- Maintenance/repair downtime. If you can run flat out 24 hours a day for 3 months and get enough inventory to cover that one month of repair work, you're not encouraged to improve the maintenance program or maintainability of the system.
- Tooling changeover (a big one in TPS). Related to the preceding one. If tooling changeover introduces a week of downtime, but your inventory is high enough to absorb that loss, then you're not motivated to improve the changeover.
By reducing inventory (it's not the only way, but it is a way) these issues become much more apparent and can then be addressed. Continuing with this, in TPS/Lean as you improve one area you can either produce more (which means selling more) or produce enough (but less than before, because there's less need to paper over issues with high inventory) and free up capital and resources for the next area with issues. Which also introduces slack into the system so you can handle surges in demand more effectively.
At least in the morning the airlines have had the overnight to unsnarl the previous day's mess because of the reduced revenue traffic overnight and the corresponding slack that accrues as a happy side effect.
This is also why things like the healthcare system, transportation network, and postal system shouldn't be run for maximum utilization/efficiency under normal load: if there isn't any slack in the system it gets real ugly when things get squirrelly. In cases of localized disturbance, we get by on mutual aid: linemen and bucket trucks from far away respond in the aftermath of e.g. a hurricane or tornado. Likewise, fire departments from all over The greater Boston area responded to the gas explosions in the Merrimack Valley[0] and companies from even further away repositioned trucks to cover the departments that responded directly.
When it's national or global scale event, you're left leaning on whatever slack was in the system. As we're all painfully aware, there hasn't been enough. The public health and healthcare systems have been doing heroic work, but if they weren't stretched so tight in the name of efficiency beforehand, there'd be less need for the heroic efforts.
[0] https://en.wikipedia.org/wiki/Merrimack_Valley_gas_explosion...
A couple years before the pandemic, the counties neighboring mine (where I lived at the time) cut out most public medical services (mind you, these aren't free, just publicly funded, people still paid for them). There wasn't enough money for a private hospital to bother so this was a critical piece of infrastructure. They kept emergency and urgent care clinics in each county, but directed people to the other (more populated, higher income) counties like mine for many services. Last year was not a good year for those counties as people were now being shuttled 40+ miles if they needed to be treated (at a hospital) for COVID.
Lean means cutting the fat, not the meat. They didn't just cut the fat and meat, they cut to the bone. This is when organizations find themselves in trouble and fail their clients/customers: eliminating things they don't think they need because they're "underutilized", only to discover later there was a sound reason to have that capability to begin with.
I will upvote you just for coming up with this illustration. This should be useful for future reference.
Although, the effects start showing after the optimization makes people believe they can squeeze even more work. The initial optimized schedules create more slack than handcrafted ones.
https://fivethirtyeight.com/features/fly-early-arrive-on-tim...
"The best time to fly is between 6 and 7 in the morning. Flights scheduled to depart in that window arrived just 8.6 minutes late on average. Flights leaving before 6, or between 7 and 8, are nearly as good.
But delay times build from there. Through the rest of the morning and the afternoon, for every hour later you depart you can expect an extra minute of delays. Delay times peak at 20.7 minutes — more than twice as long as for early-morning flights — in the block between 6 and 7 p.m. They remain at 20-plus minutes through the 9 p.m. hour."
However, this means that there's a long tail of flights that get delayed an hour or more, which brings the average up. Not a good situation to be in if one has to make a late connection during the day.
I'm looking to avoid the one flight that's delayed an hour that means I miss my connection because my layover was an hour. Only 5 flights have to arrive on time to hit an average arrival delay of 10 minutes. Those are not favorable odds, as I see them. Especially if it's cutting into my vacation time.
There's been a lack of tolerance for that. We tend to feel that if our tax dollars are paying for someone, that someone had better be busy all day long. Politicians have made political capital from "cutting slack" in public services.
We need a cultural recognition that slack is good. And I doubt that's going to happen any time soon.
Firemen have lots of idle time? I think they play floorball for example (good for them to stay fit)
Politicians do like to turn things into moral issues, this is one of their favourites.
That plan has now been effectively scrapped. The large number of ICU beds per capita has been one of the main reasons why Germany got so well through the first wave. The plan also overlooked that the main bottleneck has always been staff, who were already running with very low slack.