HN Explorer

Ask HN: Literature on crisis response?

72 points by veddox ↗ HN
So I've always been interested in how teams respond to crises. I love reading about teams that handle crises well, and try to figure out what I can learn from their actions for the teams I lead. (Apollo 13 continues to be an all-time favourite movie of mine...)

Recently, there've been two discussions on HN [1,2] that have gotten me thinking about this topic again. And now I'm wondering: are there any good books on the topic that you can recommend? I'm not restricting myself to any domain - business, politics, engineering, natural disasters, could all be interesting.

[1] https://news.ycombinator.com/item?id=26506920

[2] https://news.ycombinator.com/item?id=26539495

38 comments

[ 5.4 ms ] story [ 88.6 ms ] thread
markedathome
The CDC has the Crisis & Emergency Risk Communication (CERC) manual[1], as well as a website covering this topic[2].

[1] https://emergency.cdc.gov/cerc/ppt/cerc_2014edition_Copy.pdf https://emergency.cdc.gov/cerc/manual/index.asp

[2] https://emergency.cdc.gov/cerc/index.asp

secfirstmd
If you want tackle advice on common types of crisis that can happen to people (malware, kidnap etc) feel free to check out our free open source app Umbrella. It's about 110k words, 40+ lessons on best practices, in 7 languages. You can look for it on the app stores or find out more (https://www.secfirst.org)
Leparamour
Thanks, I'll check it out. Does the app work offline too in a situation where the crisis also includes lack of internet connection? (eg. natural disasters or places without reliable internet coverage)
secfirstmd
Yep. All the content is downloaded as it's designed to work in remote locations or with internet shut downs.
pjmorris
For more on the Apollo 13 crisis response, check out 'Apollo: Race to the Moon' by Murray and Cox. I haven't met a more detailed popular examination of the engineering and management effort behind the Apollo program, and they spend some time on the 'back room' of engineers depicted in the film.

I'm reminded of 'The Medical Detectives', Roueche, but only by reputation (I own a copy I haven't read.) "In each true story, local health authorities and epidemiologists race against time to find the clue to an unknown and possibly fatal disease."

If you interpret 'The enemy might get the bomb before we do' as a crisis, 'The Making of the Atomic Bomb', Rhodes, is a detailed (and Pulitzer Prize-winning) examination of how we got from discovering the atom's nucleus to the consequences of deploying city-destroying weapons in a generation or so.

You might find general systems theory interesting, maybe 'Thinking In Systems', Meadows, and/or 'An Introduction to General Systems Thinking', Weinberg.

yodon
If you're operating at a scale or in a domain where crisis-like issues are expected (which is probably true if you're asking a question like this), The Checklist Manifesto[0] is a great read.

[0] https://www.amazon.com/Checklist-Manifesto-How-Things-Right/...

cinntaile
I read this book a while ago based on the fact that a lot of HN users recommended it, but I didn't think it was that great. It could have been 5 pages to get the point across, but that doesn't sell a book. It's been a while but if I recall correctly you could summarize it to: "If it's important to not miss any steps, use a checklist. Don't have too many or people will skip it. Only include relevant steps. Checklists are useless if they're not followed." I'm probably butchering it a little though to be fair, if it interests you give it a chance.
yodon
Like most "business books" it's designed to be read during a single bowel movement by a normal person with normal digestive functions.

The book offers a single unit of recommendations, a good one that's worth reading, but definitely not the complete works of Shakespeare.

yesenadam
(This is not only a response to this particular comment but also a general rant about such comments)

What is the point of this kind of comment? You can summarize an X page book in X/10 words. So what? Would that summary get the book's message out effectively to people?

A 200 page book (about anything, I don't mean about checklists) written entirely in the style of your summary would be unreadable. Or it would be some kind of reference, like man pages, usually not something people read with pleasure, and not something to introduce you to a topic. The less a book invites "This could be X pages"-type criticisms, the more unreadable it is.

Aside from that, I found the memorable thing about the book to be the stories of how surprisingly effective checklists are, what a difference they can make, and how and why things have gone wrong without them. All that stuff you would cut as superfluous.

Non-fiction books, like (e.g. C or Python) functions, don't have to be "great"; it's enough if they do one thing and do it well. The book did that, and very memorably.

And maybe someone will read between the lines and learn or see something else about the world or people or our systems from the book. I find such criticisms mistakenly assume only one thing can be gotten from a book, and their summary presents that. In university courses, a lot of the benefit comes not from the content but seeing an expert in action, how they think, how they talk, what it looks like to be a professional in that subject.

I am often reminded, by the frequent criticisms that speakers should speak faster, books/blog posts should be shorter, people AskingHN how to learn music as fast as possible etc, of this Zen parable:

    A young but earnest Zen student approached his teacher, and asked the Zen Master:
    “If I work very hard and diligent how long will it take for me to find Zen ?”
    The Master thought about this, then replied, “Ten years.”
    The student then said, “But what if I work very, very hard and really apply myself to learn fast — How long then ?”
    Replied the Master, “Well, twenty years.”
    “But, if I really, really work at it. How long then ?” asked the student. “Thirty years,” replied the Master.
    “But, I do not understand,” said the disappointed student.
    “At each time that I say I will work harder, you say it will take me longer.
    Why do you say that ?”
    Replied the Master, “When you have one eye on the goal, you only have one eye on the path.”
It is also reminiscent of "efficiency" in modern economics, where we are worried about systems being as efficient as possible—to get where we are going sooner—but where we are actually going, is not discussed.

You could summarize this comment in 5 words, but you would lose the unforgettable, epic journey this comment has taken you on.

cinntaile
> Aside from that, I found the memorable thing about the book to be the stories of how surprisingly effective checklists are, what a difference they can make, and how and why things have gone wrong without them. All that stuff you would cut as superfluous.

I summed up the book in one line, one line doesn't do it justice. The points you mention would be included in a good summary.

I guess my expectations simply differed from reality. I was expecting more practical advice on how to design good checklists, what pitfalls to avoid, good practices and so on. What I got instead is different stories where he shows that checklists work. Personally I did not think that more stories added much substance to the general message, but I get your point and it's certainly a valid criticism.

yesenadam
Ok thanks, all fair enough. I guess mainly "It could have been 5 pages to get the point across, but that doesn't sell a book." seemed...unreasonably hostile/cynical.

Someone is going to complain this exchange is too civil–it's happened to me before on here. :-)

Darkstryder
With regard to the specific topic of communicating through a crisis, I've found Masters of Disaster: The Ten Commandments of Damage Control by Christopher Lehane, Mark Fabiani and Bill Guttentag to be pretty interesting.

The authors worked with multiple organizations and celebrities during time of crises and helped them with the public relation side of it. For instance they managed Bill Clinton's PR during the Monica Lewinsky events.

The book is not perfect (I think it could be shortened a bit and retain the same information) but it is still very interesting and I think about it every time I witness someone getting themselves into a big, public crisis and making things worse by not managing their PR properly.

tjalfi
Total Loss[0] has 45 stories of yachting disasters; the lesson I took away is to carry a knife when you're on a boat.

[0] https://www.amazon.com/Total-Loss-Collection-First-hand-Acco...

eschneider
As someone who used to work on lobster boats, they tell you to bring TWO knives. When the pot goes to the bottom accidentally roped to your leg, you always panic and drop knife one. You need knife two to actually cut yourself free.
jenkstom
In the US all first responders are required to go through various levels of incident management. You can get free training on NIMS and ICS online. It works, and it works very well.
cjbprime
The obvious choice is aviation -- there are thousands of commercial accident reports, many of which lead to process improvements for everyone else afterwards, and general aviation emergencies every day. YouTube especially is full of ATC audio combined with radar visuals and commentary for emergencies.

For tech, Dan Liu maintains a list of tech company incident public post-mortems: https://github.com/danluu/post-mortems

rockmeamedee
The term "crisis response" will get you info on PR crises and brings to mind the TV show Scandal.

The term you want for our field is "Incident Response", and the practice of 1)preventing them and 2)handling them 3)learning from them is Resilience Engineering. It's about investigating air plane crashes, nuclear meltdowns, errors during surgery, etc, and learning how humans keep complex systems running.

I recommend "Behind Human Error" by David Woods as a great starter there. A key insight of this field is that incidents aren't just "some idiot didn't follow the safety checklist", but often the safety checklist itself will cause the issue; at some level the errors happen because of complicated interactions between the system and even the safety mechanisms.

An interesting tech industry related document is the STELLA report [1] from a few tech companies comparing notes on incidents.

[1] https://snafucatchers.github.io/

thaumaturgy
Several countries and lots of agencies use something called the Incident Command System (ICS). They use it to coordinate multi-agency response to things like explosions, toxic spills, massive wildfires, large-scale searches for missing persons, and more, all the way down to small and localized incidents.

ICS courses are free to anyone who wants them. You can get started with ICS-700 here: https://training.fema.gov/is/courseoverview.aspx?code=IS-700...

There are a few basic principles of ICS that should be useful to company incident response:

1. ICS defines specific roles and their responsibilities. In ICS, there is Planning, Logistics, Operations, Management and/or Coordinator, and Finance, among others. Each of these roles are defined ahead of time, and disaster response teams practice these roles regularly. Each role has defined ways of handing the role off to another person during a shift change and often includes specific forms that need to be filled out. This data collection is integral to being able to review the incident while it's happening, as well as after the fact for improving training.

2. ICS is scalable. For a very small incident, one person may be responsible for all roles. For a very large incident, response may be further subdivided into branches and divisions. This flexibility is an extremely important part of ICS, and it only works because everyone understands the different roles involved.

3. Under ICS, everyone has exactly one boss, supervisor, etc. that they report to. Any of you who have had to try to go spelunking through logs while multiple suits keep contacting you for updates already understands how important this is. This structure also helps to minimize miscommunication during an incident.

4. In the planning section specifically, there's a process called the "Planning P" that describes a lifecycle of information gathering, decision-making, and communication. It's pretty straightforward and it resolves a lot of common issues in incident response. This is covered in ICS-201: https://training.fema.gov/is/courseoverview.aspx?code=is-201

Companies developing their own incident response strategies will want to customize forms, data collection, roles, &etc., but the basic principles of ICS are an effective framework that should be adaptable to a wide variety of situations. Most companies on their worst day aren't dealing with an actual or potential loss of life; experienced ICS people can sleep-walk through a company's worst incident.

qbasic_forever
The climbing and mountaineering community is great about reviewing past accidents (especially deadly ones) to understand the root causes. Every year the American Alpine society produces a new volume of Accidents in North American Climbing and it's well worth a read: http://publications.americanalpineclub.org/about_the_acciden... Most things are human error like rappelling off the end of a rope, but there's usually a few really big team or trip planning failures like avalanches, etc.

There's a podcast they do related to it called The Sharp End and it's worth a listen too: https://www.thesharpendpodcast.com/

teachrdan
I can't recommend this series enough. Most accidents are described in plain language in a page or two with a concise analysis at the end. From one analysis of people hurt in an ice avalanche: "if it looks like it is going to fall it is probably going to fall."

You learn a lot of interesting facts you might never otherwise find out. In the above example, the avalanche was so powerful it generated an air blast in front of the debris that ejected sleeping campers out of their tents below.

The authors also have a low-key sense of humor and a subdued flair for language. This series is where I learned the phrase "injuries incompatible with life," as in, "Rescuers spotted the missing climbers from a helicopter the next day but saw that they had sustained injuries incompatible with life." It really puts a failed git rebase in perspective.

sjg007
Is the best idea to plan for a crisis before it happens? I guess it would be a process that you engage in because you won't necessarily be able to predict the crisis in advance otherwise you'd mitigate it.
veddox
Well, I think having contingency plans for various scenarios (and ideally practising those at regular intervals) can be a tremendous help when those scenarios actually take place. I know that NASA spent months doing full-scale simulations of every mission with astronauts and ground control, and that experience (especially of all the failure simulations) was invaluable for getting the actual missions right. I can echo this from personal experience: in a high-pressure situation, you function much better if you've thought out what to do ahead of time.

Of course, the glaring problem is that this preparation comes at a big opportunity cost - it takes a lot of time, may never be needed, and may in fact prepare for the wrong crisis. But I think in any case one ought to make sure that one has a communication and decision-making structure in place that can function smoothly in crises, regardless of what they are.

jkingsbery
This week's EconTalk talked about the general problem of responding to crises, and I found the guest's (and host's) take interesting (https://www.econtalk.org/megan-mcardle-on-catastrophes-and-t...). They talked about how often we respond to infrequent crises by trying to prevent them from happening again, but then that investment goes to waste because that crises doesn't happen again (or doesn't happen for a long time, after which the investment has depreciated). They both advocated for focusing on being more responsive to crises, since adaptability can help more generally across many types of crises.
myth_drannon
Volokalamsk Highway by Alexandr Bek https://www.goodreads.com/book/show/1769643.Volokolamsk_High... It's an old book, so your best chance is an ebook. http://ciml.250x.com/archive/literature/english/alexander_be...

It's about leadership during crises and it's based on real events. Telling the story of a small battalion stopping German army en route to Moscow in 1941. At some point it was a required reading in some military schools (like in Israel for example, maybe even now).

mronge
Here's a list of some of my favorite books dealing with crisis:

- In Thin Air - About a mountaineering expedition that turned into disaster on Mount Everest.

- Black Hawk Down - The story of hundreds of US special forces trapped in Mogadishu overnight after a mission went completely sideways

- Leadership in Turbulent Times - About different US presidents leading through crisis and how there is no one singular type of leadership

- The Hard Thing about Hard Things - Leading a startup on the verge of failure to an eventual massive acquisition

- The Sledge Patrol - How a small group, outgunned and out manned fought back Nazi invaders in Greenland

kevin_nisbet
Specific to books, "Failure Is Not an Option: Mission Control From Mercury to Apollo 13" was pretty good from Gene Kranz and as I recall covered many of the incidents in the US space program. I don't think I ever found a copy of the Chris Kraft one but heard it's pretty good as well.

And as others have suggested, reading flight accident reports, or watching the videos made off of them tends to be valuable.

Also, I think NASA published a bunch of research on human factors at one point, but it's been a long time since I've looked it up.

And last, specific to our industry, the SRE Books have a couple chapters on incident response: https://sre.google/books/

veddox
> Specific to books, "Failure Is Not an Option: Mission Control From Mercury to Apollo 13" was pretty good from Gene Kranz

Yes, absolutely fantastic! I've read a fair number of books on the space program, that one was probably my favourite.

villasv
Jared Diamond's Upheaval is a bit off-topic I think but I still recommend it. It starts talking about personal crisis to extrapolate into nation crisis, which are both extremes compared to company or team crisis.

Even if tangentially related, it's an interesting read if only for the historical content.

throwmeaway_66
I don't see it mentioned in the comments so far, but one very enlightening book on crisis and crisis response is "Thinking through crisis", by Amy L. Fraher. Recommended.
trop
Two more, by authors coincidentally with the same initials:

Thinking in an Emergency (2012) by Elaine Scarry -- She argues for the importance of planning and procedure. Examples include the Swiss shelter system (civil defense), CPR training, and compacts in rural Canada to deal with grain silo fires. She suggests that careful thought before the emergency is vital for "civilization", or at lest for democratic governance. As a converse, the country is destabilized when an opportunist leader comes along, cowboy-style, and says, "This is a crisis, and I'm going to shoot from the hip and fix it". Hence it's both a pragmatic study and a lucid work of political philosophy.

Command and Control (2014) by Eric Schlosser -- This is more of an anti-study: not what to do in an emergency, but the inevitable flaws in complex systems, the limited efficacy of administration, and the inherent failings of human effort. Sort of the mirror-book to Scarry's, but also utterly fascinating. It's a history of disasters of the U.S. nuclear weapons program, and in an effort as huge as that, the disasters certainly exist at scale. (Interestingly, Scarry's book centers on the importance of governance in a nuclear state, or perhaps that democratic governance is not compatible with being a nuclear power.)

[deleted]
(comment deleted)