> Then I realised that "I don't want the details" wasn't being dismissive. The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".
Something about this feels wrong:
- If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
- If you don't trust them completely, how can you know if "what happens next" is appropriate without knowing the details?
Isn't this just reinforcing the idea that leadership doesn't need to have their feet on the ground?
The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
> The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
These two statements are at odds. You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more.
It's not just about trust. It's about knowing what your team is up to so you can provide them the resources necessary while also working to remove any obstacles in their way so they can get shit done. That's what good management does, it's just rare to see in low-trust organizations that treat everyone like a cog to be micromanaged because no one trusts anyone to get shit done of their own accord for whatever reason.
We are arguing because there is an IT maturity line that goes from chaos to corporate and the more like a startup the more it will be like chaos (by design because maturity is too restrictive). So horses for courses. But as the team grows the same lessons get learned everywhere you bring in ex-bigco to implement changes to be more mature. That will include incident management process that will make this situaton routine and not worth blogging about. Probably in a mature process leaders already read the summary and or postmortem if they felt they needed to.
> These two statements are at odds. You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more.
But doesn't that same logic apply to whether they need to know the details?
> It's about knowing what your team is up to so you can provide them the resources necessary while also working to remove any obstacles in their way so they can get shit done.
That makes sense - I don't know if that's what the blog post was implying though. Ironically, this blog post could probably benefit from a few more details about what the meeting was supposed to achieve :)
> I believe you. I don’t need the details. Tell me what we're changing.
Yeah, for me in these situations it’s often about messaging to stakeholders, and so management in general feels like the problem has been addressed. I do think there’s one important thing missing from the final statement of the post, that he sort of covered briefly when talking about too much process, and that is what is the tradeoff/cost of implementing the change we’re making. Engineering is fundamentally about tradeoffs, and any significant proposed change requires some tradeoff and all too often I see management not asking about this explicitly. I’ve had enough once in a blue moon errors have retros, where the team decided it made no sense to make changes, and others, where the cost was high enough it was worth increasing workflow headaches. I think these tradeoffs being explicit are important at the director and above level, since there are potential real impacts to the team, and that is the level to deal with morale or headcount pressures due to changes to processes.
The meeting was about the SVP teaching the team that incidents happen, what's important is to prevent the same issue from occuring again.
The fact the author had never been in a call with the SVP implies incidents that happened before were explained in details, without much process change.
A perfectly reasonable explanation is that this SVP needs to explain in a different room what steps are being taken to prevent it from occurring again.
A lot of times, people just need a gentle nudge, or an official blessing from an authority figure, to go ahead and make the change that is necessary. You can tell people to manage up, to fully own their work and their process, etc. But a lot of humans don’t want to step on others’ toes or be perceived as acting out of line. A worker on the assembly line might know the best way to make the line run more smoothly but it’s easy to think that’s “not my job” or “above my pay grade”. Sometimes all that’s needed is to tell them, “Yeah, go ahead and do it.”
Getting that to happen more automatically is one way out of this situation, but that tends to cause problems of its own, with blurry lines of responsibility and authority. So it works best with relatively flat organization structures, which aren’t used much in large organizations.
> But a lot of humans don’t want to step on others’ toes or be perceived as acting out of line.
It's conditioning. In all organizations above a certain size (> 100 people), stepping out of line receives an official reprimand. Someone always complains. Receive enough reprimands and the person is terminated. People learn to stay in line because they've been punished whenever they step out of it.
Unless the organization really truly rewards that behavior. Which is rare.
The senior-most people (VPs, founders) often say they want it. But it’s the middle managers who will stab you in the front in revenge, while staring you in the eye. And if the VP doesn’t step in, they are endorsing the outcome.
The legend of Richard Montañez and inventing Flaming Hot Cheetos itself is not supported by facts, but the underlying story of someone low level who jumped the chain of command and contacted executives at the company seems to hold up. Which means there's at least one organization out there for which it's true.
First of all, he didn't say the executive "trusted them completely". Those are your words.
He said "The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".".
That can be true even if they don't trust them completely. Let's say he trusted them 80% and not 100%. Isn't that pretty much quite trusting anyway?
Besides, the executive can trust them even 100%, but still want them to give him the general picture. Among other things, it's his job to know and be able to report when asked.
The VP also needs to know if the fix will require some sort of resource reallocation, or if the timeline is long so they can deal with the other stakeholders, or whatever. These are all forward looking things that the details influence but the details aren't what matter.
Asking what the OP was going to change is not an automatic nod of approval for his approach. I’ve been part of some large interventions spanning multiple quarters, that started from a simple conversation with an exec just like this. There are downstream effects to these decisions that affect many stakeholders and departments - so they cannot be made in a vacuum or delegated completely (I trust you, go ahead and fix whatever). Depends on the severity and scope of the issue of course. A simple deployment gone wrong is different from a fundamental issue with the architecture that is straining progress.
As an example, a major rehaul of a core system could push back product (new feature dev) timelines, that can affect customers and their releases, bottomline of the company, reputation, tax repercussions, if it’s an American public company then it’ll affect SOX compliance & audits, could lead to spicy investor/analyst conference calls, and so on.
In a large enough organization it is simply impossible for executives to be on the ground floor with everyone else. And not only is it impossible, it is essentially a waste of their time. They could maybe be maximally involved in only a few projects at a time.
I think its entirely contextual as to what the size of the org is and the scope of the projects. At a startup an "executive" or VP would legitimately be able to be working side by side with everyone else, or hearing about the ground level operations, but just by nature of the role being SVP I imagine this is a large enough company that this is no longer the case. At a certain point that becomes somebody else's job that is much closer to the team at hand.
The role entirely morphs and their job becomes to enable teams, get blockers out of the way, fast track or assist things like permissions/processes, shoot down ideas that legitimately don't add business value or are not worth wasting existing engineering effort on.
And I imagine in this scenario the point wasn't for the executive to literally hear what happens next, it was to ensure the proper steps are being taken and make sure "somebody" is handling the what happens next part of things, and so they can report up through the chain or delegate tasks for exactly that "what happens next" task.
> In a large enough organization it is simply impossible for executives to be on the ground floor with everyone else. And not only is it impossible, it is essentially a waste of their time. They could maybe be maximally involved in only a few projects at a time.
This is a very strong argument that no organization should be allowed to get that big.
I get the sentiment, but this is the reason we have airplanes, MRI machines, and computers that fit in our pockets. Advanced technology simply isn't possible in an economy of only ~100 person organizations. And I say this as someone with a deep sense of unease with modernity and all of the diseases of civilization. We somehow have managed to create systems that support literally billions of humans and allow many of them to have a quality of life undreamed of even a few hundred years ago. I dislike big orgs as much as they next guy but we need to understand why they exist if we want to criticize and reform them.
Big organizations don't enable greater technological progress, which is why companies do 98% of their innovation before they get huge, and get bad at innovating once they are.
Big organizations enable scaling and associated economies of scale, so it's possible for Apple to sell millions of iPhones cheaper than if they were a smaller company. I don't think that tradeoff is worth it for any individual consumer (pay a little less to be much more abused by the company), and I don't think it's worth it for society.
Most of the big corporations in America aren't providing things as useful as MRI machines, they're providing net negative garbage like social media and advertising. Big corporations like United Healthcare Group are inflating the cost of medical care rather than providing economies of scale and reduced cost for their clients.
The US desperately needs to bring back antitrust and start breaking these massive companies into much smaller pieces.
Ironically, those same systems that enabled high QoL are also the ones involved in killing us, whether that's carcinogens everywhere, climate change, resistant bacteria, or even just plain old nukes (or soon drone swarms).
I wouldn't be alive if not for advanced medicine, but if the structures that saved my life are the same ones that constitute our own Great Filter, my life was most definitely not worth the trade. There are too many other people I love to think that.
There may be something common across many of the ground floor parts, such that even just knowing what's going on in a fraction of them, they are much further in their vibe-calibration of what ground floor is like than if they know zero.
The best leaders know what they need to pay attention to. You cannot know everything. You can know any one thing (or perhaps several), but there are more things to know than you will have time to learn in your lifetime. The question is what is important for you to know, and what you leave to somebody else.
You will have to trust someone will figure out the details and come up with a reasonable solution. Sometimes you need to know that solution, sometimes in detail, sometimes just enough to know that they solved it. Sometimes you are the person who needs to figure out the details and make the solution. Knowing which you are is important.
“Trust” is usually bounded. I trust my elementary aged son to get dressed for school without supervision. That doesn’t mean I trust him to start a campfire without supervision. A chef might trust that their new employee is a hard worker and well intentioned. That doesn’t mean they also trust them to create and serve original dishes.
Here you are recasting “I trust that, despite the bad outcome, everyone’s actions here were reasonable” as “I trust that you are going to solve this fully without my guidance”. The fact that the word “trust” appears in both of these statements does not make them equivalent. Trusting that people are capable and well intentioned does not mean trusting that they will accomplish the same thing without you.
That’s not even true though, leadership is coordination, even if you trust someone else to do the right thing wrt their own problems that still may be globally suboptimal wrt the goals of the overall team / project.
I don't read the article this way. It's pretty sane for an exec to draw a line where the team's autonomy meets leadership's responsibility for structure and incentives around which said team needs to organize.
"You did the right things and met the goals we set, but we hit a problem and we all agree it shouldn't happen again. Let's talk about what needs to change to address this."
You can trust someone to know the details and reasons for why something happened, and still want to provide the specific direction of "what are you doing to make sure it doesn't happen again". The organizational default is often to explain a fault (or worse, focus on blame) without specifically working to find a process improvement to prevent it from happening again.
Plan is the key here. In reality the SVP is hoping that they can make the decision to go with your solution.
Q: What are you going to do about missing this bug?
A1: "I'm going emphasize "BE" in Hamlets' "to BE or not to BE". - You should be fired for this, but that is not the planned decision.
A2: "I'm going to spend $$$$ on some tool". - This tool might solve the problem, but the budget doesn't allow for it and you should know that: I'm forced to make the decision to override you but I didn't plan on making that decision.
A3: "I'm going to [summarize something reasonable]" - good, you are doing what I expected.
Note that A3 doesn't have to work, it just has to be reasonable. If it works great, if not I will ask again next time and I expect you have different answer. Not working at all better be rare though since each not working loses confidence, but fortunately this is rare. However often working is sometimes all you can get (the next bug might be unrelated - no single answer could solve both). The idea is we repeat this for each bug until the rate of missed bugs is acceptably low.
> The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
If I have a handle on things from bottom-up to top-down, I don't need them to rehash weed-level technical and specific details. I need them to tell me what we're going to change, and it needs to be a BIG systemic change, not minutiae, which is where many engineers thrive.
If execs demand a big change every time something goes wrong, they will demoralize everyone working for them, and probably waste a lot of money.
Most of the time, operational failures are either the result of a new feature being sent off without enough testing, or a dozen small failures lining up to remove resilience. Big changes without well-thought out plans primarily serve to stoke executive egos.
> If I have a handle on things from bottom-up to top-down, I don't need them to rehash weed-level technical and specific details.
I should have been clearer.
When I said that good leaders pay attention to things from the bottom-up, that doesn't mean they always know everything needed for any discussion. It just means they are able and keen to understand the low-level details when they matter, because they routinely look at such details in the projects that they're involved in.
Of course it's possible that this story is just told badly and the SVP already knew the technical details, in which case the blog post is a bit misleading.
> I need them to tell me what we're going to change, and it needs to be a BIG systemic change, not minutiae, which is where many engineers thrive.
How can one know if the engineers' proposed "BIG systemic change" is appropriate and complete, without knowing the details of the problem it is addressing?
> Of course it's possible that this story is just told badly and the SVP already knew the technical details, in which case the blog post is a bit misleading.
I don't think it's necessarily misleading; it's just reminding folks that senior leaders have a different perspective than those below them and tailoring your message to the audience is important.
In this case of this, it was a reminder that while it's easy to believe the SVP is being dismissive, that's not the case at all. Instead, they want systems-oriented decisioning, not detail-oriented.
If anything, this was the SVP helping to grow the leaders under him by forcing them to think at the levels required to interact w/the rest of the organization.
> How can you know if their "big change" is appropriate and good without knowing the details of the problem the change is addressing?
I'm a Senior Director leading three teams of engineering and science resources. I hired smart and capable people. Why should I, as a SrD be a technical bottleneck?
> How can one know if the engineers' proposed "BIG systemic change" is appropriate and complete, without knowing the details of the problem it is addressing?
That’s the trust thing. Maybe the SVP trusts the people in his organization are capable of making the appropriate change without his supervision and what he’s doing is coaching them on how to do an incident postmortem properly.
Of course as you called out, the SVP might already have all the necessary details to do more than that. But simply pushing people in the right direction is a big piece of leadership’s job, at least when they have people they trust.
If I trust you know what happened, and trust you to know what to do so it doesn’t happen again, why do I need to know? And how will I know (and endorse) your decision if I don’t know what happened?
> - If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
You might need to know timescales so that you can inform upwards/outwards/other. You might need to make sure other resources that might be necessary are available. The planned "what next" and the proposed timescales might have impacts elsewhere that they haven't realised (and maybe even until now had no reason to be aware of). If they tell you the what next and there are no such complications, then they can carry on, otherwise there might need to be some changes to the plan or changes to other plans to allow time/resources to be available for this plan or just to stop potential near future conflicts.
One reason is prioritization. It’s enough to know what the effects of the failure were, the likely cost of what needs to change, and what areas and processes the changes will affect, to decide if and when this should happen.
Leadership still needs to understand these aspects on a conceptual, topical, and cost level to decide on the resource allocation (including time and schedule). They don’t need to understand the technical details, unless they are a technical leader with higher relevant technical competence than their reports.
They trust your ability to do a root cause analysis. They don’t trust that you feel empowered to do anything about it, so they are verifying that you actually do.
Also, not every IC or EM has the same skills and experience. There may be certain categories of work that you can be 100% hands-off and trust them to just do it, and others where you really need to understand bottom-up. Part of being an eng leader is to know when to insert yourself and when to "trust the process".
Leadership still holds engineers accountable. And for that, they need to know the plan. Leaders are also held accountable, so again, they need to know the plan.
Not wanting to know the details is a red flag. Sometimes the process of expressing the details can reveal things that were unknown by one or both parties, leading to improved processes. The only acceptable exception would be "Let me explain. No, there is too much. Let me sum up."
The details are sometimes important and sometimes irrelevant. If you insist on skipping them all the time, you will often be right, but when the day comes that you are wrong, your unwillingness to listen to the details will be the reason the problem can't get fixed.
To me trust means youre willing to treat people like a black box. If the detail about the outputs is another i/o or something about an output that would appear on a spec sheet, it should be included.
Inner details shouldn't matter. Thats what makes a good summary
Another explanation is that not wanting the details of operations is based on complete lack of trust, not in the their competence but in the way they would describe things. The boss doesn't want the details because they are sure the details would make it seem like the underling did the right thing and moreover, monitoring that detail would just give the underling leverage over them.
The boss wants to know about changes to avoid this but whatever it is, they'll push back on its extent and won't want to hear about details then either.
The devs job is to impliment and advise how to attain the owners goals and priorities, not to set goals or priorities except at layers below, and in the service of, the c suite's directives.
I'm not saying c suite are unquestionable gods, I'm saying that there is no such trust-or-not dichotomy. It's 2 different things.
No matter how ignorant I am in some domain and no matter how knowlegeable some expert is I hire to do something for me in that domain, they can only tell me how to get what I want, or what's the closest that is humanly possible, or the costs of various conflicting priorities and compromises. They can't tell me what to want.
Maybe I DO actually want to burn my whole budget and 5 years of time on some facet that they and most people would say is not important and not worth it to the point of being irrational. Their job is to inform me not to decide for me. I can trust them to inform me honestly and with good judgment and deep knowledge, and then to impliment what I decide also honestly and with competent skill. And yet I still need to be the one who is informed, and then makes a decision and issues a directive.
It's 2 different things.
It's like Steve Jobs being totally unreasonable about tiny details of fit & finish in the hardware products. It makes no sense (to most people) to have custom ethernet jacks manufactured (back when it wasn't so effortless and flexible) just to get them a certain color or material feel, when off the shelf jacks already exist from multiple sources that work just fine for a 10th or 100th the cost, while already meeting a stack of compatibility standards and even safety regulations. No trustworthy engineer would do that. They would actually see it as violating that trust.
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
I work in a competent team and leadership is required.
Yup, several things are wrong with it even though it may be a good idea as a good shortcut on the part of the SVP
But it should be used and phrased better.
Used better:trust and only sometimes ask for the detials, like cutting the just-shuffled deck in a card game, sometimes the player to the dealer's right will just tap the deck instead of cutting it, in effect saying "I trust that shuffle". Sometimes, certainly enough to both keep the SVP well-grounded in the details, and to keep the team knowing the SVP is both watching and cares about the details, the SVP MUST ask for the details. Trust and verify.
In terms of phrasing, "I don't want the details" is all about the SVP sending the message s/he doesn't want to be bothered. The phrasing should be: "I trust you on the details on this one; let's go straight to what do we do to manage the next one of these?".
Notice leading with "I trust you on this", not "don't waste my time".
I would agree with you, this just sets up a scenario where the SVP positions themselves as a leader, without having to understand something technical, and without making the decisions on how to change things. It isnt about trust, it is about performing accountability so the SVP can say they did their job.
I think the author of the article is going out of their way to be charitable to the SVP for no reason. That meeting was a reassertion of power, not a wise move made by a sage.
> "What happens when requirements change inside the launch window?"
> "The alert fired but..."
In the running example I think it is clear that you could stop at any point and address the problem from that point rather than continuing. Questions such as "Why do we permit last minute changes at all?" are given no space.
In a complex system sometimes there is no root cause (cf. aviation crash investigations and the Swiss Cheese nature of risk). By allowing one person to shoulder the entire context of the incident and for that person be the one to suggest lasting corrective actions it leaves little need for the VP to even be involved. How would they know if the proposed corrections are worth the time and money to implement? I don't see any leadership in the example described in the article - I just see a knee-jerk reaction rather than any kind of proactive reflection.
> "I thought we fixed this last time!?"
> "This time was different in a way that our corrective actions didn't address."
I sympathise with this, and a lot of teams do operate that way, but I fundamentally disagree with it.
Part of the reason Amazon's CoE-driven engineer culture had such operation excellence is that the responsibility was driven the whole way up the management chain. If your manager didn't dig down to the root cause of a sev2, the director was damn well going to, and if the director didn't, Andy Jassy was going to call them on it.
That sounds like a lot of busy work, and a bunch of it undoubtedly is, but the flip side of it is that the actual root causes were addressed, and if the root cause needed serious cross-functional resources to address, the escalation would get you what you needed. Need approval to bounce 3 months of work off your roadmap to address? You have a VP on the line to make that call. Need a couple of other teams to fix their shit? Here's a principal engineer who outranks their directors to drive the work. And so on...
Quoting myself here: Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org. I don't want to name bad orgs but what org are you talking about?
I'm here for the sentiment expressed by the SVP in this article: "I already believe we reached this point through rational choices. Let's talk about how to change the system so it doesn't happen again."
I do think his choice of words was a bit suboptimal. But, it got the point across.
> I do think his choice of words was a bit suboptimal. But, it got the point across.
As I've gotten older, I've grown to appreciate people who are willing to be a little rude in order to avoid wasting people's time or giving them false impressions.
For example, when recruiters reach out to me on LinkedIn now, I almost always say something like "I don't want to waste your time or mine: what is the salary range?" It might be considered a bit "rude" to immediately jump to money, but I figure that that's less rude than wasting 30-90 minutes of time for a job that I have no chance of accepting.
I also try and be a little upfront if I don't like food someone has prepared. I try not to be too much of a dick, but if someone prepared something that I don't like and they ask my opinion, I will say that I wasn't a huge fan. Again, maybe a bit rude, but I also don't want to mislead people.
It kind of goes against my gut; I, like most people, don't really want to make people feel bad and I certainly am not immune to people-pleasing, but I've tried to fight that for the greater cause of being more-honest and upfront with how I speak.
I actually think it’s more about the emotional maturity to cut to the chase and address someone’s actual concerns.
I’m a big fan of bluntness / honest feedback, but it can still be done very rudely when inappropriate
like the svp observed that the ic was saying the details, not because the ic needed to share them, but because the ic was anxious that the svp didn’t view them as competent. So the svp cut to the chase and addressed that directly
For your food example, a host might ask if you like the food - not because they care about their cooking skills - but just want to make sure you had a good time. So, saying you don’t like the food and giving cooking suggestions would need to be accompanied by some reassurance that you still enjoyed their company or beautiful home or something. Otherwise it’s still rude
I frame this differently - it’s about getting out of your own head and think about what’s useful to the other person. And making this a habit so it’s easy.
Providing feedback on the meal is useful for their next guest, if they are interested in improving. If they aren’t, just say it’s fine and move on.
Shifting focus outside of yourself is covered in a lot of areas but Never Split The Difference stuck with me, written by a hostage negotiator.
Talking about changing the system "so it doesn't happen again" can only lead to more "rational choices". This is a thinly veiled tantrum spoken in a calm voice.
If you want to reduce incidents, you need to lean into the problems instead of dismissing them or delegating even more all over again. Doubling down on trying to keep your hands clean is shit leadership.
The basis of this blog post is delusional. The language from the SVP is the desire to fast-forward past the thought process that comes with listening to "the story". This could only lead to more problems. They're basically admitting they weren't going to listen in the first place. "The story" doesn't always lead to blame game, and sometimes the changes needed are the context in which rational and trustworthy people are working in. That is not a change to the system, but to the organization.
I think there's something valuable in your take. The idea that technical failures and process failures are parallel tracks is not new.
Certainly not the kind of thing where an exec needs to pull out the "I don't want problems, I want solutions" card to illuminate that they are only valuable to the process domain resolution, not the technical one.
The accommodating reader can read that interaction as entirely because of mismatched insight into the issue. I'm willing to budge that maybe too much time in the agile dens does really does necessitate the conversational roundabouts. It's taught people to taboo process they don't see the immediate need for.
The incentive is not causing an even bigger mess and keeping their role. Contrary to popular belief, executives really are held accountable for outcomes.
It's a question of how much failure is visible and whether anyone else cares, including themselves. If they were only ever motivated by money, they might already have enough since so long ago that now they really only care about harsher legal action that money can't fix. Getting fired isn't a motivator anymore. That's a bad fit for any organization. Someone else is also dropping the ball there.
> "I don't want the details. I want to know what we're changing."
This really feels like the sort of senior manager who would say "oh, this/these team/s screwed up, so here's the plan they came up with", talking about their team/s.
Speaking as a C-suite guy, "they" is about as contrary as I can think of what I expect a leader to do: it's "we", and your job is to take responsibility, both in the problems (big problems need to be explained all the way up and sometimes externally) and in the solutions (which to be comfortable with, you obviously need a sufficiently thorough understanding of the problem).
Because up the chain, I will take responsibility for it.
> To drive change in your organization, don't ask "why did this happen?".
> Instead, ask:
> What are we changing so that the same class of failure is less likely next time?
Why are these framed as being almost mutually exclusive? I don't get it. "How/why did this happen?" informs how you answer "What are we changing?" Even the following section, "Reasonable people", functions in this way with a "Why did this happen?" followed by "What are we changing?".
Just feels like this article has some internal conflict with itself in order to achieve a "don't do that, do this" type of style.
Because answering the “why” feels like you’ve achieved something. You all blame Gary, pat yourselves on the back and then go home satisfied.
It’s like when cloudflare does a blog post saying they were out for three days and it was because they let someone slopcode the DNS. How has that achieved anything? It will go down again the following week. The customers want to know how you will prevent it happening again, they don’t really care how entertaining your blog is.
I don't get it neither. It's important for context to document "why something happened" alongside action items to prevent it from happening again. This is part of history that will be extremely valuable as company knowledge on why processes exists in the first place.
Otherwise, what will be the discussion when it's time to question it later?
One thing that's often left out that tends to grind organizations to a halt is that continuous improvement processes tend to be additive.
You always add rules, and alerts, and so on. You need to revisit existing processes to and see what can be removed or replaced.
My other nit with these is a term that's abused a lot "Root Cause", most complex issues have many contributing factors, not a single root cause, and if you force the teams to find one, they will.
I dislike RCA (Root Cause Analysis) it puts you in the wrong mindset, CFA would be better (Contributing Factor Analysis).
I recently had a vendor CAPA rejected because the root cause was listed as "data entry error." Oh no, the root cause can't be data entry human error! It has to be that the deadline was too tight, or they were rushing, or they were distracted etc etc. I ask the vendor, were you distracted? No. Were you rushing? No. Just a simple typo. Update root cause to typo. Oh no the root cause can't be a typo it has to be the free text field in the form design that allows for typos. It's a process error that there's not an automated QC to catch these typos! Etc etc.
Yeah I've seen this a lot. People freaking about issues that have small or moderate impact on deadlines, and very little to no revenue impact. And suddenly you're moving 50% slower because you have weeks of review processes to make sure nothing is missed.
There's still some problem with that approach. OK, I will tell what I'm going change to prevent recurrence of the issue. How does that makes sense to the audience? Unless they just want hear that "some" change will be there and don't care about how that change would make any sense.
It boils down to what exactly is the ownership or accountability of your SVP around the issue. Why are they even bothered to ensure that there will be some change? If they are responsible for ensuring that the issue doesn't happen again, then they do need to say "I want the details".
I think there is a subtle additional level of trust that the author didn't call out. As the author noted, the SVP trusted that everyone involved in the issue responded rationally and professionally. But they left out the part where the SVP also trusted that the author and their engineering counterpart had already analyzed what happened and how.
The SVP doesn't have time to dig into the "who did what and when" timeline of a problem. They do need to communicate that they're willing to invest resources in preventing recurrence. Was the team burned out by noisy alarms? Then the SVP needs to hear "We're spending time improving automation to reduce alarms, and tuning remaining alarms to make them less noisy. This will take engineering time away from feature work." Or maybe the system had a single point of failure that broke, and the SVP needs to hear "We're redoing X to eliminate a single point of failure, but it will increase our infrastructure budget by $Y".
The SVP is inviting a higher level discussion around what resources need to be consumed in order to be sure this doesn't happen again. And they're opening the door to defending that use of resources.
What you describe is exactly what a 'leader' is supposed to 'see'. When to know when details are needed, and when to trust their delegate. The trust is also balanced: if what you say does not align with what the leader is 'seeing', then trust falls apart.
Everyone has some measure of the 'big picture' and the person at top is supposed to see the biggest, final picture. Where everyone and their processes fit, within the big picture. Delegation could be perfectly aligned (and trust would be easy) if not for quality:
Quality determines how well the relationship works between leader and delegate.
The amount of quality is determined by the amount of need, and urgency available. The relationship (and communication) evolves over time as quality is determined by leaders.
Regardless, there's always going to be difficulty because there's never an ideal system where quality is perfectly measured. That's why communication with people, as people, is also very important. When we perceive disrespect: communication shuts down. Humility must be respected.
If you can't communicate, there's definitely no measure of quality available for the relationship between leader and delegate. If communication is there (and communication should be measured as transmission of real information), then trust is more easily measured.
Once you know how much you trust something, then you know how much detail you need when you 'need to get something done'. Because you know the big picture, and where this thing you need fits. And how well it works.
OP is already at the director level. The further you go up the management chain, the less people are involved in process or technical decision making -- they are primarily concerned with organizational decision making. It sounds like the issue was something that was fully within the management scope of lower levels of management here:
> Something had gone wrong that shouldn't have.
The SVP is trying to determine what change in management is taking place, not what technical change is taking place.
There are a set of principles at play: used by leaders and engineers, when trying to 'do work'.
When you need to get something done, unless you can build the entire thing, you have to trust someone to help you. There's very few people who choose to live a life where they are 100% self-sufficient, living-off-the-land.
Leaders have to trust people, engineers have to trust components.
It's not just people. Shuffling people around is not fix everything. People, process and technology - any of them could hold the key for a solution. That would mean, you do need to understand whether is is people issue, process issue or is it tech issue. That requires listening to details.
"Okay, everyone is overworked, and we need you to hire more people for this team. We have too much tech debt, and we can't predict which bit of it will explode next week. The change needs to start with the attitudes of our executive leadership."
I suspect the author still misunderstands the SVP. If I were the SVP of engineering saying this to someone, especially someone who is in a non-core engineering role like product, I probably recognize that they have a tendency to verbally spew and I want them to focus on the risk mitigation and response plan. I’ve already talked to my engineering leader because they informed me immediately, and I’ve already read the incident details.
But also the post reads like it comes from an inexperienced/immature service org, so maybe they’re just sorting their operational response process out.
In order to figure out a good solution, you have to have mastery over the problem. That takes a bunch of work.
> Understanding an issue is not the same as fixing it. A good explanation can make things worse. Once everyone agrees that the behaviour was reasonable, the urgency to change anything disappears.
This is a non-sequitor. We're not here to evaluate whether the behavior was reasonable, we're here to figure out how to avoid bad outcomes. If all the behavior was reasonable, then the problem lies somewhere else, and that's an important finding when crafting a proper solution.
The author is literally saying that asking why is the wrong question, and instead asking "What will we change?" is the right question. I get where this comes from, but you can't really answer what will change until you deeply understand why it happened.
> The SVP wasn't interested in understanding how the issue happened or who was involved. They didn't want to be convinced that everyone involved behaved reasonably. That's a baseline expectation.
Their question became:
“Given that reasonable people produced this outcome, what needs to change?”
Of course you have to know the problem, but he trust and excpect a sulution from them. So this does just work this way, if you can trust the people. If not, then the top person has to check the solution for himself.
A good explanation is not a fix but is a good starting point to a good diagnosis which is the foundation of a good fix.
The issue is when the calls and urgency are raised before making some mental space to even verify that there is enough clarity on what happen and these interrupt the clarification and remediation planning process.
Saying you shouldn't ask "Why did this happen?" and you should ask "What we're changing?" instead is wanting to jump the gun and pretend a required step doesn't exist.
It does exists, is just the executive anxiety reasonable in terms of wanting to be protective of the business but as protective as a family member of a patient wanting a good outcome from the surgeon during the surgery.
Only worst because the surgeon's boss is this anxious family member, and she has a laboral gun pointing to the doctor's neck.
Yeah, I disagree with SVP's approach in this story. First, get the current issue understood, resolved and remedied. Then, thank the team for their good work in fixing this issue, and do a post-mortem to see what process improvement could have prevented the issue from rising in the first place, or at least detected and mitigated it more quickly.
"Okay, then I'm going to rewomble the dinglehop, which should un-galvatrate the percapitator." Why is the SVP in the meeting at all? If they don't want the details, they can't contribute to the change, or even evaluate whether the change makes sense at all. The SVP could be completely removed from this situation and the outcome would be exactly the same. Which shouldn't be surprising, since that's almost always true of all executives anywhere.
> Everyone nods their head, says "that makes sense", and we all move on with our day.
Why? Why do you need to be intimidated into actually making effective change? Why is everyone walking around with their hands tied, unable or afraid to enact change? Probably because of the terrible leadership at this company.
>> "We missed it because Alice was on holiday and Bob thought the Widgets team owned it".
> Okay. How do we make ownership unambiguous when someone is unavailable?
You cannot take this step without details. That Bob-Alice story is the details. You do actually need to know that story before you can take the step of knowing that "unambiguous ownership" would be a worthwhile change to make. What exactly is this SVP doing again?
This entire post could be summarized as "executives are useless" and "do the 5-whys", both of which everyone already knows.
It's crazy how far i had to scroll to find someone saying exactly what i was thinking - how come AI sloppy writing like this is showing up on Hn front page?
The post-mortem process can cover all of these things:
- what happened
- impact
- root causes
- what will be changed (commitments)
You can't really talk about what to change w/o talking about root causes, and you can't talk about that without talking about what happened. You also can't consider the costs of commitments to be made w/o discussing actual and potential impact because resources are limited and the cost of opportunity is real.
It's fine for executives to get only a summary of impact + commitments. But the whole post-mortem process has to be followed, and some people have to be aware of all the details.
I can't help but notice that a climate of fear around being deemed non-essential or otherwise getting on an authority figure's bad side tends to create significant mismatches in priorities and breakdowns of trust that could be upstream of significant organizational inefficiency in a variety of situations. Perhaps solutions to meta-problems of this nature might be high-leverage, as the thought leaders like to say
I can't be the only person who thought this is a highly intelligent person being inadvertently gaslit by the SVP into thinking the exec is right to be dismissive with "I don't want the details" because introspection is common in intelligent people.
Had someone said that to me I'd have walked away after saying "fine I'll sort it'.
This illustrates that "trust" to do the job, and since they didn't want the details of the problem, they therefore don't need the specific solution description. This kind of response is also the same level of respect, it's either seen as trust in ability, or just downright disdain with plausible deniability for being rude.
Action items need an a forcing feature (like an SLO) to insure they are picked up. And someone needs to follow up when the item isn't handled timely. It should show up in team performance metrics and guide changes to processes to insure they get done.
The important part is the action items are assigned an owner and executed.
I’ve done plenty of postmortems and many meetings where we go through the motions but “leadership” does not assign and address follow-ups. Most organizations pay lip service to reliability.
Actionable items go onto the backlog. The backlog grows from the top.
As a stopgap, a manual pre-flight check is added. The number of manual pre-flight checks becomes a ridiculous overhead but it gives a clear person to blame when something goes wrong.
Some of the best life advice I learned during conscription as a squad leader: do not explain if an explanation is not asked for. If you (or, importantly, your subordinates!) mess something up and are confronted by a superior, the only correct reply is "Yes sir I screwed up sir I’m sorry sir it won’t happen again".
That could make sense in the military. It could make sense for an offshore IT org whose role is to provide warm-bodied yes-men, or similar situations. But it’s catastrophic for a creative organization that’s actually doing anything other than CRUD apps or something equally mechanical, because it means management can’t possibly understand the systems that the company’s income depends on, which means they can’t possibly manage effectively.
It’s still up to them to ask. If they want to know the whole causal chain from distal to proximal, so be it. And if they never ask, I doubt it would help at all trying to explain anyway – they’d just zone out – and might be a good idea to start looking for a new job.
In any case, the actually important question "How to make sure this won’t happen again?" would presumably include the relevant parts of what "this" is as well. But in a more actionable frame.
I think the problem is when the execs aren’t looking at change-for-the-better. Instead they’re looking to appoint blame. Moving to a position of assumed competency in your staff and taking the perspective of how-to-improve-next-time is a real step forward in maturity. My guess is it’s rare and blame is easier: most management doesn’t reach this level, instead it’s immature.
I think that's a problem domain in itself. Identify when failures that occur even when skilled people doing their jobs properly, adjust the environment so that they no longer happen.
It is understandably a completely different task compared to what those skilled people are specialised to do. You probably need a dedicated role to do it.
Perhaps you could call them a manager. Their job is to see the multiple parts of the system. They should ask for the Details of what happened so they can determine why the problem occurred.
Consider one of the problems listed in the article
>"The alert fired, but the on-call engineer had already dealt with twenty low-value alerts that evening".
The engineer can say they were busy, they didn't see the alert, that they are swamped with things they think are low-value. Someone else can say the alert fired. Each person involved may have their own perspective, with different ideas as to what the problem actually is.
It's easy when you see problem described in terms of what the solution is. Someone needs to figure that out, to do that they need the details.
An SVP of engineering should care about the details, this is the essence of leadership. When you don’t care about the details you are just a manager and worthless i.e. you should be replaced with a leader.
Lots of folks are commenting on the internal dynamics, but the other part that didn't resonate with me was actually the what happens next part. This may be an overreaction on my part, stemming from watching lots of engineers suggest expensive solutions to relatively minor issues. And it's a mistake I've made myself before as well, I started in telecom with very strict standards for availability, so it's hard to beat out of me the desire to look through the most minor alarm for a potential problem brewing or get to a root cause of the most minor incidents.
And maybe this is just a part that the article dodges, but for something described as non-catastrophic, I would suggest that there shouldn't be a presumption of changes. In my view there should be an assessment of the risk of re-occurrence, and if it was a near miss, what is the risk had it not been a near miss. And then even should changes be part of the outcomes, how expensive are those to implement compared to whatever the issue was, and if they're too expensive compared to the risk, don't put resources into it.
Yes, if a director and SVP are discussing the technical details of an incident, I would be concerned -- they should be concerned with putting the right people and processes in place to make those decisions.
When "Something had gone wrong that shouldn't have" bubbles up that far, you have an organizational issue, not a technical issue.
If the person who is charged with resolving the issue is capable, there's no point in a senior manager knowing the details outside of professional curiosity.
“If your corrective action depends on people remembering a conversation from six months ago, you don't have a corrective action. You have organizational folklore.”
I think i’m going to call our ticket support system “folklore” from now on. It sure is used that way.
This is a really really smart SVP. The answer to “how do we prevent this in the future” is to identify decisions that reduce that problem from happening again. This doesn’t have to be perfect. You can say “we are going to do X next time” and also say “but we don’t know how far X will work”. When the failure happens again, you can retire process X and move on to Y. The aim is to make a series of decisions that eventually get to the heart of the issue.
Everything else is a conversation that takes up space on a post-mortem or runbook.
The assumption that something always has to change is the culture that leads startups to knee jerk their way into miles of red tape and performative bureaucracy
So in summary, instead of details propose how to change the system so that instead of hearing perfectly reasonable details about why the current system failed, go straight for changing the systems to prevent the chance of failure again.
I like this plan of action, it removes focus from what happened in the past to how can we prevent it from happening in the future.
There's a reason 5 Why's Analysis is a thing. If you just say "here's a bunch of stuff we're changing," but you haven't said enough about the root cause, then no one will understand whether we're doing enough (or too much; or the wrong things). I like the framing of being empathetic - everyone is smart, doing the best they can with their knowledge and experience at the time - but sometimes we didn't do all the right things, and getting feedback from more senior engineers helps drive that improvement.
> There's a reason 5 Why's Analysis is a thing. If you just say "here's a bunch of stuff we're changing," but you haven't said enough about the root cause, then no one will understand whether we're doing enough (or too much; or the wrong things).
No, that's not the point at all.
Root cause analysis and context and tradeoff analysis and plans are still critical requirements.
But not everyone needs to do root cause analysis and evaluate tradeoffs and review plans. Some stakeholders don't need any of that. They pay you to care, so that they don't need to.
What these stakeholders care about is whether you can fix things, how are you planning on how to fix things, how many resources you need to allocate to fix things, and when are things fixed.
Different audiences require different messages because they have different concerns and responsibilities.
Does a project manager need to know the failure rate of a DSL connection of a customer that reported an outage? You might, but does the project manager need to?
I've never seen an incident report or postmortem that didn't include action items to cover the core issue noted in this writing: "can we / how do we prevent this failure mode from recurring?"
I'm not saying these moments of discovery / epiphany aren't valuable, they are and this retelling is enjoyably written.
I am saying that action items borne of incidents should be de rigueur.
Organizations don’t choose whether or not to ask questions. People do. The behavior of an organization is the sum of the actions of the people in it. If you are a member of an organization, and you ask ‘how do we prevent this from happening again?’ Guess what? The organization just learned how to ask that question.
Now it just needs to learn how to answer it, and act on the answer.. but step one seems solvable.
Interesting. At our org we always do both. Root cause, what are we changing. I assumed that was standard practice.
If you just explain what happened and why, that's fine but...how are you going to make sure it doesn't happen again?
The problem I found more vexing as a manager was: how can you prevent this KIND of problem from occurring?
I'd have someone on my team make a technical error and xyz wouldn't work. Wed talk thru it, and they wouldn't make that exact mistake again. But there's literally 10k things that can go wrong in our system, so then a related mistake would happen later.
What they needed was improved pattern recognition vs if this / then that which comes from post mortems.
This reads like a LinkedIn post. I know kagi translate[1] supports "LinkedIn" language - could someone with an account there copy that article and translate from "LinkedIn" to "English"? I wonder if we'd uncover some simpler meaning this way.
Edit: I was wrong. It still reads like LinkedIn to me but “translating” doesn’t help. Thanks in any case.
This is similar to the approach in healthcare, at least my experience in pharmacy.
When an error occurs we find the root cause but in 99% of cases a change to the environment/process is required to avoid it in the future.
Humans are all imperfect, their competence will change hour to hour let alone day to day. But you can control a system and put checks in place (admittedly a human can still do the process wrong, but then you need to think how the process can be clearer).
No-blame culture is very effective at providing an open environment to share mistakes, learn, but most importantly avoid reoccurance.
If I recall correctly, in The Pragmatic Engineer, this was described simply as "provide options, not excuses" and is a critical way of building credibility.
Love this idea; won't go over well at all with the other accountants I work with. MY CFO doesn't like graphs because they "hide too many details' and insists on everything presented to her being in a table. MY VP pulls out a calculator and re-does the math on tables in decks by hand to help him 'understand'.
A lot of leaders like to hide in the details because the bigger picture is a harder problem.
>If your corrective action depends on people remembering a conversation from six months ago, you don't have a corrective action. You have organizational folklore.
We call this incident postmortem, part of a company process. After serious problems and firefighting, when the patches and hacks and people working 24 hour and it admins wake on a Sunday morning to do work, we sit down and try to figure out how this will _never_ happen again.
> At first I thought "I don't want the details" sounded dismissive.
Well that is because it was delivered by someone who was actually being dismissive!
If they want to know what "we are changing" but don't want to know any detail as to why, even presented at an appropriate level, they are very likely looking for you to take all the risk of the decision and they are dismissing your need to have the necessary informed consent for it to become the we.
Now, if you're presenting that information at a level of detail that is truly irrelevant or deep, that is your problem to fix.
But otherwise be very wary of this pattern of communication where "we" might mean "you". Anyone with any sense of risk management should know not to do this.
If you have not seen it, watch Margin Call. When you get to the boardroom scene [0] you'll see this play out, but you will also see the (pretty rapacious, direct) boss understand properly that the information he is getting from a low ranking employee is nevertheless his problem to understand, even if only at a 10,000 feet level.
Beyond the drama of the way the scene ends, it's actually a very good model of how that interaction should play out for everyone — making it clear that everyone in the room shares in the "we" of it, including the decisionmaker.
I have been in rooms where it played out at least something like that, where the boss wants to make an attempt to share the burden of the detail, and I've been able to leave them feeling somewhat supported, or at least without feeling like I was being shafted.
And I've been in rooms with people who don't want to know the details but just want to know what "we" are going to fix — those guys have never been as good to work for. If they say "we" but leave you with the impression they may mean "you", it's bad.
[0] said scene, but really, really, watch the film if you never have:
How about explain the problem and solution in 3 to 5 levels of details in a top-down hierarchical manner sort of like a decision tree. Let people decide how far they want to drill down.
>> I know that if we get into the details, the reasons will be perfectly reasonable. You'll explain what happened, I'll understand why everyone made the decisions they made, and I'll empathise with you.
>> Then it'll happen again.
>> So I don't want the details. I want to know what we're changing.
Even if the manager's approach was correct, this is just a hurtful way to frame it to your team. It's great that the author could reverse-engineer what the manager was doing, but it shouldn't come to that.
Why couldn't the manager say something like:
> You are all great engineers, so I already know that everyone made the best decision available with the information they had at the time. What are we going to do differently moving forward?
I'm still not sure I agree with the strategy, because I don't see how you can understand what happens next without understanding what went wrong. But I can see how the framing might help steer the discussion.
If you don’t know what happened how would you be able to tell if the changes are addressing the problem? If you don’t care about the problem being addressed why jump in a call in the first place?
Learned this doing incident reviews. If the exec asks for details it usually means they don't believe you yet, so "skip to what changes" is actually the good outcome.
Post-mortem, 5-why, retrospection - all this to understand what and how it happened so the org may introduce change so that it doesnt happen again.
Then another edge case happens and the cycle repeats.
Humans do this cycle all the time - big orgs invented the above structure for this process to be visible and collaborative and so that the rest of the org can learn too.
Then you put execs on top, they will come, ask some questions so they can sleep better at night.
Corporate playbook, some people hate it, the rest understands it… and plays the game.
Idiotic article. Nobody collects "why"s so they can not change anything. Not changing anything is never the failure mode. The failure mode is changing the wrong thing. If you are a leader, and something egregious goes wrong, it is your job to ensure that your people don't just put some bandaid fix on it that will keep causing issues indefinitely. Understanding the complete "why" is a precondition for that. Not bothering to hear a complete explanation because you trust your people to do the right thing is fine, but then why were you on that call in the first place? Let them do their job.
We write up timelines. We reconstruct decisions. We explain dependencies. At the end of it, we hand over a document that contains the specific combination of events that led to the incident. Everyone nods their head, says "that makes sense", and we all move on with our day.
...huh? I've been in corporate America for 19 years now and I've never once had this happen. I've never seen anyone just look for the cause and not ask what can be done to prevent it in the future. Everyone KNOWS that question is coming next.
How you can have an informed conversation about change without hearing the problem out?
If you trust someone that much, why are you even in the loop? Why are you badgering them at all? This is just bad management pretending to offer underlings autonomy. As soon as it fails, everyone loses their job.
I thought keeping details away from c-suit regardless of how technical these people are was the standard thing to do. Your job is to make these guys work the minimum. We fucked up, we fixed it and it won’t happen again. They don’t need to know anything else. You could upload the forensics report to a share drive and make the link available if they want to see it, but most times they won’t bother.
I can understand the inclination, but actually I /do/ want the details, because without the details I cannot consider and measure if the change proposed is a reasonable pathway to help manage the class of problem. Many times, people in organizations while well-meaning are siloed and focused specifically in their own niche and therefore will propose changes that cannot be coherent in any holistic manner across all the parts of the organization that would need to be affected by the change. That means you /do/ need to understand the details, and also have a big picture view.
The executive is there to help with the big picture, but you can't see the big picture if the canvas is blank.
SVP seems like such a useless appendage in the story.
He doesn’t want the details, he just wants someone else to tell him what they’re gonna do. But if he doesn’t know the details, he won’t understand if the plan will make things better or worse.
So… what exactly is this presumably valuable person there to do? Just to gage the confidence in the room? Presumably everyone was confident before the mistake happened.
What does this person who just wants a solution add? Why isn’t he fostering a discussion and adding his insight? Why is he in the room at all?
If I'm to explain what needs to change, that will by definition address processes or lack thereof that led to the problem, no? Which is then explaining what happened and why in an indirect manner, so many of the proposed changes may not make sense without more context.
I would not have said that. I would have asked for the details and tried to find the gaps because, almost always, something could have been done. Then the question is how much risk reduction are you willing to pay for.
Presumably the SVP joined the meeting because the incident was significant. The team can present the options but the executive may need to make a judgement call, and he can't do that responsibly if he turns his brain off. For instance, what if it happened due to understaffing or things falling between the cracks among teams, possibly indicating inadequate org design? Such things are the responsibility of management.
The quote box at the start of the article is the essence of it, the rest is AI-speak dressing it up.
TLDR:
Some outage causes a meeting with an SVP. SVP preempts the discussion with the following:
“I know that if we get into the details, the reasons will be perfectly reasonable. You'll explain what happened, I'll understand why everyone made the decisions they made, and I'll empathise with you.
Then it'll happen again.
So I don't want the details. I want to know what we're changing.”
I get triggered from these Linkedin bs type of articles. In the real world time and money is limited and everything gets prioritized. I worked at corps were MOST of the incidents reoccurred cause fixes was never prioritized. Cheaper to have people on call than to fix stuff.
I'm giving you the details because I'm going to give you options for solutions and I need priority information about those solutions, and you need to be informed about the trade-offs and costs of each option because you have better knowledge of our overarching objectives than I do.
If our overarching objectives don't impact our on-the-ground decisions, what are you even for?
> Then I realised that "I don't want the details" wasn't being dismissive. The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".
Something about this feels wrong:
- If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
- If you don't trust them completely, how can you know if "what happens next" is appropriate without knowing the details?
Isn't this just reinforcing the idea that leadership doesn't need to have their feet on the ground?
The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
You answered your own question.
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
> The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
These two statements are at odds. You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more.
It's not just about trust. It's about knowing what your team is up to so you can provide them the resources necessary while also working to remove any obstacles in their way so they can get shit done. That's what good management does, it's just rare to see in low-trust organizations that treat everyone like a cog to be micromanaged because no one trusts anyone to get shit done of their own accord for whatever reason.
We are arguing because there is an IT maturity line that goes from chaos to corporate and the more like a startup the more it will be like chaos (by design because maturity is too restrictive). So horses for courses. But as the team grows the same lessons get learned everywhere you bring in ex-bigco to implement changes to be more mature. That will include incident management process that will make this situaton routine and not worth blogging about. Probably in a mature process leaders already read the summary and or postmortem if they felt they needed to.
> These two statements are at odds. You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more.
But doesn't that same logic apply to whether they need to know the details?
> It's about knowing what your team is up to so you can provide them the resources necessary while also working to remove any obstacles in their way so they can get shit done.
That makes sense - I don't know if that's what the blog post was implying though. Ironically, this blog post could probably benefit from a few more details about what the meeting was supposed to achieve :)
> I believe you. I don’t need the details. Tell me what we're changing.
Yeah, for me in these situations it’s often about messaging to stakeholders, and so management in general feels like the problem has been addressed. I do think there’s one important thing missing from the final statement of the post, that he sort of covered briefly when talking about too much process, and that is what is the tradeoff/cost of implementing the change we’re making. Engineering is fundamentally about tradeoffs, and any significant proposed change requires some tradeoff and all too often I see management not asking about this explicitly. I’ve had enough once in a blue moon errors have retros, where the team decided it made no sense to make changes, and others, where the cost was high enough it was worth increasing workflow headaches. I think these tradeoffs being explicit are important at the director and above level, since there are potential real impacts to the team, and that is the level to deal with morale or headcount pressures due to changes to processes.
The meeting was about the SVP teaching the team that incidents happen, what's important is to prevent the same issue from occuring again.
The fact the author had never been in a call with the SVP implies incidents that happened before were explained in details, without much process change.
A perfectly reasonable explanation is that this SVP needs to explain in a different room what steps are being taken to prevent it from occurring again.
A lot of times, people just need a gentle nudge, or an official blessing from an authority figure, to go ahead and make the change that is necessary. You can tell people to manage up, to fully own their work and their process, etc. But a lot of humans don’t want to step on others’ toes or be perceived as acting out of line. A worker on the assembly line might know the best way to make the line run more smoothly but it’s easy to think that’s “not my job” or “above my pay grade”. Sometimes all that’s needed is to tell them, “Yeah, go ahead and do it.”
Getting that to happen more automatically is one way out of this situation, but that tends to cause problems of its own, with blurry lines of responsibility and authority. So it works best with relatively flat organization structures, which aren’t used much in large organizations.
> But a lot of humans don’t want to step on others’ toes or be perceived as acting out of line.
It's conditioning. In all organizations above a certain size (> 100 people), stepping out of line receives an official reprimand. Someone always complains. Receive enough reprimands and the person is terminated. People learn to stay in line because they've been punished whenever they step out of it.
Unless the organization really truly rewards that behavior. Which is rare.
The senior-most people (VPs, founders) often say they want it. But it’s the middle managers who will stab you in the front in revenge, while staring you in the eye. And if the VP doesn’t step in, they are endorsing the outcome.
The legend of Richard Montañez and inventing Flaming Hot Cheetos itself is not supported by facts, but the underlying story of someone low level who jumped the chain of command and contacted executives at the company seems to hold up. Which means there's at least one organization out there for which it's true.
I think that's overthinking it.
First of all, he didn't say the executive "trusted them completely". Those are your words.
He said "The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".".
That can be true even if they don't trust them completely. Let's say he trusted them 80% and not 100%. Isn't that pretty much quite trusting anyway?
Besides, the executive can trust them even 100%, but still want them to give him the general picture. Among other things, it's his job to know and be able to report when asked.
The VP also needs to know if the fix will require some sort of resource reallocation, or if the timeline is long so they can deal with the other stakeholders, or whatever. These are all forward looking things that the details influence but the details aren't what matter.
Asking what the OP was going to change is not an automatic nod of approval for his approach. I’ve been part of some large interventions spanning multiple quarters, that started from a simple conversation with an exec just like this. There are downstream effects to these decisions that affect many stakeholders and departments - so they cannot be made in a vacuum or delegated completely (I trust you, go ahead and fix whatever). Depends on the severity and scope of the issue of course. A simple deployment gone wrong is different from a fundamental issue with the architecture that is straining progress.
As an example, a major rehaul of a core system could push back product (new feature dev) timelines, that can affect customers and their releases, bottomline of the company, reputation, tax repercussions, if it’s an American public company then it’ll affect SOX compliance & audits, could lead to spicy investor/analyst conference calls, and so on.
In a large enough organization it is simply impossible for executives to be on the ground floor with everyone else. And not only is it impossible, it is essentially a waste of their time. They could maybe be maximally involved in only a few projects at a time.
I think its entirely contextual as to what the size of the org is and the scope of the projects. At a startup an "executive" or VP would legitimately be able to be working side by side with everyone else, or hearing about the ground level operations, but just by nature of the role being SVP I imagine this is a large enough company that this is no longer the case. At a certain point that becomes somebody else's job that is much closer to the team at hand.
The role entirely morphs and their job becomes to enable teams, get blockers out of the way, fast track or assist things like permissions/processes, shoot down ideas that legitimately don't add business value or are not worth wasting existing engineering effort on.
And I imagine in this scenario the point wasn't for the executive to literally hear what happens next, it was to ensure the proper steps are being taken and make sure "somebody" is handling the what happens next part of things, and so they can report up through the chain or delegate tasks for exactly that "what happens next" task.
> In a large enough organization it is simply impossible for executives to be on the ground floor with everyone else. And not only is it impossible, it is essentially a waste of their time. They could maybe be maximally involved in only a few projects at a time.
This is a very strong argument that no organization should be allowed to get that big.
I get the sentiment, but this is the reason we have airplanes, MRI machines, and computers that fit in our pockets. Advanced technology simply isn't possible in an economy of only ~100 person organizations. And I say this as someone with a deep sense of unease with modernity and all of the diseases of civilization. We somehow have managed to create systems that support literally billions of humans and allow many of them to have a quality of life undreamed of even a few hundred years ago. I dislike big orgs as much as they next guy but we need to understand why they exist if we want to criticize and reform them.
Big organizations don't enable greater technological progress, which is why companies do 98% of their innovation before they get huge, and get bad at innovating once they are.
Big organizations enable scaling and associated economies of scale, so it's possible for Apple to sell millions of iPhones cheaper than if they were a smaller company. I don't think that tradeoff is worth it for any individual consumer (pay a little less to be much more abused by the company), and I don't think it's worth it for society.
Most of the big corporations in America aren't providing things as useful as MRI machines, they're providing net negative garbage like social media and advertising. Big corporations like United Healthcare Group are inflating the cost of medical care rather than providing economies of scale and reduced cost for their clients.
The US desperately needs to bring back antitrust and start breaking these massive companies into much smaller pieces.
Ironically, those same systems that enabled high QoL are also the ones involved in killing us, whether that's carcinogens everywhere, climate change, resistant bacteria, or even just plain old nukes (or soon drone swarms).
I wouldn't be alive if not for advanced medicine, but if the structures that saved my life are the same ones that constitute our own Great Filter, my life was most definitely not worth the trade. There are too many other people I love to think that.
There may be something common across many of the ground floor parts, such that even just knowing what's going on in a fraction of them, they are much further in their vibe-calibration of what ground floor is like than if they know zero.
The best leaders know what they need to pay attention to. You cannot know everything. You can know any one thing (or perhaps several), but there are more things to know than you will have time to learn in your lifetime. The question is what is important for you to know, and what you leave to somebody else.
You will have to trust someone will figure out the details and come up with a reasonable solution. Sometimes you need to know that solution, sometimes in detail, sometimes just enough to know that they solved it. Sometimes you are the person who needs to figure out the details and make the solution. Knowing which you are is important.
“Trust” is usually bounded. I trust my elementary aged son to get dressed for school without supervision. That doesn’t mean I trust him to start a campfire without supervision. A chef might trust that their new employee is a hard worker and well intentioned. That doesn’t mean they also trust them to create and serve original dishes.
Here you are recasting “I trust that, despite the bad outcome, everyone’s actions here were reasonable” as “I trust that you are going to solve this fully without my guidance”. The fact that the word “trust” appears in both of these statements does not make them equivalent. Trusting that people are capable and well intentioned does not mean trusting that they will accomplish the same thing without you.
That’s not even true though, leadership is coordination, even if you trust someone else to do the right thing wrt their own problems that still may be globally suboptimal wrt the goals of the overall team / project.
I don't read the article this way. It's pretty sane for an exec to draw a line where the team's autonomy meets leadership's responsibility for structure and incentives around which said team needs to organize.
"You did the right things and met the goals we set, but we hit a problem and we all agree it shouldn't happen again. Let's talk about what needs to change to address this."
You can trust someone to know the details and reasons for why something happened, and still want to provide the specific direction of "what are you doing to make sure it doesn't happen again". The organizational default is often to explain a fault (or worse, focus on blame) without specifically working to find a process improvement to prevent it from happening again.
Except if you don’t know the details, how will you be able to make an informed decision on what happens next?
- We’ll just fire joe - Ok cool
I agree with the parent comment. You need SOME level of detail, otherwise your decision might as well be a guess.
The SVP doesn’t plan on making any decisions here.
Plan is the key here. In reality the SVP is hoping that they can make the decision to go with your solution.
Q: What are you going to do about missing this bug?
A1: "I'm going emphasize "BE" in Hamlets' "to BE or not to BE". - You should be fired for this, but that is not the planned decision.
A2: "I'm going to spend $$$$ on some tool". - This tool might solve the problem, but the budget doesn't allow for it and you should know that: I'm forced to make the decision to override you but I didn't plan on making that decision.
A3: "I'm going to [summarize something reasonable]" - good, you are doing what I expected.
Note that A3 doesn't have to work, it just has to be reasonable. If it works great, if not I will ask again next time and I expect you have different answer. Not working at all better be rare though since each not working loses confidence, but fortunately this is rare. However often working is sometimes all you can get (the next bug might be unrelated - no single answer could solve both). The idea is we repeat this for each bug until the rate of missed bugs is acceptably low.
The SVP doesn't need to make any decisions here. Part of leadership is giving your team the space and time to make the decisions themselves.
> The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
If I have a handle on things from bottom-up to top-down, I don't need them to rehash weed-level technical and specific details. I need them to tell me what we're going to change, and it needs to be a BIG systemic change, not minutiae, which is where many engineers thrive.
If execs demand a big change every time something goes wrong, they will demoralize everyone working for them, and probably waste a lot of money.
Most of the time, operational failures are either the result of a new feature being sent off without enough testing, or a dozen small failures lining up to remove resilience. Big changes without well-thought out plans primarily serve to stoke executive egos.
I think you have a distrust of your leaders and are taking this from a very different place the author of the article was.
I have a healthy distrust of modern business structures, and an awareness of humans being humans.
> If I have a handle on things from bottom-up to top-down, I don't need them to rehash weed-level technical and specific details.
I should have been clearer.
When I said that good leaders pay attention to things from the bottom-up, that doesn't mean they always know everything needed for any discussion. It just means they are able and keen to understand the low-level details when they matter, because they routinely look at such details in the projects that they're involved in.
Of course it's possible that this story is just told badly and the SVP already knew the technical details, in which case the blog post is a bit misleading.
> I need them to tell me what we're going to change, and it needs to be a BIG systemic change, not minutiae, which is where many engineers thrive.
How can one know if the engineers' proposed "BIG systemic change" is appropriate and complete, without knowing the details of the problem it is addressing?
> Of course it's possible that this story is just told badly and the SVP already knew the technical details, in which case the blog post is a bit misleading.
I don't think it's necessarily misleading; it's just reminding folks that senior leaders have a different perspective than those below them and tailoring your message to the audience is important.
In this case of this, it was a reminder that while it's easy to believe the SVP is being dismissive, that's not the case at all. Instead, they want systems-oriented decisioning, not detail-oriented.
If anything, this was the SVP helping to grow the leaders under him by forcing them to think at the levels required to interact w/the rest of the organization.
> How can you know if their "big change" is appropriate and good without knowing the details of the problem the change is addressing?
I'm a Senior Director leading three teams of engineering and science resources. I hired smart and capable people. Why should I, as a SrD be a technical bottleneck?
> How can one know if the engineers' proposed "BIG systemic change" is appropriate and complete, without knowing the details of the problem it is addressing?
That’s the trust thing. Maybe the SVP trusts the people in his organization are capable of making the appropriate change without his supervision and what he’s doing is coaching them on how to do an incident postmortem properly.
Of course as you called out, the SVP might already have all the necessary details to do more than that. But simply pushing people in the right direction is a big piece of leadership’s job, at least when they have people they trust.
He wants to direct/align on strategy, not tactics. I'd say that's a good division of responsibility so I'm not sure I see your objection.
What I got from parent comment, and I agree, is:
If I trust you know what happened, and trust you to know what to do so it doesn’t happen again, why do I need to know? And how will I know (and endorse) your decision if I don’t know what happened?
It seems weird.
> - If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
You might need to know timescales so that you can inform upwards/outwards/other. You might need to make sure other resources that might be necessary are available. The planned "what next" and the proposed timescales might have impacts elsewhere that they haven't realised (and maybe even until now had no reason to be aware of). If they tell you the what next and there are no such complications, then they can carry on, otherwise there might need to be some changes to the plan or changes to other plans to allow time/resources to be available for this plan or just to stop potential near future conflicts.
Trust, but verify. And perhaps even assist.
One reason is prioritization. It’s enough to know what the effects of the failure were, the likely cost of what needs to change, and what areas and processes the changes will affect, to decide if and when this should happen.
Leadership still needs to understand these aspects on a conceptual, topical, and cost level to decide on the resource allocation (including time and schedule). They don’t need to understand the technical details, unless they are a technical leader with higher relevant technical competence than their reports.
They trust your ability to do a root cause analysis. They don’t trust that you feel empowered to do anything about it, so they are verifying that you actually do.
Also, not every IC or EM has the same skills and experience. There may be certain categories of work that you can be 100% hands-off and trust them to just do it, and others where you really need to understand bottom-up. Part of being an eng leader is to know when to insert yourself and when to "trust the process".
Trust them completely != "assumed that we were competent"
I've worked with plenty of competent people. Only a few were so good I trusted them completely.
Leadership still holds engineers accountable. And for that, they need to know the plan. Leaders are also held accountable, so again, they need to know the plan.
Not wanting to know the details is a red flag. Sometimes the process of expressing the details can reveal things that were unknown by one or both parties, leading to improved processes. The only acceptable exception would be "Let me explain. No, there is too much. Let me sum up."
The details are sometimes important and sometimes irrelevant. If you insist on skipping them all the time, you will often be right, but when the day comes that you are wrong, your unwillingness to listen to the details will be the reason the problem can't get fixed.
To me trust means youre willing to treat people like a black box. If the detail about the outputs is another i/o or something about an output that would appear on a spec sheet, it should be included.
Inner details shouldn't matter. Thats what makes a good summary
Well,
Another explanation is that not wanting the details of operations is based on complete lack of trust, not in the their competence but in the way they would describe things. The boss doesn't want the details because they are sure the details would make it seem like the underling did the right thing and moreover, monitoring that detail would just give the underling leverage over them.
The boss wants to know about changes to avoid this but whatever it is, they'll push back on its extent and won't want to hear about details then either.
It gives LinkedIn post vibes
That's all context management to all the way up
The devs job is to impliment and advise how to attain the owners goals and priorities, not to set goals or priorities except at layers below, and in the service of, the c suite's directives.
I'm not saying c suite are unquestionable gods, I'm saying that there is no such trust-or-not dichotomy. It's 2 different things.
No matter how ignorant I am in some domain and no matter how knowlegeable some expert is I hire to do something for me in that domain, they can only tell me how to get what I want, or what's the closest that is humanly possible, or the costs of various conflicting priorities and compromises. They can't tell me what to want.
Maybe I DO actually want to burn my whole budget and 5 years of time on some facet that they and most people would say is not important and not worth it to the point of being irrational. Their job is to inform me not to decide for me. I can trust them to inform me honestly and with good judgment and deep knowledge, and then to impliment what I decide also honestly and with competent skill. And yet I still need to be the one who is informed, and then makes a decision and issues a directive.
It's 2 different things.
It's like Steve Jobs being totally unreasonable about tiny details of fit & finish in the hardware products. It makes no sense (to most people) to have custom ethernet jacks manufactured (back when it wasn't so effortless and flexible) just to get them a certain color or material feel, when off the shelf jacks already exist from multiple sources that work just fine for a 10th or 100th the cost, while already meeting a stack of compatibility standards and even safety regulations. No trustworthy engineer would do that. They would actually see it as violating that trust.
The SVP has to explain what they’re doing to prevent reoccurance again to investors, customers, ceo who don’t care about the widget on x k8s
Can't do that w/o talking about the cost of what you're going to do, and you can't do that w/o talking about the impact of the incident.
So you can elide some details when you talk to leadership, investors, and regulators, but you can't elide everything.
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
I work in a competent team and leadership is required.
Yup, several things are wrong with it even though it may be a good idea as a good shortcut on the part of the SVP
But it should be used and phrased better.
Used better:trust and only sometimes ask for the detials, like cutting the just-shuffled deck in a card game, sometimes the player to the dealer's right will just tap the deck instead of cutting it, in effect saying "I trust that shuffle". Sometimes, certainly enough to both keep the SVP well-grounded in the details, and to keep the team knowing the SVP is both watching and cares about the details, the SVP MUST ask for the details. Trust and verify.
In terms of phrasing, "I don't want the details" is all about the SVP sending the message s/he doesn't want to be bothered. The phrasing should be: "I trust you on the details on this one; let's go straight to what do we do to manage the next one of these?".
Notice leading with "I trust you on this", not "don't waste my time".
I would agree with you, this just sets up a scenario where the SVP positions themselves as a leader, without having to understand something technical, and without making the decisions on how to change things. It isnt about trust, it is about performing accountability so the SVP can say they did their job.
I think the author of the article is going out of their way to be charitable to the SVP for no reason. That meeting was a reassertion of power, not a wise move made by a sage.
> "What happens when requirements change inside the launch window?" > "The alert fired but..."
In the running example I think it is clear that you could stop at any point and address the problem from that point rather than continuing. Questions such as "Why do we permit last minute changes at all?" are given no space.
In a complex system sometimes there is no root cause (cf. aviation crash investigations and the Swiss Cheese nature of risk). By allowing one person to shoulder the entire context of the incident and for that person be the one to suggest lasting corrective actions it leaves little need for the VP to even be involved. How would they know if the proposed corrections are worth the time and money to implement? I don't see any leadership in the example described in the article - I just see a knee-jerk reaction rather than any kind of proactive reflection.
> "I thought we fixed this last time!?"
> "This time was different in a way that our corrective actions didn't address."
I sympathise with this, and a lot of teams do operate that way, but I fundamentally disagree with it.
Part of the reason Amazon's CoE-driven engineer culture had such operation excellence is that the responsibility was driven the whole way up the management chain. If your manager didn't dig down to the root cause of a sev2, the director was damn well going to, and if the director didn't, Andy Jassy was going to call them on it.
That sounds like a lot of busy work, and a bunch of it undoubtedly is, but the flip side of it is that the actual root causes were addressed, and if the root cause needed serious cross-functional resources to address, the escalation would get you what you needed. Need approval to bounce 3 months of work off your roadmap to address? You have a VP on the line to make that call. Need a couple of other teams to fix their shit? Here's a principal engineer who outranks their directors to drive the work. And so on...
https://news.ycombinator.com/item?id=48945241#48951538
Quoting myself here: Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org. I don't want to name bad orgs but what org are you talking about?
I'm here for the sentiment expressed by the SVP in this article: "I already believe we reached this point through rational choices. Let's talk about how to change the system so it doesn't happen again."
I do think his choice of words was a bit suboptimal. But, it got the point across.
> I do think his choice of words was a bit suboptimal. But, it got the point across.
As I've gotten older, I've grown to appreciate people who are willing to be a little rude in order to avoid wasting people's time or giving them false impressions.
For example, when recruiters reach out to me on LinkedIn now, I almost always say something like "I don't want to waste your time or mine: what is the salary range?" It might be considered a bit "rude" to immediately jump to money, but I figure that that's less rude than wasting 30-90 minutes of time for a job that I have no chance of accepting.
I also try and be a little upfront if I don't like food someone has prepared. I try not to be too much of a dick, but if someone prepared something that I don't like and they ask my opinion, I will say that I wasn't a huge fan. Again, maybe a bit rude, but I also don't want to mislead people.
It kind of goes against my gut; I, like most people, don't really want to make people feel bad and I certainly am not immune to people-pleasing, but I've tried to fight that for the greater cause of being more-honest and upfront with how I speak.
I actually think it’s more about the emotional maturity to cut to the chase and address someone’s actual concerns.
I’m a big fan of bluntness / honest feedback, but it can still be done very rudely when inappropriate
like the svp observed that the ic was saying the details, not because the ic needed to share them, but because the ic was anxious that the svp didn’t view them as competent. So the svp cut to the chase and addressed that directly
For your food example, a host might ask if you like the food - not because they care about their cooking skills - but just want to make sure you had a good time. So, saying you don’t like the food and giving cooking suggestions would need to be accompanied by some reassurance that you still enjoyed their company or beautiful home or something. Otherwise it’s still rude
This resonates as I'm trying to do this more, especially since I'm historically more people-pleasey.
Any resources you suggest?
I'm not sure I needed a resource; I just kind of felt like I should optimize for "not wasting people's time" and "not misleading people".
I frame this differently - it’s about getting out of your own head and think about what’s useful to the other person. And making this a habit so it’s easy.
Providing feedback on the meal is useful for their next guest, if they are interested in improving. If they aren’t, just say it’s fine and move on.
Shifting focus outside of yourself is covered in a lot of areas but Never Split The Difference stuck with me, written by a hostage negotiator.
> Providing feedback on the meal is useful for their next guest,
I mean honestly the last time I gave negative feedback on food I didn't like, it's because I didn't really want to feel pressured to have more.
Talking about changing the system "so it doesn't happen again" can only lead to more "rational choices". This is a thinly veiled tantrum spoken in a calm voice.
If you want to reduce incidents, you need to lean into the problems instead of dismissing them or delegating even more all over again. Doubling down on trying to keep your hands clean is shit leadership.
The basis of this blog post is delusional. The language from the SVP is the desire to fast-forward past the thought process that comes with listening to "the story". This could only lead to more problems. They're basically admitting they weren't going to listen in the first place. "The story" doesn't always lead to blame game, and sometimes the changes needed are the context in which rational and trustworthy people are working in. That is not a change to the system, but to the organization.
I don't think the two of us read the same post.
You're inferring a lot of emotion into this writing that I simply did not read.
Right, maybe saying "tantrum" was suboptimal.
I think there's something valuable in your take. The idea that technical failures and process failures are parallel tracks is not new.
Certainly not the kind of thing where an exec needs to pull out the "I don't want problems, I want solutions" card to illuminate that they are only valuable to the process domain resolution, not the technical one.
The accommodating reader can read that interaction as entirely because of mismatched insight into the issue. I'm willing to budge that maybe too much time in the agile dens does really does necessitate the conversational roundabouts. It's taught people to taboo process they don't see the immediate need for.
How do you say that so they actually change?
They also have rational choice to not have to get their hands dirty. What incentive is there?
The incentive is not causing an even bigger mess and keeping their role. Contrary to popular belief, executives really are held accountable for outcomes.
It's a question of how much failure is visible and whether anyone else cares, including themselves. If they were only ever motivated by money, they might already have enough since so long ago that now they really only care about harsher legal action that money can't fix. Getting fired isn't a motivator anymore. That's a bad fit for any organization. Someone else is also dropping the ball there.
> "I don't want the details. I want to know what we're changing."
This really feels like the sort of senior manager who would say "oh, this/these team/s screwed up, so here's the plan they came up with", talking about their team/s.
Speaking as a C-suite guy, "they" is about as contrary as I can think of what I expect a leader to do: it's "we", and your job is to take responsibility, both in the problems (big problems need to be explained all the way up and sometimes externally) and in the solutions (which to be comfortable with, you obviously need a sufficiently thorough understanding of the problem).
Because up the chain, I will take responsibility for it.
> To drive change in your organization, don't ask "why did this happen?".
> Instead, ask:
> What are we changing so that the same class of failure is less likely next time?
Why are these framed as being almost mutually exclusive? I don't get it. "How/why did this happen?" informs how you answer "What are we changing?" Even the following section, "Reasonable people", functions in this way with a "Why did this happen?" followed by "What are we changing?".
Just feels like this article has some internal conflict with itself in order to achieve a "don't do that, do this" type of style.
Because answering the “why” feels like you’ve achieved something. You all blame Gary, pat yourselves on the back and then go home satisfied.
It’s like when cloudflare does a blog post saying they were out for three days and it was because they let someone slopcode the DNS. How has that achieved anything? It will go down again the following week. The customers want to know how you will prevent it happening again, they don’t really care how entertaining your blog is.
How can you talk about what you'll do w/o talking about cost, cost of opportunity, impact of motivating incidents, etc.??
Maybe if the changes you're doing are small and cheap, sure. But if they are invasive and costly...
I don't get it neither. It's important for context to document "why something happened" alongside action items to prevent it from happening again. This is part of history that will be extremely valuable as company knowledge on why processes exists in the first place.
Otherwise, what will be the discussion when it's time to question it later?
This should not be exclusive.
One thing that's often left out that tends to grind organizations to a halt is that continuous improvement processes tend to be additive.
You always add rules, and alerts, and so on. You need to revisit existing processes to and see what can be removed or replaced.
My other nit with these is a term that's abused a lot "Root Cause", most complex issues have many contributing factors, not a single root cause, and if you force the teams to find one, they will.
I dislike RCA (Root Cause Analysis) it puts you in the wrong mindset, CFA would be better (Contributing Factor Analysis).
I recently had a vendor CAPA rejected because the root cause was listed as "data entry error." Oh no, the root cause can't be data entry human error! It has to be that the deadline was too tight, or they were rushing, or they were distracted etc etc. I ask the vendor, were you distracted? No. Were you rushing? No. Just a simple typo. Update root cause to typo. Oh no the root cause can't be a typo it has to be the free text field in the form design that allows for typos. It's a process error that there's not an automated QC to catch these typos! Etc etc.
Yeah I've seen this a lot. People freaking about issues that have small or moderate impact on deadlines, and very little to no revenue impact. And suddenly you're moving 50% slower because you have weeks of review processes to make sure nothing is missed.
Doesn’t eliminating RCA case by case also lead to fewer CFA thereby eliminating them as well?
It's rarely a single cause, there's no root, it's a forest.
It's usually a combination of things that are not quite right, specially in mature orgs where all the simple fixes have already been applied.
It's semantic, if you do RCA right you don't stop at a single cause, but that's what usually ends up happening.
There's still some problem with that approach. OK, I will tell what I'm going change to prevent recurrence of the issue. How does that makes sense to the audience? Unless they just want hear that "some" change will be there and don't care about how that change would make any sense.
It boils down to what exactly is the ownership or accountability of your SVP around the issue. Why are they even bothered to ensure that there will be some change? If they are responsible for ensuring that the issue doesn't happen again, then they do need to say "I want the details".
I think there is a subtle additional level of trust that the author didn't call out. As the author noted, the SVP trusted that everyone involved in the issue responded rationally and professionally. But they left out the part where the SVP also trusted that the author and their engineering counterpart had already analyzed what happened and how.
The SVP doesn't have time to dig into the "who did what and when" timeline of a problem. They do need to communicate that they're willing to invest resources in preventing recurrence. Was the team burned out by noisy alarms? Then the SVP needs to hear "We're spending time improving automation to reduce alarms, and tuning remaining alarms to make them less noisy. This will take engineering time away from feature work." Or maybe the system had a single point of failure that broke, and the SVP needs to hear "We're redoing X to eliminate a single point of failure, but it will increase our infrastructure budget by $Y".
The SVP is inviting a higher level discussion around what resources need to be consumed in order to be sure this doesn't happen again. And they're opening the door to defending that use of resources.
What you describe is exactly what a 'leader' is supposed to 'see'. When to know when details are needed, and when to trust their delegate. The trust is also balanced: if what you say does not align with what the leader is 'seeing', then trust falls apart.
Everyone has some measure of the 'big picture' and the person at top is supposed to see the biggest, final picture. Where everyone and their processes fit, within the big picture. Delegation could be perfectly aligned (and trust would be easy) if not for quality:
Quality determines how well the relationship works between leader and delegate.
The amount of quality is determined by the amount of need, and urgency available. The relationship (and communication) evolves over time as quality is determined by leaders.
Regardless, there's always going to be difficulty because there's never an ideal system where quality is perfectly measured. That's why communication with people, as people, is also very important. When we perceive disrespect: communication shuts down. Humility must be respected.
If you can't communicate, there's definitely no measure of quality available for the relationship between leader and delegate. If communication is there (and communication should be measured as transmission of real information), then trust is more easily measured.
Once you know how much you trust something, then you know how much detail you need when you 'need to get something done'. Because you know the big picture, and where this thing you need fits. And how well it works.
OP is already at the director level. The further you go up the management chain, the less people are involved in process or technical decision making -- they are primarily concerned with organizational decision making. It sounds like the issue was something that was fully within the management scope of lower levels of management here:
> Something had gone wrong that shouldn't have.
The SVP is trying to determine what change in management is taking place, not what technical change is taking place.
There are a set of principles at play: used by leaders and engineers, when trying to 'do work'.
When you need to get something done, unless you can build the entire thing, you have to trust someone to help you. There's very few people who choose to live a life where they are 100% self-sufficient, living-off-the-land.
Leaders have to trust people, engineers have to trust components.
It's not just people. Shuffling people around is not fix everything. People, process and technology - any of them could hold the key for a solution. That would mean, you do need to understand whether is is people issue, process issue or is it tech issue. That requires listening to details.
"Okay, everyone is overworked, and we need you to hire more people for this team. We have too much tech debt, and we can't predict which bit of it will explode next week. The change needs to start with the attitudes of our executive leadership."
Oops, now I'm fired.
Yeah, I think the article assumes the employees doing the work have the leverage to implement the changes themselves. How often is that the case?
>> "The requirements changed three days before launch."
> Of course they did! What happens when requirements change inside the launch window?
What answer is wanted here? "Moving forward, we'll ignore the VP of Marketing's last minute 'must haves'"?
> What answer is wanted here?
"We will be more proactive and agile, and volunteer to work during the entire weekend to keep our customers happy, sir!"
"Good, good, but don't bother me with the details."
"We are under-qualified"
I suspect the author still misunderstands the SVP. If I were the SVP of engineering saying this to someone, especially someone who is in a non-core engineering role like product, I probably recognize that they have a tendency to verbally spew and I want them to focus on the risk mitigation and response plan. I’ve already talked to my engineering leader because they informed me immediately, and I’ve already read the incident details.
But also the post reads like it comes from an inexperienced/immature service org, so maybe they’re just sorting their operational response process out.
Strongly disagree with this.
In order to figure out a good solution, you have to have mastery over the problem. That takes a bunch of work.
> Understanding an issue is not the same as fixing it. A good explanation can make things worse. Once everyone agrees that the behaviour was reasonable, the urgency to change anything disappears.
This is a non-sequitor. We're not here to evaluate whether the behavior was reasonable, we're here to figure out how to avoid bad outcomes. If all the behavior was reasonable, then the problem lies somewhere else, and that's an important finding when crafting a proper solution.
The author is literally saying that asking why is the wrong question, and instead asking "What will we change?" is the right question. I get where this comes from, but you can't really answer what will change until you deeply understand why it happened.
I think you did not read that part.
> The SVP wasn't interested in understanding how the issue happened or who was involved. They didn't want to be convinced that everyone involved behaved reasonably. That's a baseline expectation.
Their question became:
“Given that reasonable people produced this outcome, what needs to change?”
Of course you have to know the problem, but he trust and excpect a sulution from them. So this does just work this way, if you can trust the people. If not, then the top person has to check the solution for himself.
A good explanation is not a fix but is a good starting point to a good diagnosis which is the foundation of a good fix.
The issue is when the calls and urgency are raised before making some mental space to even verify that there is enough clarity on what happen and these interrupt the clarification and remediation planning process.
Saying you shouldn't ask "Why did this happen?" and you should ask "What we're changing?" instead is wanting to jump the gun and pretend a required step doesn't exist.
It does exists, is just the executive anxiety reasonable in terms of wanting to be protective of the business but as protective as a family member of a patient wanting a good outcome from the surgeon during the surgery.
Only worst because the surgeon's boss is this anxious family member, and she has a laboral gun pointing to the doctor's neck.
Yeah, I disagree with SVP's approach in this story. First, get the current issue understood, resolved and remedied. Then, thank the team for their good work in fixing this issue, and do a post-mortem to see what process improvement could have prevented the issue from rising in the first place, or at least detected and mitigated it more quickly.
"Okay, then I'm going to rewomble the dinglehop, which should un-galvatrate the percapitator." Why is the SVP in the meeting at all? If they don't want the details, they can't contribute to the change, or even evaluate whether the change makes sense at all. The SVP could be completely removed from this situation and the outcome would be exactly the same. Which shouldn't be surprising, since that's almost always true of all executives anywhere.
> Everyone nods their head, says "that makes sense", and we all move on with our day.
Why? Why do you need to be intimidated into actually making effective change? Why is everyone walking around with their hands tied, unable or afraid to enact change? Probably because of the terrible leadership at this company.
>> "We missed it because Alice was on holiday and Bob thought the Widgets team owned it".
> Okay. How do we make ownership unambiguous when someone is unavailable?
You cannot take this step without details. That Bob-Alice story is the details. You do actually need to know that story before you can take the step of knowing that "unambiguous ownership" would be a worthwhile change to make. What exactly is this SVP doing again?
This entire post could be summarized as "executives are useless" and "do the 5-whys", both of which everyone already knows.
This article may be more appropriate for LinkedIn.
Yeah it reads to me like it's written by an LLM, although I start to doubt myself these days.
I do like the term "organizational folklore" though. Actually properly documenting processes & expectations is important (and rare).
Give AI vibes, so agree.
Good article write-up, I didn't need to read more than one paragraph to get all the information from the whole page.
It's crazy how far i had to scroll to find someone saying exactly what i was thinking - how come AI sloppy writing like this is showing up on Hn front page?
Haha ouch.
The post-mortem process can cover all of these things:
You can't really talk about what to change w/o talking about root causes, and you can't talk about that without talking about what happened. You also can't consider the costs of commitments to be made w/o discussing actual and potential impact because resources are limited and the cost of opportunity is real.It's fine for executives to get only a summary of impact + commitments. But the whole post-mortem process has to be followed, and some people have to be aware of all the details.
I can't help but notice that a climate of fear around being deemed non-essential or otherwise getting on an authority figure's bad side tends to create significant mismatches in priorities and breakdowns of trust that could be upstream of significant organizational inefficiency in a variety of situations. Perhaps solutions to meta-problems of this nature might be high-leverage, as the thought leaders like to say
The executive did not display empathy.
People need to tell their stories. They need to be heard and have their work and its difficulty respected.
That's what went wrong. Everything else is an abuse victim rationalizing their abuse.
Agreed, I came looking for a post like this.
I can't be the only person who thought this is a highly intelligent person being inadvertently gaslit by the SVP into thinking the exec is right to be dismissive with "I don't want the details" because introspection is common in intelligent people.
Had someone said that to me I'd have walked away after saying "fine I'll sort it'.
This illustrates that "trust" to do the job, and since they didn't want the details of the problem, they therefore don't need the specific solution description. This kind of response is also the same level of respect, it's either seen as trust in ability, or just downright disdain with plausible deniability for being rude.
Every post-mortem I've done had:
1. timeline of events
2. impact
3. 5 whys — the details of how and why things happend
4. actionable items
I've never seen a post-mortem without actionable items.
Do the actionable items usually get implemented?
You can expect what you inspect - Peter Drucker
Action items need an a forcing feature (like an SLO) to insure they are picked up. And someone needs to follow up when the item isn't handled timely. It should show up in team performance metrics and guide changes to processes to insure they get done.
The important part is the action items are assigned an owner and executed.
I’ve done plenty of postmortems and many meetings where we go through the motions but “leadership” does not assign and address follow-ups. Most organizations pay lip service to reliability.
I agree. Finding the time to update docs, processes and code based on a post-mortem is, from my experience, the hardest part. Critical, but difficult.
Actionable items go onto the backlog. The backlog grows from the top.
As a stopgap, a manual pre-flight check is added. The number of manual pre-flight checks becomes a ridiculous overhead but it gives a clear person to blame when something goes wrong.
Some of the best life advice I learned during conscription as a squad leader: do not explain if an explanation is not asked for. If you (or, importantly, your subordinates!) mess something up and are confronted by a superior, the only correct reply is "Yes sir I screwed up sir I’m sorry sir it won’t happen again".
That could make sense in the military. It could make sense for an offshore IT org whose role is to provide warm-bodied yes-men, or similar situations. But it’s catastrophic for a creative organization that’s actually doing anything other than CRUD apps or something equally mechanical, because it means management can’t possibly understand the systems that the company’s income depends on, which means they can’t possibly manage effectively.
It’s still up to them to ask. If they want to know the whole causal chain from distal to proximal, so be it. And if they never ask, I doubt it would help at all trying to explain anyway – they’d just zone out – and might be a good idea to start looking for a new job.
In any case, the actually important question "How to make sure this won’t happen again?" would presumably include the relevant parts of what "this" is as well. But in a more actionable frame.
I think the problem is when the execs aren’t looking at change-for-the-better. Instead they’re looking to appoint blame. Moving to a position of assumed competency in your staff and taking the perspective of how-to-improve-next-time is a real step forward in maturity. My guess is it’s rare and blame is easier: most management doesn’t reach this level, instead it’s immature.
I think that's a problem domain in itself. Identify when failures that occur even when skilled people doing their jobs properly, adjust the environment so that they no longer happen.
It is understandably a completely different task compared to what those skilled people are specialised to do. You probably need a dedicated role to do it.
Perhaps you could call them a manager. Their job is to see the multiple parts of the system. They should ask for the Details of what happened so they can determine why the problem occurred.
Consider one of the problems listed in the article
>"The alert fired, but the on-call engineer had already dealt with twenty low-value alerts that evening".
The engineer can say they were busy, they didn't see the alert, that they are swamped with things they think are low-value. Someone else can say the alert fired. Each person involved may have their own perspective, with different ideas as to what the problem actually is.
It's easy when you see problem described in terms of what the solution is. Someone needs to figure that out, to do that they need the details.
An SVP of engineering should care about the details, this is the essence of leadership. When you don’t care about the details you are just a manager and worthless i.e. you should be replaced with a leader.
Lots of folks are commenting on the internal dynamics, but the other part that didn't resonate with me was actually the what happens next part. This may be an overreaction on my part, stemming from watching lots of engineers suggest expensive solutions to relatively minor issues. And it's a mistake I've made myself before as well, I started in telecom with very strict standards for availability, so it's hard to beat out of me the desire to look through the most minor alarm for a potential problem brewing or get to a root cause of the most minor incidents.
And maybe this is just a part that the article dodges, but for something described as non-catastrophic, I would suggest that there shouldn't be a presumption of changes. In my view there should be an assessment of the risk of re-occurrence, and if it was a near miss, what is the risk had it not been a near miss. And then even should changes be part of the outcomes, how expensive are those to implement compared to whatever the issue was, and if they're too expensive compared to the risk, don't put resources into it.
My version of this is: "This is the kind of mistake that we get to make once. How do we prevent it from happening again."
I usually do want the details, but I also want to communicate my expectations up top and unambiguously.
A Senior Vice President of ENGINEERING doesn't want the technical details?
Yes, if a director and SVP are discussing the technical details of an incident, I would be concerned -- they should be concerned with putting the right people and processes in place to make those decisions.
When "Something had gone wrong that shouldn't have" bubbles up that far, you have an organizational issue, not a technical issue.
Of course, details might point upwards - plausible deniability could get damaged.
If the person who is charged with resolving the issue is capable, there's no point in a senior manager knowing the details outside of professional curiosity.
“If your corrective action depends on people remembering a conversation from six months ago, you don't have a corrective action. You have organizational folklore.”
I think i’m going to call our ticket support system “folklore” from now on. It sure is used that way.
This is a really really smart SVP. The answer to “how do we prevent this in the future” is to identify decisions that reduce that problem from happening again. This doesn’t have to be perfect. You can say “we are going to do X next time” and also say “but we don’t know how far X will work”. When the failure happens again, you can retire process X and move on to Y. The aim is to make a series of decisions that eventually get to the heart of the issue.
Everything else is a conversation that takes up space on a post-mortem or runbook.
> What they didn't want was for empathy to become the mechanism by which the organisation absolved itself of having to change.
That line is absolute gold
The assumption that something always has to change is the culture that leads startups to knee jerk their way into miles of red tape and performative bureaucracy
So in summary, instead of details propose how to change the system so that instead of hearing perfectly reasonable details about why the current system failed, go straight for changing the systems to prevent the chance of failure again.
I like this plan of action, it removes focus from what happened in the past to how can we prevent it from happening in the future.
There's a reason 5 Why's Analysis is a thing. If you just say "here's a bunch of stuff we're changing," but you haven't said enough about the root cause, then no one will understand whether we're doing enough (or too much; or the wrong things). I like the framing of being empathetic - everyone is smart, doing the best they can with their knowledge and experience at the time - but sometimes we didn't do all the right things, and getting feedback from more senior engineers helps drive that improvement.
> There's a reason 5 Why's Analysis is a thing. If you just say "here's a bunch of stuff we're changing," but you haven't said enough about the root cause, then no one will understand whether we're doing enough (or too much; or the wrong things).
No, that's not the point at all.
Root cause analysis and context and tradeoff analysis and plans are still critical requirements.
But not everyone needs to do root cause analysis and evaluate tradeoffs and review plans. Some stakeholders don't need any of that. They pay you to care, so that they don't need to.
What these stakeholders care about is whether you can fix things, how are you planning on how to fix things, how many resources you need to allocate to fix things, and when are things fixed.
Different audiences require different messages because they have different concerns and responsibilities.
Does a project manager need to know the failure rate of a DSL connection of a customer that reported an outage? You might, but does the project manager need to?
Exactly. The point of the article is that the boss just needs to know that you have a real plan, not 3 Hopes in a trench coat.
I've never seen an incident report or postmortem that didn't include action items to cover the core issue noted in this writing: "can we / how do we prevent this failure mode from recurring?"
I'm not saying these moments of discovery / epiphany aren't valuable, they are and this retelling is enjoyably written.
I am saying that action items borne of incidents should be de rigueur.
Right, this is not even table stakes, it’s the whole point of a post-mortem.
When you have a bug in a program, you fix it. This is just a variation of that incredibly obvious “advice” applied at a slightly different level.
What if you work at an organisation that doesn't ask "How do we prevent this from happening again?"
Organizations don’t choose whether or not to ask questions. People do. The behavior of an organization is the sum of the actions of the people in it. If you are a member of an organization, and you ask ‘how do we prevent this from happening again?’ Guess what? The organization just learned how to ask that question.
Now it just needs to learn how to answer it, and act on the answer.. but step one seems solvable.
That feels like a tough place to work. I would worry about burnout and churning employees if I didn't work at a place that tried to improve every day.
Maybe I'm a bit of a pollyana, but most places I worked had folks who cared and wanted to improve things.
>most places I worked had folks who cared and wanted to improve things
Where are these places? It feels like they are hard to come by, and their hiring requirements can be highly competitive
Interesting. At our org we always do both. Root cause, what are we changing. I assumed that was standard practice.
If you just explain what happened and why, that's fine but...how are you going to make sure it doesn't happen again?
The problem I found more vexing as a manager was: how can you prevent this KIND of problem from occurring?
I'd have someone on my team make a technical error and xyz wouldn't work. Wed talk thru it, and they wouldn't make that exact mistake again. But there's literally 10k things that can go wrong in our system, so then a related mistake would happen later.
What they needed was improved pattern recognition vs if this / then that which comes from post mortems.
What can be done, when writing an article, to not stumble during reading when an unknown abbreviation such as SVP occurs?
This reads like a LinkedIn post. I know kagi translate[1] supports "LinkedIn" language - could someone with an account there copy that article and translate from "LinkedIn" to "English"? I wonder if we'd uncover some simpler meaning this way.
Edit: I was wrong. It still reads like LinkedIn to me but “translating” doesn’t help. Thanks in any case.
[1]: https://translate.kagi.com/
SVP sounds like a dingle to me. Author is justifying it with some magical thinking, "Oh he's not being rude, he's being smart"
This is similar to the approach in healthcare, at least my experience in pharmacy.
When an error occurs we find the root cause but in 99% of cases a change to the environment/process is required to avoid it in the future.
Humans are all imperfect, their competence will change hour to hour let alone day to day. But you can control a system and put checks in place (admittedly a human can still do the process wrong, but then you need to think how the process can be clearer).
No-blame culture is very effective at providing an open environment to share mistakes, learn, but most importantly avoid reoccurance.
If I recall correctly, in The Pragmatic Engineer, this was described simply as "provide options, not excuses" and is a critical way of building credibility.
My father who was also involved in IT tells a similar story where the executive cut him off and said "I asked you for the baby, not the labor pains."
Love this idea; won't go over well at all with the other accountants I work with. MY CFO doesn't like graphs because they "hide too many details' and insists on everything presented to her being in a table. MY VP pulls out a calculator and re-does the math on tables in decks by hand to help him 'understand'.
A lot of leaders like to hide in the details because the bigger picture is a harder problem.
Best article I’ve read this month. The SVP expressed unqualified trust in the team. Asking the team for the next steps emphasizes this trust.
Just give me the solution to the problem of Lack of care to understand the details by people who can actually make a difference.
Yes and no.
Solutions aren't always good solutions. We need to drill into the details, but not the problem's details. We need to look at the solution's details.
https://littlegreenviper.com/problems-and-solutions/
>If your corrective action depends on people remembering a conversation from six months ago, you don't have a corrective action. You have organizational folklore.
I appreciate this.
We call this incident postmortem, part of a company process. After serious problems and firefighting, when the patches and hacks and people working 24 hour and it admins wake on a Sunday morning to do work, we sit down and try to figure out how this will _never_ happen again.
> At first I thought "I don't want the details" sounded dismissive.
Well that is because it was delivered by someone who was actually being dismissive!
If they want to know what "we are changing" but don't want to know any detail as to why, even presented at an appropriate level, they are very likely looking for you to take all the risk of the decision and they are dismissing your need to have the necessary informed consent for it to become the we.
Now, if you're presenting that information at a level of detail that is truly irrelevant or deep, that is your problem to fix.
But otherwise be very wary of this pattern of communication where "we" might mean "you". Anyone with any sense of risk management should know not to do this.
If you have not seen it, watch Margin Call. When you get to the boardroom scene [0] you'll see this play out, but you will also see the (pretty rapacious, direct) boss understand properly that the information he is getting from a low ranking employee is nevertheless his problem to understand, even if only at a 10,000 feet level.
Beyond the drama of the way the scene ends, it's actually a very good model of how that interaction should play out for everyone — making it clear that everyone in the room shares in the "we" of it, including the decisionmaker.
I have been in rooms where it played out at least something like that, where the boss wants to make an attempt to share the burden of the detail, and I've been able to leave them feeling somewhat supported, or at least without feeling like I was being shafted.
And I've been in rooms with people who don't want to know the details but just want to know what "we" are going to fix — those guys have never been as good to work for. If they say "we" but leave you with the impression they may mean "you", it's bad.
[0] said scene, but really, really, watch the film if you never have:
https://www.americanrhetoric.com/MovieSpeeches/moviespeechma...
Thats funny. When I try to only give the conclusion, I'm often asked for the details.
How about explain the problem and solution in 3 to 5 levels of details in a top-down hierarchical manner sort of like a decision tree. Let people decide how far they want to drill down.
> [The Manager] continued:
>> I know that if we get into the details, the reasons will be perfectly reasonable. You'll explain what happened, I'll understand why everyone made the decisions they made, and I'll empathise with you.
>> Then it'll happen again.
>> So I don't want the details. I want to know what we're changing.
Even if the manager's approach was correct, this is just a hurtful way to frame it to your team. It's great that the author could reverse-engineer what the manager was doing, but it shouldn't come to that.
Why couldn't the manager say something like:
> You are all great engineers, so I already know that everyone made the best decision available with the information they had at the time. What are we going to do differently moving forward?
I'm still not sure I agree with the strategy, because I don't see how you can understand what happens next without understanding what went wrong. But I can see how the framing might help steer the discussion.
If you don’t know what happened how would you be able to tell if the changes are addressing the problem? If you don’t care about the problem being addressed why jump in a call in the first place?
Learned this doing incident reviews. If the exec asks for details it usually means they don't believe you yet, so "skip to what changes" is actually the good outcome.
Post-mortem, 5-why, retrospection - all this to understand what and how it happened so the org may introduce change so that it doesnt happen again. Then another edge case happens and the cycle repeats. Humans do this cycle all the time - big orgs invented the above structure for this process to be visible and collaborative and so that the rest of the org can learn too. Then you put execs on top, they will come, ask some questions so they can sleep better at night. Corporate playbook, some people hate it, the rest understands it… and plays the game.
Idiotic article. Nobody collects "why"s so they can not change anything. Not changing anything is never the failure mode. The failure mode is changing the wrong thing. If you are a leader, and something egregious goes wrong, it is your job to ensure that your people don't just put some bandaid fix on it that will keep causing issues indefinitely. Understanding the complete "why" is a precondition for that. Not bothering to hear a complete explanation because you trust your people to do the right thing is fine, but then why were you on that call in the first place? Let them do their job.
The SVP needs something to report back to his manager and that’s it.
We write up timelines. We reconstruct decisions. We explain dependencies. At the end of it, we hand over a document that contains the specific combination of events that led to the incident. Everyone nods their head, says "that makes sense", and we all move on with our day.
...huh? I've been in corporate America for 19 years now and I've never once had this happen. I've never seen anyone just look for the cause and not ask what can be done to prevent it in the future. Everyone KNOWS that question is coming next.
This is just a LinkedIn post.
> ...I'll empathise with you. Then it'll happen again.
I'm not so sure this means "I trust you."
How you can have an informed conversation about change without hearing the problem out?
If you trust someone that much, why are you even in the loop? Why are you badgering them at all? This is just bad management pretending to offer underlings autonomy. As soon as it fails, everyone loses their job.
> Understanding an issue is not the same as fixing it. A good explanation can make things worse.
I couldn't disagree more. "A problem well stated is a problem half solved."
Is this Five Whys from First Principles?
I thought keeping details away from c-suit regardless of how technical these people are was the standard thing to do. Your job is to make these guys work the minimum. We fucked up, we fixed it and it won’t happen again. They don’t need to know anything else. You could upload the forensics report to a share drive and make the link available if they want to see it, but most times they won’t bother.
I can understand the inclination, but actually I /do/ want the details, because without the details I cannot consider and measure if the change proposed is a reasonable pathway to help manage the class of problem. Many times, people in organizations while well-meaning are siloed and focused specifically in their own niche and therefore will propose changes that cannot be coherent in any holistic manner across all the parts of the organization that would need to be affected by the change. That means you /do/ need to understand the details, and also have a big picture view.
The executive is there to help with the big picture, but you can't see the big picture if the canvas is blank.
Your svp is a bad manager
Why are these SVPs even needed anymore? They seem to do nothing of value.
If you ask them, of course they will make up a bs story with their fake answer. The fact is that they fail the ablation test.
No. They should always get the full details.
Neurotypical executive toxic attitudes should be shamed.
Making informed decisions based on assumptions or guesses is always going to result in terrible outcomes.
> The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".
Apropos, I was once told a Japanese saying: "Fix the problem, not the blame."
I must have repeated this charming story a dozen times before realizing it makes no sense in Japanese -- it relies entirely on English idioms.
SVP seems like such a useless appendage in the story.
He doesn’t want the details, he just wants someone else to tell him what they’re gonna do. But if he doesn’t know the details, he won’t understand if the plan will make things better or worse.
So… what exactly is this presumably valuable person there to do? Just to gage the confidence in the room? Presumably everyone was confident before the mistake happened.
What does this person who just wants a solution add? Why isn’t he fostering a discussion and adding his insight? Why is he in the room at all?
If I'm to explain what needs to change, that will by definition address processes or lack thereof that led to the problem, no? Which is then explaining what happened and why in an indirect manner, so many of the proposed changes may not make sense without more context.
I would not have said that. I would have asked for the details and tried to find the gaps because, almost always, something could have been done. Then the question is how much risk reduction are you willing to pay for.
Isn't that the same with just more steps? Seems the svp was focusing on what they will do differently next time
Presumably the SVP joined the meeting because the incident was significant. The team can present the options but the executive may need to make a judgement call, and he can't do that responsibly if he turns his brain off. For instance, what if it happened due to understaffing or things falling between the cracks among teams, possibly indicating inadequate org design? Such things are the responsibility of management.
The quote box at the start of the article is the essence of it, the rest is AI-speak dressing it up.
TLDR:
Some outage causes a meeting with an SVP. SVP preempts the discussion with the following:
“I know that if we get into the details, the reasons will be perfectly reasonable. You'll explain what happened, I'll understand why everyone made the decisions they made, and I'll empathise with you.
Then it'll happen again.
So I don't want the details. I want to know what we're changing.”
i don't want to sound like a douchebag but isn't figuring out what went wrong and how to prevent next time kind of an obvious goal
Bootlicking apologia for a bad boss.
I learned this myself while yelling at AI.
I get triggered from these Linkedin bs type of articles. In the real world time and money is limited and everything gets prioritized. I worked at corps were MOST of the incidents reoccurred cause fixes was never prioritized. Cheaper to have people on call than to fix stuff.
I'm giving you the details because I'm going to give you options for solutions and I need priority information about those solutions, and you need to be informed about the trade-offs and costs of each option because you have better knowledge of our overarching objectives than I do.
If our overarching objectives don't impact our on-the-ground decisions, what are you even for?