Everything Breaks at Scale

Get in gear for growth before it arrives

A few weeks ago, I was on-site with a fast-growing company we’ve been working with for a few years. They’re on a tremendous growth trajectory, with annual sales increases of 30-35% over the last few years. This year, they are on track for even bigger gains. It’s not far-fetched to predict they might double their revenues before too long.

Their incredible success is exciting to be around and support. But it can also be nerve-racking.  Early this year, just as their team was preparing for their biggest order ever, from a major national customer, a critical piece of production equipment failed. I don’t mean a temporary outage or a simple repair job, either. The machine failed completely…down for an indefinite time waiting for replacement parts.

And in that moment, we got to watch a universal truth play out in real time: Everything breaks at scale.

Things don’t break at scale because people are incompetent, or the company’s strategy is flawed. Things break because the systems, processes, and tools that worked at one level can’t cut it at the next one.

Simply put: What got you here won’t get you there.

The Swiss cheese moment

In safety science, there’s something called the “Swiss cheese model.” Catastrophes don’t usually happen because of one failure. They happen when multiple small gaps line up at the same time, like holes in a stack of Swiss cheese slices. 

That’s what scale does: lines up the holes. Here’s what that looked like for our client:

• A critical machine went down.

• They didn’t have the critical spares on hand.

• Order tracking was largely manual.

• Logistics scheduling lived mostly in peoples’ heads.

• Severe storms disrupted suppliers and trucking.

On their own, any of these hiccups would have been manageable. But when they hit all at once, as the company was on the verge of a make-or-break order, it was a real crisis. 

Regardless of your industry, if you’re a growing company, there’s a key lesson we can take from this situation: none of those weaknesses were visible while everything was running well. Growth, and the distractions and multiple new pressures it brings (along with new opportunities), masked fragility. (I just finished lunch with a coaching client leading a fast growing people services firm. He’s also experiencing that “everything breaks at scale” and there’s not a piece of manufacturing equipment in sight!)

It’s NOT the People

At one point during the crisis, we discovered that several outbound trucks had been scheduled, but had not entered them into a central system (which was just an Excel spreadsheet). They were all in the shipping coordinator’s head, which meant there was no visibility tracking. Not surprisingly, it resulted in chaos as delivery trucks arrived for orders that hadn’t been picked because no one in the warehouse knew about them. 

The easy reaction would have been to blame the shipping coordinator. But that would have been unfair, because what actually happened was a process failure from an outdated system, not a people failure or incompetence.

When a business doubles, and the workload quadruples, the old systems need to evolve with it. When they handled a few trucks a day, Bethany’s manual system and her memory were sufficient. But handling dozens of trucks a day and growing? They needed to invest in new processes and tools, including scheduling software, to keep everything running smoothly. 

So the next time something in your organization breaks at scale, resist the instinct to find a villain. Instead, ask: What system do we need to build now?

The key thing that didn’t break

Spoiler alert: this story has a happy ending, in part because our client had made the decision  the year before to invest in their leadership team, and get the right people in the right roles with the right capabilities. This is a fundamental of Get in Gear, our framework for aligning strategy to execution and, ultimately, results. When the crisis hit, our client already had in place:

• A strong Chief Operating Officer

• A seasoned VP of Operations

• A highly capable CFO

• A highly engaged, dedicated team

So when the production line went down, those leaders didn’t panic, turn on each other, or otherwise melt down. They stayed calm, rolled up their sleeves, and got to work. Here’s what I saw (and it was impressive!): 

• They built temporary tracking systems in Excel.

• They managed inventory pallet-by-pallet.

• They worked on supplier relationships.

• They rebuilt schedules, and adjusted them multiple times per day.

• They met twice per day in highly focused touchpoint meetings to react to the constantly changing situation and to keep everyone informed.

Collectively, they “flew the plane all the way to the ground,” a metaphor a former military pilot shared with me. When you’re losing control, you don’t give up. You keep flying the plane, assuming you will find a way to make a safe landing. For my client, their leadership was able to land their proverbial plane, meeting every delivery.

This time, everything broke at scale. Except their team. 

Getting growth-ready

Growth exposes weaknesses you didn’t know you had, manual processes that once worked become bottlenecks, and heroic efforts like the one I described above lead to burnout. 

If you are growing and you’re not proactively upgrading your people, systems, and operating architecture, you are building fragility into your future. Because the crisis will come. The only question is whether you’ll have the capacity to manage it when it does.

In my next post, I’ll outline how to prepare for scale with some fundamentals of our Get In Gear framework, so that when things break (and they will), your business doesn’t.

Everything Breaks at Scale......