This is a story about a company that came within 15 minutes of losing everything. And then learned absolutely nothing from it.
I don’t say that to be harsh. I say it because it’s the most honest example I have of how businesses actually calculate risk, and why so many of them get hit more than once.
The Setup
A manufacturing company (I’m keeping this one vague on purpose, because they never filed a public claim and never made the news) had a storage area network, a SAN, that a third-party vendor had originally set up. Their internal IT team managed it day-to-day, but they hadn’t built it and didn’t know every detail of the configuration.
They got ransomware. All their servers were encrypted.
Their team did what you’d expect: they bought servers from another data center and started the recovery process. It was slow. Painful. The kind of recovery where you’re pulling data from wherever you can find it and hoping the pieces fit together.
One of our engineers and I were on-site helping. We’d been there for the better part of 24 to 48 hours at that point. Not sleeping much. Doing what we could.
The Question Nobody Thought to Ask
About eight hours into the recovery effort, I asked a simple question:
“Did you check the snapshots on this SAN?”
The room went quiet. Their team looked at each other. “Oh no, I didn’t even know they did that.”
They didn’t set up the SAN. They didn’t know the third-party vendor had configured automatic snapshots. As far as they knew, everything on that storage was encrypted and gone.
We logged into the SAN console. Snapshots existed. Pre-ransomware snapshots of their server volumes, sitting right there.
Here’s the part that still gets me: those snapshots were on an automatic rotation schedule. They were going to be overwritten, replaced with new snapshots of the now-encrypted volumes, within approximately 15 minutes of when we found them.
Fifteen minutes. If I’d asked that question during lunch instead of when I did, those snapshots would have been gone. The company’s servers, the critical infrastructure for a 1,500-employee operation, would have been unrecoverable through that path.
The Recovery
We stopped the snapshot rotation immediately. Then we restored the servers from those pre-ransomware snapshots.
The workstations were still lost, and those had to be rebuilt. But the servers were the critical piece. That’s where the business lives: the applications, the databases, the file shares, the line-of-business systems that 1,500 people depend on every day.
We got them operational. It wasn’t pretty, and it wasn’t fast, but it was a fundamentally different outcome than “start over from nothing.”
The Frustrating Part
Here’s where the story takes a turn that I wish I could tell differently.
This company dodged a bullet. They were 15 minutes from catastrophic, unrecoverable data loss. The kind of loss that shuts businesses down permanently.
And then they went back to operating more or less the same way.
No significant investment in security. No managed detection and response. No real changes to their backup strategy. They’d survived, and because they survived, the internal calculation shifted. The pain faded fast.
I’ve seen the CFO math on this. For a 1,500-employee environment, proper security runs somewhere between $100 and $300 per secured user per month. Call it $60,000 to $80,000 a month for meaningful protection.
The calculation goes like this: “I can spend $60-80K a month on security infrastructure, or I can let insurance pay it out every five or six years when something happens.”
On a spreadsheet, that math can look rational. In the real world, it’s a bet. And the thing about bets is that they work right up until the moment they don’t.
Luck Is Not a Security Strategy
I think about this company more than almost any other client interaction I’ve had. Not because of the technical challenge (finding those snapshots was just experience and knowing what questions to ask), but because of what happened after.
They had the clearest possible demonstration that their environment was vulnerable. They experienced the downtime, the chaos, the all-night recovery sessions. They were 15 minutes from a scenario that might have ended the business. And they chose to keep rolling the dice.
It’s shocking how many companies operate this way. They treat cybersecurity like a lottery: “it probably won’t happen to us, and if it does, insurance will cover it.” But insurance doesn’t cover the lost customers. It doesn’t cover the reputation damage. It doesn’t cover the three days your 1,500 employees can’t work. And it definitely doesn’t cover the scenario where your snapshots rotated 15 minutes before someone thought to check.
If your security strategy depends on someone asking the right question at exactly the right moment, you don’t have a security strategy. You have luck. And luck runs out.
Brian Largent
Father to five, husband to one, founder, CEO, and all around swell fella (or so I'm told)
