I’m going to tell you about a ransomware attack we helped recover. This wasn’t one of our managed services clients, which is part of the story, honestly. We’d done a mail migration for this manufacturing company about a year earlier. Good team, good business. But they weren’t under our watch day-to-day.
Their sysadmin worked out of state and remoted into everything. One morning he calls us and says, “Hey, one of my servers is offline. Can you send somebody over to check it out?”
So we send an engineer on-site.
It Wasn’t an Offline Server
Within seconds of walking in, our guy knew this wasn’t a hardware failure. It was ransomware. The ransom note was sitting right there on the screen.
The on-site technicians hadn’t recognized it. They just saw a server that wouldn’t come up. I don’t blame them. If you haven’t seen it before, you might not know what you’re looking at. But our engineer had seen it plenty of times.
Here’s the thing about ransomware: the first hour matters more than people realize. Not because you can stop it (by the time you see the note, the encryption is done). The first hour matters because of every decision you make after that.
Insurance, Attorneys, and a Forensics Team That Needed Hand-Holding
The first thing we told them was: call your insurance company right now. Not tomorrow. Not after you’ve “assessed the damage.” Right now.
Why? Subrogation. If you start touching things, recovering things, making changes before your insurance carrier is looped in, they can deny your claim. I’ve seen it happen. You think you’re being proactive, and you’re actually voiding your coverage.
Their insurance company gave them a list of pre-approved attorneys and a list of pre-approved forensics companies. Standard process. The problem is, you’re picking names off a list under extreme pressure. There’s no time to vet anyone. You just point at a name and go.
It took three full days before the forensics team was ready to begin recovery work.
Three days. For a manufacturing company is an eternity.
This business was losing roughly $100,000 per day in downtime. Production was stopped. Orders weren’t shipping. And we’re all sitting there waiting for the forensics company to get their people together.
The Forensics Company Was… Not What You’d Hope
I’m here to tell you, the forensics company was underwhelming. Their first instruction was basically: “Start turning your computers on and we’ll install our software.”
Our engineer looked at me and I looked at him. We suggested booting into safe mode first, so the ransomware wouldn’t re-execute and encrypt anything that might still be clean. The forensics team hadn’t thought of that or at least conceded that was the best first step.
From that point on, we were essentially leading the recovery while the forensics company followed along. We held their hands through the technical work. That’s not how it should be, but that’s how it was.
The Good News: Backups Existed
The single best thing this company had going for them was recoverable backups. When we confirmed the backups were intact, that was the first moment I could take a breath.
But here’s where it gets complicated. You can’t just restore onto the existing hardware. Insurance and forensics need to preserve the state of every infected machine for their investigation. Those servers and workstations become evidence. You don’t touch them.
So now you need hardware to recover onto. Where do you get servers on zero notice in the middle of a crisis?
Cold Spares: The Thing Nobody Has Until They Need It
This is something we do that most IT companies don’t. ArcLight maintains over $100,000 worth of cold spare servers in our inventory at all times. Racked, ready, waiting for exactly this scenario.
We provided two high-end loaner servers at $1,000 per month each. The client used them for three months while they purchased and configured their permanent replacements. Without those spares, they would have been sourcing servers during a crisis, which adds days or weeks to an already devastating timeline.
Rebuilding Everything
The forensics investigation ultimately found that the attackers got in fast and encrypted fast. They didn’t embed deep into the environment or set up persistent backdoors. The backup window was clean, with less than 24 hours of exposure. That was the best possible finding, given the situation.
But they still didn’t feel confident without rebuilding almost every computer in the company. When ransomware touches your environment, you never fully trust those systems again. It’s not paranoia. It’s just reality. You don’t know what’s sitting dormant on a workstation.
The remote employees were the hardest part. Laptops had to be shipped back, wiped, rebuilt, and shipped back out. It was incredibly time-consuming.
Some servers weren’t in the backup set, so they had to be rebuilt from file copies found on other systems. They had an in-house developed CRM application that could have been catastrophic if lost. Fortunately, it was recoverable, but that was a cold sweat moment.
The Hidden Cost: Your Reputation
Here’s something that doesn’t show up on a spreadsheet. Once you’re down for more than 24 hours, you can’t hide it. Customers call and nobody picks up (if the network is down your phone system is likely down also!). Orders don’t ship. Emails bounce. People talk.
The financial damage is measurable: $100K a day, three-plus days of downtime, months of cleanup. But the reputation damage? That’s harder to quantify and harder to repair.
What I’d Want Every Business Owner to Take Away
- Your insurance process will be slower than you expect. Three days before recovery even starts is not unusual. Budget for that reality.
- The forensics company on your insurance list might not be great. You’re picking a name under pressure. If you have an IT partner who’s been through this before, they can help navigate, but you’re still constrained to the approved list.
- You need hardware ready to go. You cannot recover onto infected machines. If you don’t have spare servers available within hours, you’re adding days to your downtime at whatever your daily loss rate is.
- Backups are everything, but only if they’re tested and complete. This company’s backups saved them. But some servers weren’t backed up at all. If their custom CRM had been on one of those unprotected servers, we’d be telling a very different story.
- Managed, proactive security changes the math entirely. If this company had been under active monitoring (endpoint detection, 24/7 alerting, regular vulnerability patching), there’s a very good chance this attack never succeeds in the first place. And if it does, the response starts in minutes, not days.
What ArcLight Provides That Would Have Changed This
We build our managed services specifically for this scenario. Not because ransomware is theoretical, but because we’ve walked into these rooms and seen what it does.
- 24/7 endpoint monitoring designed to catch threats before encryption starts
- Managed backup and disaster recovery with periodic restore testing, not “we think the backups are running”
- $100K+ in cold spare servers ready to deploy within minutes
- Incident response experience: we’ve been through this enough to lead the recovery, not watch from the sideline
- A relationship that exists before the crisis, not a vendor you’re calling cold while your business bleeds $100K a day
If you’re running a business and your IT strategy is “we’ll deal with it when it happens,” I’m here to tell you: “when it happens” looks like this. And it doesn’t have to.
Brian Largent
Father to five, husband to one, founder, CEO, and all around swell fella (or so I'm told)
