Every time I get in a conversation about server patching, I think about my grandmother. My grandfather was a plumber, didn’t make a whole lot of money, and was “pretty hard” on his clothing. She was forever darning his socks, repairing the holes that would mysteriously appear almost on a weekly basis. Her thriftiness and talent probably saved hundreds of dollars over the years and gained her the nickname “the patch queen.”
When it comes to security patching, I think Grandma could teach a few IT executives a thing or two.
Patching reluctance is a cultural thing but reluctance to patch will lead to security incidents. The speed at which vulnerabilities are discovered and weaponized is measured in days, not months or years.
So where does that reluctance come from? Grandma might call them the three deadly sins.
#1: We can’t risk the downtime (during the patch or from unanticipated failures).
Grandma didn’t mince words so her answer to this one would be pretty simple. If it’s so important, you can’t afford NOT to patch it.
If the box in the corner that can’t be shutdown is so critical to your business, it deserves investment to make it more robust. If you’re concerned about the services not coming back up after patching, again you should be focused on investment to modernize it. A service or system that crashes when you look at it sideways, isn’t a service on which you can run your business.
Rebuild, replace, upgrade… but realize patching isn’t the issue. It merely exposes something you need to address anyway. Are you concerned about the patch itself having faults? You have to have trust in your vendors. We all have to patch at a dizzying pace these days. The world feels that fear with you. It’s no excuse not to patch. There are protocols you can put into place such as patching a clone of the system first and testing thoroughly or identifying some early adopter users who get the patch first. Either way, if a service is so critical to your organization, you should really be investing in those controls anyway.
Many years ago, we mandated that all Windows systems under our control would be patched and rebooted automatically via automation during an established maintenance window. Every month, thousands of servers, desktops and laptops, patch and reboot without a peep, while we sleep. Yes, occasionally Microsoft slips us a curve ball but it’s still far better than our previous patching strategy of ad-hoc patching and rebooting. Servers and services that did not behave were targeted for replacement. That gave us the peace and energy to turn our attention to patching Linux and VMware systems and accomplishing that, turning our attention to IoT, appliances, and anything with an IP address. Today, we have a comprehensive patching strategy covering all devices. We matured and we urge you to mature as well. Cybersecurity demands it.
#2: If it’s not broken, there’s no need to patch (primarily relating to non-Windows devices).
Grandma wore glasses but even she could see a short-sighted argument when it was presented. Most new patches, whether it’s a switch OS, a BIOS, a web server, or a library, are riddled with security fixes. If you have the time to review the changelog and validate that the fixes do not apply to your environment, wonderful! But you better get it right. Otherwise, just apply the patch and move on with your life (because another patch will be along shortly)
I get it. Network devices are tricky to patch but here’s where the culture thing comes in. Your CISO and CIO need to participate in the culture of patching and let the business agree upon and adapt to maintenance windows. If the business cannot, then that should naturally lead to investment in infrastructure that can be patched with minimal downtime. See #1 above.
#3: We don’t have the time.
Benjamin Franklin coined the phrase “a stich in time, saves nine.” Grandma would tell you that if you’re too busy to do it yourself, find some else who can. Outsource it. We’ve heard from CISOs repeatedly that outsourced patching is a lifesaver. Someone is accountable! It’s literally the thing they get paid to do and they will do it well (or find another vendor who will). CISOs don’t have to rely on the IT team, which is usually over-worked and not under their control, to do the patching (and accept the excuses when they do not). Hire a service and hold their feet to their SLAs. The patching will get done.
PATT: Patch All The Things!
I miss a lot of things about my grandmother but I miss her practicality the most. Our world is infinitely more complicated than it was in the days she sat out on the front porch with her needle and thread but in so many ways, things still come down to an old-fashioned dose of common sense.
Patching cannot be ignored and the cultural changes have to go all the way to the top (but don’t have to start there!!). If you’re scrambling to patch (and praying everything comes back up) every time the headlines light up with another vulnerability, it’s time to try a different approach.
We scramble too but I know the SANs are going to patch just fine because we’ve practiced on them for years! We know the servers are going to work, again because we’ve practiced on them for years! The ones that misbehave have been kicked out. The vendors that misbehave have been kicked out. The business knows that when patching is being done, it’s because it’s very important and doesn’t get skipped. The business knows that when a device is so old it cannot accept patches anymore, it’s destined for the e-cycle bin.
Grandma patched until the holes got too big. Lesson learned.