Software Updates: Building a Resilient (and Repeatable!) Strategy

We all recently witnessed the consequences of a software update gone wrong — the Crowdstrike outage, while brief, caused havoc across the globe. The issue stemmed from a faulty software update to Crowdstrike Falcon Sensor, but rather than diving into the specifics of that incident, I want to talk about how this could’ve been prevented — a takeaway that any modern company can benefit from. 

The Increasing Reliance on Software

Today’s businesses rely on devices — both for employees and customers. But let’s be clear: it’s not just about the hardware. The software that powers mission-critical applications and processes drives our critical operations and customer experiences. This dependence on software is here to stay, fueling business into the future.

So, what happens when those devices fail? The answer is disruption, sometimes on an unprecedented scale. We often take for granted how integral these devices are to daily operations until they go down, leaving us without alternatives. Business grinds to a halt, plunging operations into chaos. 

Most of us likely remember the threat of Y2K at the turn of the millennium. It was potentially global chaos, but fortunately, we were able to correct the issue through foresight and extensive preparation. However, not all threats can be anticipated. 

The need for reliable software deployment tools has never been more critical. 

The Importance of Safe Software Deployments

Our dependence on software makes glitches and errors inevitable. It’s clear how issues can arise, but difficult to understand how we allow them to happen. While there isn’t a specific answer there, there are a few areas of consideration. 

• Rushed software deliveries: When software updates are pushed without rigorous testing and verification, the possibility of mistakes grows. The bigger the update, the more potential there is for disaster, but make no mistake — small updates can break things just as easily. 
• Complacency: Repetition builds a false sense of security. We assume nothing can go wrong… until it does.
• The wrong tools for the job: Sometimes, it’s up to software developers and IT teams to make a square peg fit into a round hole, leading to inefficiencies and problems..

The solution lies in developing a repeatable, scalable software delivery model. Whether it’s a new application, an app update, a security patch, or a full system software update, having a vetted and verified process in place is critical to prevent issues of all sizes and severity. 

For some companies, this might mean rethinking software deployment and updates from the ground up. For others, it might mean small tweaks to internal processes or tools. 

Best Practices for Resilient Software Deployments

Regardless of your current position on software deployments, there are proven processes that can enhance your approach so you can scale with confidence and update without fear.

• Assume the worst: Always assume that your update will break something. Proceed with caution, even beyond your test devices. Which brings me to the next point.
• Test, test, test: Your test fleet should ideally mirror everything you have in the field. The more comprehensive your test fleet, the fewer issues you’ll see in the field.
• Ramp up slowly: Start with your test fleet, then roll your updates out to a larger (but still small) portion of your fleet in the field. Maybe that’s a single location or area, or maybe a device type. The key is to make it repeatable. 
• Stage rollouts: This is critical! Even with successful testing and a small-scale rollout, problems can still arise. That’s why staging updates over days or even weeks (depending on fleet size) minimizes the risk of widespread disruptions. 
• Continuously monitor: As your updates progress from testing to staging, continuously monitor device status and performance. If an issue surfaces, you can halt the rollout, correct the issue, and then resume. 

Ultimately, the key to successful, resilient, repeatable, scalable software delivery is staging. From a small test suite of devices to your full fleet, staging all software updates is the key to successful rollouts (and, similarly, preventing disaster). 

Why Your MDM is a Much-Needed Safety Net

You might wonder how to achieve this. Maybe you already have a software deployment solution in place, or maybe you’re starting from scratch. Regardless, your secret weapon for repeatable, reliable software delivery is your MDM. 

Of course, that hinges on having the right MDM for your needs. The right MDM will provide you with the critical software deployment tools to deliver your software successfully, minimizing the risk of major issues. When you have a device management tool that focuses on reliable software delivery with real-time monitoring and staged software updates, you can deploy with confidence. 

But effective device management goes beyond software deployment — post-update support tools are just as critical. The ability to remotely troubleshoot devices in the field eliminates the need for costly on-site visits. Meaningful ways to not only update software, but device configurations and uses offer a way to seamlessly optimize device functionality without resetting (or, in some cases, even restarting!) the device. 

This is the foundation on which Esper was founded: repeatable software delivery, scalable remote tooling, and everything the modern fleet manager needs to optimize devices efficiently and at scale. Our DevOps approach to device management brings critical tools that device fleets were missing. 

Our company mission is to power exceptional device experiences. Part of a curated, secure device experience is regularly updating software, and we aim to make that as easy, repeatable, and safe as possible. We strive to minimize potentially catastrophic issues with your devices by providing the tools for the most effective device management possible. Our priority is to support our customers and all of their needs.