Esper Deep Cuts: The MDM Flip

At Esper, we specialize in building a platform and offering management tools for edge, dedicated, and fully managed device solutions for Android and iOS. We pride ourselves on understanding endpoint OS behavior, which makes our management of these devices precise and reliable. We even build our own Android distribution for dedicated devices, called Foundation. This means we have capable AOSP engineers on staff, which can come in handy for certain customer situations. 

A common situation we encounter is a fleet already managed by a Device Owner MDM, meaning switching device management infrastructure requires a factory reset for every device in the field. 

The ODM may supply the incumbent MDM, a common scenario in AOSP. Or the customer may roll their own MDM and then discover that the engineering investment required to maintain an MDM does not make sense for their business. In both cases, the MDM is typically built into the OS build — about as locked in as you can be. That’s where Esper’s Android OS expertise can come into play.

Deep Cuts: The MDM Flip

You know the story. Your solution requires a fleet of dedicated Android endpoints. Requirements are straightforward, and you can get a better deal by working directly with an ODM to source your hardware, which typically means AOSP. You are good with the advantages of using AOSP, and are okay with the drawbacks of not having Android Enterprise. The ODM happens to have an off-the-shelf offering that nicely fits the bill. Plus, they will even give you a great price on device management and OTA, built into the device’s OS build! Wonderful!!!

Now you are rolling, your business is building, and you discover that the supplied device management is woefully inadequate for the scale and sophistication of your booming business. Piling on to it all, you must add new use cases requiring different hardware configurations. With your growing business comes the ability to spend more per device to get what you want, moving you away from doing business with that ODM. You discover Esper, which is perfect for what you need to be able to grow, giving you the flexibility to manage many different Android device types and use cases, plus iOS, with one pane of glass.

Except now you have two panes (one being a pain, perhaps) — the ODM MDM system is closed off, and you can’t remove it because they baked it into the OS image. The same situation applies if it happens to be your own MDM that you built and maintained. Are you stuck? Maybe not.

Let’s look at a case where an Esper customer had a deployed fleet at a wide variety of business locations and needed to convert to Esper-managed.

Base Requirements

There is a particular set of circumstances that enable a seamless flip from an incumbent MDM to Esper to manage these types of devices:

• Access to the OS source
• An OTA service in place, enabling updates of the OS for devices in the field (with the potential for exceptions)

Conceptually, the process is straightforward. Esper’s Android engineering team gets access to the source and builds an OTA package that removes and replaces the current MDM with Esper. The ODM pushes that firmware update to the devices in the field, and after installation, they’re ready for Esper.

 We also build a full OS image so new devices can ship Esper-ready.

This means that we need to cover each unique hardware configuration with a distinct OS build and that all devices in the field should move from a single reference OS build to the new one. Thus, if a set of the same device model is not on the same OS version, they need to update before performing the move, or we have to build multiple OTA packages per OS version with corresponding Android OS source drops and apply the right OTA package for each resident OS build on the device. Luckily, we were not in the situation here — it was a consistent single OS build. We do have the means to handle a heterogeneous OS build environment — it just requires a lot more engineering and operational work by Esper which ultimately increases the overall cost.

Sounds simple! Let’s check out the operational side of it, connecting with reality, so to speak.

Challenge: Getting the Source Code

When buying from an ODM, most customers don’t even think about contractually covering getting access to the Android OS source code. Once the initial hardware sale is done, the ODM loses economic incentive to provide the Android OS source. Another common consideration is that the ODM is hesitant to show the state of the source code—the changes they made to the OS source, the quality of those changes, and comments are potentially included are there for all to see.

Thus, the customer must have some means of incentive with the ODM to share the source code with the proper licensing if they didn’t request it in the upfront contract. In this case, we worked with the customer to help them build the case to entice the ODM to share the source --  it was securing a follow on hardware order. 

The key lesson here is that you should have this discussion before placing the first order. It doesn’t mean you actually ask for the Android OS source, but at a minimum, you have contractual guidelines in place for obtaining the source, such as an agreed-upon price that may diminish as you meet hardware purchase volumes.

But without that done upfront, it can be very tricky.

The other part is making the ODM confident that having an outside party peek into its source code won't burnish its reputation. Esper has a set of longstanding Android OS experts, so we were able to build respect and trust with the ODM that by supplying the source code to Esper, they don’t risk industry burnishment.

Once the source code is in hand, it’s “trust but verify” — which means scanning it for anomalies or nefarious code. This has happened before, with Android OS builds continuing nefarious code slipping by and reaching the market.

In this particular case, the target device was Rockchip silicon, which has a reputation all its own. We got the source code via a hard drive shipped from Asia, classic sneaker net, which takes us to what’s next!

The other consideration is if the ODM needs to update the Android OS in the future. It’s relatively uncommon for ODMs to provide security patches, so this is more for the customer to address any bugs that crop up. In order to do this, the ODM needs the source code back so they can modify it as needed and build an update package. We were able to accommodate that with the ODM via the customer in this particular situation. Once we finished our work, the ODM could still maintain the deployment as needed by the customer.

(Sidenote: if you want regular security patches on AOSP and a robust, USA-hosted OTA service, that’s Esper Foundation for Android.)

Challenge: Access to the OTA Service

Often, the ODM provides its own hosted OTA service. That means you can't do anything remotely, even if you have an update, unless the ODM is willing to push out the OTA update package. Updating via USB stick, while possible, is operationally impractical. So, hand-in-hand with getting the source is also obtaining the commitment from the ODM to perform the required OTA push, which also includes precise targeting during the validation phase to make sure it all works. 

This whole flow requires close cooperation with the ODM, the customer, and Esper. We have to handle it carefully.

Modifying the Source Code

With the source code in hand, we had to inspect it and get a feel for the code base. The key was verifying how they implemented MDM and making the necessary changes to the source. Some source trees are cleaner than others; fortunately, this one was not bad. We’ve had others that needed a secret decoder ring to quickly figure out, and we did not have one, so it took much longer than a cleaner source tree. The hard part is that we don’t know until we have the source, but knowing the underlying silicon provides us a clue, as we’ve seen Android source for many different types of SoCs before — at least then we’ve got an idea of what we are getting into.

Creating and Validating the Build

After we make the necessary modifications, it's time to create the Android OS build. First, we have to preserve the application data so the devices remain in the exact state they were in before the update. Not a problem.

Then comes the fun part: validating it on the target hardware. This means we need the proper flashing tooling, which is typically not a big deal as these are relatively standard across the silicon vendors. The bigger deal is having the target hardware in hand and not making any mistakes because flashing can brick a device. In this particular situation, we can resurrect a bricked device by flashing a known good OS build, but that’s not the point nor the goal. The upshot is the customer needs to provide a set of hardware for our Android OS development team.

Once we flash the device, we run a validation test suite. The ODM typically handles this, which is another consideration in the source code negotiation. But in this particular case, we created the validation suite that met the customer's requirements for their very particular solution. Since the modifications were minor, the risk of impacting robustness and MTBF due to the OS was minimal. The OS had already been operating sufficiently in the field for months.

There can be some iteration here if the first build has issues, but in this case, we got it right with build #1 :-).

Generating the OTA Package

After validating the new OS build, we use the OTA package generation tool provided by AOSP to create the OTA package, moving from the deployed build to the new build with Esper device management. 

With that OTA package in hand, we need to validate again, but this time on devices running the target OS build. We do the Initial validation using local installation, so we handled that on our own.

Next was end-to-end testing by having the ODM take the update package and push it out to target the test devices. This is where we all have to be careful. The risk is that the OTA is pushed out to all devices, which can be a disaster. Neither the customer nor their customers are operationally ready for a shotgun deployment. 

We accomplished all the above and were ready to go.

Preparing the Device Configuration Specification (Esper Blueprints)

What is implicit above is that once updated, Esper manages these devices. That means they need to onboard with the required configuration as specified by Esper, yet do so without a touch on the devices in the field.

Esper’s Blueprints enabled the customer to define the fine grained configuration for the deployed devices mapped to each of their end customer deployments. Much more tightly defined and controlled compared to the previous MDM.

We enabled no touch onboarding by including Seamless Provisioning in the OTA package. After an update finished and the device automatically reboots, the device first checks in with Esper and provisions to spec via a Blueprint exactly how the customer needed the device to be configured.

This meant the customer needed the manifest mapping of all deployed devices identified by serial number against the required configuration for their customers, and then appropriately upload them to the proper Blueprints. While not catastrophic, making a mistake here would cause our customer big headaches, so they were very careful to design their Blueprints and Groups to map to their end customer deployments — and they checked it twice!

Once that’s all in place, it was time to move to the next step: deployment.

Staged Update

To minimize risk, we picked one end customer deployment site to test this update. With our customer (Esper was there) we gathered all the devices at the test location and pulled them into a small staging area. Obviously, they all need to have Internet connectivity to receive the update. While the ODM selectively targeted this cohort of devices for the OTA, we had to accommodate for the limited Internet connectivity by selectively turning on the devices so they would receive the update and not clog the Internet connection. This extended to device provisioning, as that part of the process runs through Esper and also uses limited Internet bandwidth. 

So, we updated the devices in chunks.

The end result was hundreds of devices at this deployment site updated to Esper within a day, and the customer was ready to go.

Note if these devices were running Esper Foundation for Android or using our OTA service, you could stage the rollout via Esper using either the Console or via API.

Scaled Update

From that first experience, we collectively tweaked the process and created a run book. With that run book, the customer was able to work across their deployments and get this done all across the USA. They no longer managed their fleet by ODM MDM but by Esper. This gave them a single pane of glass for their budding mixed fleet and the headroom to grow and diversify their business (new tablet device types, new use cases via kiosks, etc.).

Conclusion

Just because you have a deployed fleet running an embedded MDM — whether ODM-supplied or homegrown — doesn’t mean you can’t switch to Esper efficiently.

Yes, there is some engineering work, so clearly, there needs to be a business need to do this. There is operational overhead, but we can minimize the overhead via the OTA update technique combined with Seamless Provisioning.

This technique will only work with AOSP. Given the stringent requirements of GMS, the built in support for Android Enterprise, and the fact that GMS OEMs will not provide their Android source code, we can’t effectively do this on GMS hardware. You’ll have to utilize the existing MDM to factory reset the device and then provision it using an Android Enterprise enrollment method or a proprietary one supplied by an OEM (think Knox Mobile Enrollment or StageNow). This will be a topic for a future Esper blog post, as we’ve done this before as well.

If you are lamenting your situation with an ODM or homegrown MDM and have a sizable fleet, contact us. Even if you don’t have all your contractual ducks in a row with your ODM, we can help.

Esper MDM Solutions