Esper Deep Cuts: A Rescue Operation

One tenant of Esper’s infrastructure is enabling revenue-bearing unattended experiences. Those from commercial-oriented vertical market businesses tend to think of this in traditional use cases like POS, digital signage, AIDC, etc.

But this world also exists for consumer markets. Device-based offerings that generate tens of dollars a month in recurring consumer revenue by delivering content and experiences. Companies often use Android for these experiences, typically requiring a specific subset of skills to maintain.

Yet, as a recurring revenue-generating product in the field, it’s tough to keep these devices operating. In general, companies in this domain are fantastic at dreaming up and feeding these experiences but relatively naive about how to build and maintain Android-based solutions.

Oftentimes, they are drawn in by suppliers that specialize in that particular vertical. While that may be fine for commodity offerings in that vertical, for sophisticated OEMs, they often end up being inadequate for those OEMs to realize their business potential. But once they have you, it's too late to change course — unless you work with a technically adept partner such as Esper. Let me tell you the tale of a connected consumer device. Feel free to imagine your own consumer or commercial service as you follow along with this story.

Rescue Operation for Exercise and Revenue

Picture this: A novel piece of equipment. An exhilarating experience for consumers. An ongoing stream of value-added improvements to the software-defined world delivered via a screen. This smacks of Android, and sure enough, this customer made the Android decision. With money every month.

This customer was an expert in their particular domain, spanning the industrial and mechanical design of the equipment. Android hardware, operating systems, managing a fleet, and mating it with the Cloud-based delivery system, not so much. That said, they had a vision: A very detailed experience they wanted to create via the Android-based device — heck yeah!

If you’ve ever entered this space, you know. There are companies out there that claim to be turnkey and give you everything you need. For many commodity OEMs, it appears that’s enough. And this particular customer went that route to start. They sourced an ODM and an OS and device management infrastructure provider on their own. The ball is rolling: the march to mass production and monetization!

The bumps in the road started to rear up.

Struggles Begin

Keep in mind that this is a premium product. The customer needed to define and control the entire software experience. Yet this customer was unable to get what they needed. They were dealing with a standard AOSP build and a set of suppliers focused on cookie-cutter delivery, all with little skill in crafting an experience via Android.

Furthermore, upon doing deeper due diligence, we discovered that one of the parties shipped monitoring software disguised as part of their Android OTA service to keep the OS up-to-date. This called into question the basic edge software infrastructure they were building their entire business on.

The customer was fantastic at marketing and built an entire launch plan on the agreed-upon production dates with their suppliers. So, the marketing execution started and was ramping to plan. Unfortunately, the product development process was not, putting the entire company at risk.

The experience delivered via the OS and bundled kiosk mode application needed to be burned in at the factory, with any changes reliant on a less-than-stellar MDM infrastructure for app updates. That meant whatever the supplier burned into ROM at the factory was the initial out-of-box experience for end users. For our customer, the entire situation was unsatisfactory.

Enter Esper

Through executive connections, we were able to dive deep with the company's founders. It was a tough situation as the entire OS was built with the third-party MDM and integrated OTA functionality. They were ready to move on but made the mistake of not covering their bases to give them an escape route by licensing the OS at source — at that point, they were beholden to the two suppliers.

We jumped in to help on the business front, coaching the customer on dealing with the suppliers to get to what they needed — access to the Android OS source and finding an elegant way to remove the business relationship with the MDM/OTA provider. It wasn’t easy, but together, we accomplished that through hard negotiation that admittedly cost the customer some money, but they did what they needed to do. 

Our due diligence on the hardware design and corresponding interface led us to give the okay on the existing device design. The existing ODM satisfied the customer with the hardware pricing and manufacturing capabilities, so they stayed on board. 

The overall software stack was not ready for prime time. In order for this to work, the customer had to stick to the factory's original mass production date. The customer had a launch and selling window, and the ODM was set to ship the hardware to a USA port and then forward it to a warehouse. There was no stretch time.

Coughing up a Tarball

For Esper, the next key event was to get the OS source in hand. We had our Android OS engineering team do due diligence on the source, and they found it to be tough to navigate. One challenge we typically see is the OS source delivered as a tarball, meaning a set of files packaged together and then compressed into a single file.

There is no git history, so it is not easy to see the changes made by others to the OS build, including what third-party code they added to the image. Accomplishing what the customer needed for the user experience required more than just building the right Android app—it also required modifications to the OS.

Esper Foundation for Android was the answer for them, but given the timeline and the available budget, doing a full Android bring-up on an Arm-based CPU hardware design was out of the question. Unfortunately, they already spent a lot of NRE (non-recurring engineering) to get to where they were.

The only viable approach to meeting the timeline was via GSI, or a Generic System Image. We won’t go into detail here about what a GSI is, but short and sweet, it is the Android Framework layer that you can bolt on top of the Android Vendor layer—if the Vendor layer is Project Treble compliant, anyway. And Esper happens to provide a Foundation GSI.

Unfortunately, upon our inspection, the incoming OS was not ready to run a GSI. Our engineering team determined the scope of changes required to make the vendor layer sufficiently compliant to run the Esper Foundation for Android GSI. Since this is an AOSP build, the strict requirements for GMS builds were not an issue. It needed to work for the customer’s use case, as it should be for a dedicated device.

From that, we determined that yes, we could get it there and run the Foundation GSI, add Esper’s OTA framework, remove the added third-party code, and add the features needed for the use case (like tight availability of certain Android Settings while their app was in lockdown kiosk mode, all Esper messages changed to reflect the customer’s brand, and so forth). All within a timeline to make the factory manufacturing date.

The next problem: Application development was behind schedule. That’s where Esper’s Seamless Provisioning came into play. Using Seamless, we made it into a cloud-driven, software-defined device rather than static from the factory. The approach was for the customer to use Esper’s Blueprint infrastructure to define the configuration, app load, and app setup on the device, so upon first boot the device would configure and be ready to go.

This bought them time. They needed the app and overall experience ready when their first customer received the equipment and started setting up, which gave them much-needed months of extra time. Plus, they skipped the additional time and expense of staging—they could ship the equipment directly from the warehouse to the customer without having to break the box open.

With that, we needed to address one more detail. For commercial use cases, Esper’s provisioning process provides a set of Esper-branded screens and prompts, relying on the Android Setup Wizard to run and hand off to us. We like this flow, and IT Ops and engineers also like it. But this wasn’t the experience this customer wanted.

They wanted the on-brand experience to exude to their new customer as soon as the first screen lit up. Of course, the custom boot animation they created is part of that. But it then needed to lead to a specific set of screens that showcased someone using the machine. The customer tightly implemented the whole experience.

With this experience, we needed to have a seamless flow from first boot through provisioning, with nary a peep that this is Esper. It was all about the customer, the brand, and a stellar consumer out-of-box experience.

We knew what we collectively needed to do. Now, it was time to get to it.

Make Mass Production On Time

Working with an ODM that threw their longtime software partner out of the deal was interesting. But customer money talks, and all-in-all, the ODM was good to work with.

Because the march to make the date was intense, we had to go through the full development cycle for the Android OS build with the customer, meaning a modified vendor layer with the Foundation GSI with all the required bells and whistles in a compressed timeline. 

We also modified the Esper Agent and the provisioning process to incorporate the customer’s intended experience. This meant interfacing with creative types, and at times, the target moved a bit because there is a dash of artistic experience in the creative endeavor. To further lower the risk of delivery, we also jumped in to help the customer’s app dev team wherever they needed us.

We made the date, and the devices went to MP (mass production) shipping with Esper Foundation for Android — cloud-driven configuration and bristling with our robust OTA infrastructure built in.

Quite an Exercise!

Yes, it was indeed, and we hadn’t even been able to use the device yet! From a customer perspective, we got them out of a tight situation with external suppliers and ultimately delivered the solution and experience they needed within a tight timeline.

The strategy of Software Defined Device worked out well — the motion of drop shipping to their buyers and having them enter Wi-Fi credentials or connecting ethernet, then Esper taking care of the rest, was an amazing capability for them. Plus, it lets them adjust the configuration during the development phase (including validation!) right to the end. Then, it sets them up to update their fleet without using OTA as they added to the experience. This also enabled system updates as well.

They accomplished this with a mix of NRE and our standard per-device SaaS fee. Esper's system was a natural fit for a company making dollars off of every device every month. Our joint solution made business sense to them to solidify their ability to engage the customer and collect recurring revenue — win-win!

Our Foundation OS, our Esper Agent, provisioning, and messages are all built out and available for other customers. And guess what? We have indeed done this for other customers since then, including quite a big consumer electronics TLA.

Perhaps you need such a thing? Let us know!

Contact the Esper Team