How Google Will Use Virtualization in Android 13

Ah, fragmentation: a word that has tainted the Android platform since its inception. The way Google architected and licensed Android was designed to promote early adoption by OEMs, but those early design choices led to years of accrued technical debt that require major technical undertakings to address. Slowly but surely, Google is clawing back control over Android to address fragmentation. First, it addressed the core Android framework with Project Treble and the Generic System Image (GSI). Then it targeted core Android system components with Project Mainline. More recently, it tackled kernel fragmentation with the Generic Kernel Image (GKI). The next front in Google’s multi-year war on fragmentation is virtualization.

Virtualization on Android is a topic that you won’t find much discussion on because it’s esoteric and the work is quietly ongoing, but if you’ve followed my work, you’ll know that it doesn’t matter how niche it is—if it’s Android-related, it’s right up my alley. That’s why I’m dedicating this week’s edition of Android Dessert Bites to virtualization on Android and what Google plans to do with it.

Bringing KVM to Android

Virtualization on Android today is “the Wild West of fragmentation,” according to Will Deacon on the Android Systems team. This is because hypervisors may or may not be present on a device, and when they are, they’re often not even used for their intended purpose, which is to run an operating system in a virtual machine! Instead, they’re used for things like enhancing the security of the kernel (or at least trying to) and running miscellaneous code (such as third-party code for DRM, cryptography, and other closed-source binaries) outside of the Android OS.

To understand why the latter is particularly problematic, consider that in the Armv8/v9 exception model, the hypervisor runs at exception level 2 (EL2). In Arm’s nomenclature, the higher the number, the higher the privilege level, which means that code running at EL0 (eg. userspace applications) is least privileged, code running at EL1 (eg. the Android OS and Linux kernel) is more privileged, and so on. Thus, a lot of opaque third-party binary blobs run with higher privileges than even the OS and kernel! This is harmful for security, as it increases the attack surface of privileged code that can be exploited, as code running at a higher EL can access all the registers of the lower levels.

In order to both de-privilege this third-party code and isolate that code from Android and other third-party programs, Google is working to bring a common hypervisor solution on top of which a virtual machine running at the same privilege level as the OS and the kernel will execute that code. There exists a mature kernel virtualization mechanism called KVM which is already supported by Linux, so naturally Google is choosing to deploy it as the common hypervisor. And thanks to Google’s ongoing efforts to reduce kernel fragmentation, KVM can be enabled on a broad spectrum of Android devices shipping a recent version of the GKI.

(Sidenote: Google is actually extending KVM with additional security features and calling it pKVM, or “protected KVM”. pKVM is designed to enable data confidentiality in a virtual machine, even if the OS is compromised. The implementation is available in the mainline, android13-5.10, and android13-5.15 Android Common Kernel branches.)

To manage these virtual machines, Google is porting crosvm, the Rust-based Virtual Machine Manager (VMM) used to run Linux apps on Chrome OS, to Android, and it will be delivered to devices through a new Mainline module called “Virtualization” (com.android.virt). Currently, no Android devices on the market ship with the Virtualization module — not even Google’s own Pixel 6 — but this is set to change with the upcoming Android 13 release. In fact, Google is currently testing its new virtualization tools on the Pixel 6; if you build AOSP with the target aosp_oriole_pkvm, you’ll find that com.android.virt will be automatically inherited. I don’t know if Google will enable pKVM on the Pixel 6 series with the Android 13 update, but there is evidence that Google plans for Android 13 to include the first release of the pKVM hypervisor and virtual machine framework.

(If you’re interested in the full history of Google’s efforts to bring KVM to Android devices, watch Will Deacon’s talk from last year’s KVM Forum or this excellent summary by Jake Edge for LWN.)

Isolated compilation in a virtual machine running CompOS

With the groundwork laid to run virtual machines on a hypervisor, the question then becomes: What feature(s) does Google itself plan to use this for? Thanks to a series of code changes submitted to the AOSP Gerrit, I have learned about one way that Google plans to demo Android’s new virtualization support: isolated compilation.

Earlier this year, I shared that Google is working on a new build of Android called “microdroid”. Google describes microdroid as “a (very) lightweight version of Android that is intended to run on on-device virtual machines” and host headless (ie. without a GUI) and native workloads. It is built “from the same source code as regular Android, but it is much smaller” as it lacks the system_server process, HALs, or a GUI. Microdroid is included in the Virtualization module with its payload of APEX modules and APKs defined in one JSON file and its partitions and allocated memory defined in another.

Interestingly, it seems that microdroid is just the default name of the OS in the VM — it can be configured to be called something else. This is what I believe “CompOS” is — an instance of microdroid that’s dedicated to performing isolated compilation. (CompOS is short for “Compilation OS,” in case you’re wondering.)

You may be wondering at this point what isolated compilation means. According to Google, isolated compilation is the compilation of boot and system_server classpath JARs in a protected VM. When configuring the system ROM, device makers typically ship precompiled code for core services like system_server and other classes that zygote initializes on startup. Whenever the Android Runtime (ART) is updated, which can now happen out-of-band as it was made a Mainline module in Android 12, it’s possible that compilation artifacts for boot class path extensions and system_server need to be regenerated using the odrefresh tool (which is why ART APEX updates sometimes show a progress bar during boot up). In Android 12, this all happens on the host OS because there’s no virtualization support, but in Android 13, this could happen in CompOS.

To be perfectly honest, I am not entirely sure why it’s advantageous to move this compilation to an isolated VM, but I’ve heard a few theories. First, moving this functionality to a VM makes the process overall more secure. Any exploits during the process would need to chain into an exploit of the hypervisor to escape the VM and access data in the host OS. Keep in mind that many parts of the virtual machine framework are written in Rust, a programming language designed with the security of memory management in mind, so this is not trivial to do. Second, a compromised system_server process could be used to accept modified OAT files. By running dex2oat in a VM that’s verified by the hypervisor, the compiled OAT file can be cryptographically verified. Those are just theories, though. Only Google knows the exact reason behind this feature, but I’m not privy to the internal discussions that led to the development of CompOS.

Isolated compilation doesn’t sound like a particularly interesting use of virtualization on Android, but it’s most likely not the only use case that Google is working on. It just happens to be the one use case that’s developed mostly in public. There are a number of other things that Google can do with a virtualized Android build, but we’ll have to wait for the release of Android 13 to find out what those are. In the meantime, if you’re interested in playing with virtualization on Android yourself, Google has a guide on how to get started with protected virtual machines and microdroid.

Thanks for reading this week’s edition of Android Dessert Bites! I had to step a bit out of my comfort zone to write about this topic, and due to a lack of documentation, I had to do a lot of digging to figure out what’s going on. I did consult with a few system and kernel developers to fill in some gaps in my knowledge, so if you helped out, thank you! If you have any feedback to offer on this article, please reach out to me at mishaal@esper.io.

If you are looking for a tool to manage Android devices, give us at Esper a shout. To get an overview of mobile device management tools, check out our guide: 

Everything You Need to Know about MDM