The Android Security Patch Conundrum

Google has rolled out a monthly security update for its Pixel phones at the beginning of each month for the last several years. The January 2022 update for Pixel phones started rolling out on Tuesday, but Google’s latest Pixel phones — the Pixel 6 and Pixel 6 Pro — won’t be receiving the update until later this month. A couple of weeks of delay doesn’t sound like a big deal, but Google previously halted and pulled the big December 2021 update before it rolled out for most Pixel 6 users — meaning most users are running a build that’s two months out of date (and potentially getting their enterprise access revoked). Considering that the January 2022 update fixes the very serious emergency calling bug, it’s a bit worrying that the update has been delayed! 

I think it’s fair to say that Google strives to roll out updates as quickly as possible for its own Pixel phones, and indeed no one can argue that Google is slow at rolling out updates. Perhaps the combination of the winter holidays, the laundry list of changes to merge from the first Android 12 Quarterly Platform Release, the switch to developing for its in-house Tensor platform, and carrier testing and retesting resulted in the one-week delay and eventual cancellation of the December 2021 update and the multi-week delay of the January 2021 update. It’s normal for some bugs to go unnoticed before an update goes out, because it’s impossible to catch every issue in the brief time they have to test. Nevertheless, this delay exposed a flaw in how Google handles Android’s monthly security patches, so in this week’s edition of Android Dessert Bites, I’ll be diving into the monthly security patch process and identify an area where Google could do better.

The monthly Android Security Bulletin is always on time, even if updates aren’t

Compared to Chrome and Chrome OS, Google has far less control over the distribution of Android. Google has to work more closely with OEMs, who fork AOSP to add their own software features and branding, and SoC vendors, who write the closed source drivers that enable hardware components. Rather than submitting patches for security vulnerabilities to the public AOSP repository and waiting for OEMs to merge them, Google created the monthly Android Security Bulletin (ASB) to coordinate the disclosure of security vulnerabilities and give companies time to merge and test security patches before public disclosure.

An Android Security Bulletin has been published every month since August of 2015. Generally, the Bulletin goes public on the first Monday of every month, unless that Monday falls on a holiday observed by Google, in which case the ASB goes public on the next business day. By coordinating the disclosure of security vulnerabilities with OEMs and SoC vendors and privately sharing the patches one month in advance of public disclosure, Google is able to speed up the uptake of security patches and minimize harm to users. The more time that companies have to integrate and test security patches, the faster they’re able to roll out an update with those patches. The less time that hackers have to analyze and exploit the newly discovered security vulnerabilities, the fewer number of users that can be attacked.  

While you can predict when the next ASB will go live, you generally can’t predict when the software update with those patches will roll out. Google almost always rolls out an update on the same day the ASB goes live, though that obviously hasn’t been the case for the Pixel 6 series since its release. When it comes to most OEMs, the update may roll out weeks or months after public disclosure, but some of the faster OEMs push out updates within days. Google doesn’t like it when OEMs roll out updates too quickly, curiously enough. Essential’s former head of R&D, Jason Keats, told Android Police that the company was berated for pushing updates out too fast. “Literally, we would get yelled at by Google for being too fast… they would then say, ‘We will not allow you to push your update until the Pixel team is ready,’” said Keats in the interview. Dianne Hackborn, one of the most senior engineers on the Android team, offered an explanation for why Google disapproves of updates being pushed out by other companies on day one. According to her, security updates are pushed to Pixels only when they’re sufficiently tested to show no problems, so if other companies are pushing out updates before Google, then they may be doing so before the patches have been fully tested for regressions.

There are some OEMs like Samsung that consistently roll out security updates before public disclosure, which is odd to see given that Google shares the Bulletin and patches with its partners under an embargo that lifts when the Bulletin goes public. By pushing an update before the ASB goes public, hackers can theoretically discover what vulnerabilities have been patched by comparing the previous and current versions of the firmware, but it’s unlikely that this happens given how few days they have to do this before the ASB goes public anyway. Once the ASB goes public on Google’s website, any user can see the list of security vulnerabilities that have been addressed, though some of the technical details will remain accessible only to Google’s Android partners. 

What’s on the Android Security Bulletin?

Google is the single largest contributor to AOSP, so of course they discover many vulnerabilities themselves. Android is a huge platform, though, so many vulnerabilities are found by independent researchers or third-party companies. Vulnerabilities in AOSP aren’t the only vulnerabilities included in the ASB, though. Google also lists patches for the Linux kernel (which Android’s kernel is forked from) and closed source vendor components. Patches for AOSP and Linux kernel vulnerabilities can be picked from their respective upstream sources, but technical details on and patches for closed source vendor components are available only from the company that makes them.

Once vulnerabilities have been disclosed, Google assigns them a Common Vulnerabilities and Exposures (CVE) ID, categorizes them by type, gives them a severity assessment, and lists them alongside their affected Android version(s). In addition, the ASB preview for Android partners lists the specific subcomponent that’s affected, a brief technical description of the vulnerability, and a short description of the fix that’s being applied. Google also shares a link to download the patches in a ZIP file as well as the latest binary of the Security Test Suite (STS), a tool that automatically validates whether certain patches have been integrated properly. The STS doesn’t guarantee that every patch has been integrated properly, but if a test has been written for a patch, then there’s a good chance that it’ll be picked up.

Google discloses these vulnerabilities to Android partners about a month before the public Android Security Bulletin. The January 2022 ASB, for example, went public on January 4, 2022, but a preview of the ASB was shared with Android partners on December 6, 2021. Most of the vulnerabilities detailed in any given ASB are disclosed to Google in the two months before the Bulletin goes public, but in some cases, the time between initial disclosure and the patch being made available could be even longer. For instance, the (arguably harmless) escalation of privilege bug that I helped discover in Android 12’s Fabricated Overlay API was disclosed on October 11, 2021, and a patch was authored shortly after on October 18, 2021. However, the patch wasn’t committed to Google’s internal Android 12 branch until November 24, 2021, so CVE-2021-39630 was only addressed in the January 2022 ASB even though it’s been nearly 3 months since it was disclosed. On the other hand, the very serious emergency calling bug that I alluded to earlier was publicly disclosed by a Samsung engineer with a patch to AOSP on November 25, 2021, which gave enough time for CVE-2021-39659 to be included in the January 2022 ASB.

Regardless of when the vulnerability was actually disclosed, OEMs get about a one month head start to integrate and test their builds before public disclosure. Before the public disclosure happens, though, Google may update the ASB preview to remove patches that are incomplete or which require more time to test. This happened twice with the January 2022 ASB, once on December 13 and again on December 28, a week before public disclosure. Details on these patches are kept under embargo until a future ASB, though Android partners are not required to remove any patches they have already integrated.

(The term “Android partners”, by the way, has a specific meaning here. I’m referring to the hundreds of companies that have shipped or plan to ship an “Android-compatible” device, which is a device that meets Android’s compatibility requirements and ships with Google Mobile Services.)

It’s time to update the security patch level

Once it’s time for the public ASB to go live, Android partners can then roll out updates containing these security patches. The time it takes to roll out the update varies, as previously mentioned, because of multiple factors. For one, an OEM may need to make additional changes to integrate the security patch if the patch conflicts with existing code. Second, the OEM or SoC vendor may discover vulnerabilities that only affect their specific fork of Android (many OEMs and SoC vendors publish their own security bulletins for this reason). Third, the OEM may need to integrate additional patches for vulnerabilities disclosed by SoC vendors outside of Google’s ASB. Patches for the MediaTek-su vulnerability, for example, were ready in May of 2019, 10 months before the patch was included in the March 2020 ASB. Fourth, carriers may want to recertify the build to ensure compatibility with their networks. Lastly, OEMs may choose to wait to roll out a security update so they can also release some feature upgrades at the same time (this is a problem I’ll get to in more detail later). Initiatives like Project Treble, extended Linux kernel LTS, Project Mainline, and the Generic Kernel Image, were designed to reduce the technical complexity of merging patches, resulting in speedier security updates.

When the update is ready to be pushed, OEMs update a string in their builds to signal that the software contains the security patches listed in a particular ASB. This string, the system property ro.build.version.security_patch, must be set to the appropriate security patch level (SPL). 

Every month, there are two SPLs: a partial and complete SPL. The partial SPL ends with -01 in the string, while the complete SPL ends with -05 in the string. For example, the partial SPL for this month is 2022-01-01, while the complete SPL is 2022-01-05. The format of the partial SPL can be generalized as YYYY-MM-01 while the complete SPL can be generalized as YYYY-MM-05.

Builds declaring the partial SPL must contain patches to all of the vulnerabilities affecting AOSP, which includes the Android OS framework, the media framework, system components, and Project Mainline modules. Builds declaring the complete SPL must contain patches to all vulnerabilities affecting AOSP as well as the Linux kernel and closed-source vendor components. 

Since the SPL is denoted by a string defined by the OEM, there’s no guarantee that the build actually contains all of the patches that are required for that SPL. Researchers in mid-2018 discovered a significant “patch gap” between the SPL many phones reported and what vulnerabilities those phones were protected against. Automated tests like the STS help OEMs ensure that most patches are included in their builds, fortunately. 

What went wrong with the security patch process for the Pixel 6?

As I mentioned previously, the January 2022 update for the Pixel 6 series won’t roll out until later this month, leaving most Pixel 6 users on the older November 2021 SPL. The reason behind the delay is related to connectivity issues discovered shortly after the December 2021 update rolled out to some users. 

Why were there connectivity issues in a build that was already delayed by a week? Well, it was the winter holiday, so Google was probably short-staffed. Furthermore, the Pixel’s December 2021 update didn’t just include the patches disclosed in the ASB — it also included the changes from the big Android 12 QPR1 release which were marketed as part of the quarterly Pixel Feature Drop. Android 12 was Google’s biggest OS update in years, and given the approximately one-year turnaround for its development, it’s no surprise that the initial release had a lot of bugs and unresolved issues.

That’s another reason the December 2021 update was particularly noteworthy: it fixed nearly 100 bugs in Android 12. Fixing so many bugs and integrating the Android 12 QPR1 codebase necessitated extensive testing, but Google doesn’t do public beta tests for quarterly releases, so they had to test everything internally. The approximately 1.5 month long period for testing and certification was clearly insufficient this time around, though. Tying the release of a security update to a big feature update is risky, and this time, it resulted in the security update being significantly delayed.

Merging the December 2021 patches, Pixel Feature Drop, and Quarterly Platform Release into one update is great for marketing and convenient for users, but these updates don’t need to be combined. Instead of having Pixel 6 users sit on a build that’s vulnerable to a high severity denial of service exploit that could block emergency calling, Google should release a security update sans QPR1 changes. Alternatively, they could push an update that contains a few patches fixing only the most critical security issues, which is how they fixed the emergency calling bug on the Pixel 3 series despite leaving it on the October 2021 SPL. Google knows how severe this issue is, which is why they even backported security patches to Android 8.0-8.1 for OEMs to pick even though Google only commits to backporting security patches to Android versions that are less than 3.5 years old. Google wants to release new security updates once a month and new quarterly platform releases once every three months, but there’s no reason they can’t step out of that cycle occasionally. Committing to a once-a-month release cycle has its benefits, of course, as it makes updates more predictable and less annoying for users. However, I believe an exception needs to be made when it comes to updates that fix major security issues — patches are immediately made available all the time for critical security issues that affect other software we use on a day-to-day basis.

Thanks for reading this week’s edition of Android Dessert Bites. If you’re interested in reading another explanation about the monthly Android security patch process, check out this article by my former colleague Adam Conway for XDA-Developers.

Esper Device Management