How to Find Out What’s Actually New in Google Play System Updates

Thanks to Project Mainline, Google is able to deliver updates to core Android system components through the Play Store. These updates — which are publicly referred to as Google Play System Updates — can fix security issues or introduce new features, and they’re routinely delivered to users on GMS Android devices running Android 10 and later. 

There’s one problem with these updates, though: It’s hard to tell what’s actually changed. That’s because Google doesn’t post a detailed changelog anywhere, but thankfully, they do publish the source code for Project Mainline modules on AOSP. With that knowledge, I started to wonder if it’s possible to create an unofficial changelog of sorts detailing what’s actually new in a Google Play System Update? 

The answer is yes, but the process of generating a changelog is a bit cumbersome. In this week’s edition of Android Dessert Bites, I’ll share an early look at my Google Play System Update changelog and some of the unexpected issues I discovered with Project Mainline in the process.

(If you haven’t already given our Project Mainline explainer a read, I’d recommend you go do so now. That post explains everything you need to know about Mainline. This post assumes you’re familiar with the basic idea behind Mainline.)

The problem with the existing changelog

Before I share my own changelog, I wanted to talk about why I’m making one when this page already exists. That’s because, as I said, Google doesn’t post a detailed changelog anywhere. That page lists changes introduced through updates to the Google Play Store app, Google Play Services app, and Project Mainline modules, which combined are called “Google System Updates.” 

However, we’re only concerned with the subset of changes impacting Project Mainline modules, ie. what’s introduced through Google Play System Updates. Google does denote changes on that page by the component update that introduces them, but you’ll notice that the list of changes for each Google Play System Update basically amount to “bug fixes and performance improvements”. Well, they don’t literally put that down, but the information they do provide is basically as unhelpful. 

Out of all the changes introduced through Google System Updates in October 2022, only a handful of changes (highlighted in yellow) were denoted to be part of that month’s Google Play System Update.

To be fair, there’s usually not a lot of interesting user-facing changes being introduced with each Google Play System Update, so the changelog might as well be boiled down to “bug fixes and performance improvements” as far as most users are concerned. However, in recent months Google has ramped up the number of new features they’ve introduced through Play System Updates, so as someone who meticulously monitors AOSP for new features, I wanted to find a better way to keep track of these changes.

Building a proper Google Play System Update changelog

After each Google Play System Update has rolled out widely, Google pushes the source code for each updated Project Mainline module to AOSP. They’ve been doing this since Project Mainline was introduced back in Android 10, but recently, they’ve also started to tag each Mainline module release in AOSP in a predictable manner. Google unfortunately doesn’t tag every single release as far as I can tell, but they tag enough for me to generate a fairly comprehensive changelog of Project Mainline updates.

If you look at the tags under the AOSP repo for each Mainline module, you’ll see tags like “aml_tz4_331012050”, “t_frc_tz4_330443010”, “aml_go_tz4_330912000”, etc. These tags list the commits for a specific Mainline module release. For example, the “aml_tz4_331012050” tag lists all commits for the Time Zone Data module release with version number “331012050”. That’s the same version number as the Time Zone Data module that’s currently installed on my Pixel 6 Pro running the Android 13 QPR1 beta, meaning every change mentioned in the commits under that tag is included in the Time Zone Data module that’s on my device.

That version of the Time Zone Data module arrived on my device through a Google Play System Update I installed on November 10th. (How I know that is something I’ll share in a bit.) Coincidentally, if you look at Google’s official Play System Update changelog for the month of November 2022, you’ll notice that there’s one item that’s somewhat informative: “Support for new daylight saving time change in Fiji and Mexico.” 

Looking at the commit history for the Time Zone Data module, we can see that there was a recently merged patch that updated Android’s Time Zone database from version 2022e to 2022f. The Internet Assigned Numbers Authority, or IANA, maintains a Time Zone database utilized by many software vendors, including Google for Android. They publish release notes alongside each database update, and the release notes for release 2022f detail the aforementioned daylight saving time changes in Fiji and Mexico. (The relevant parts of the IANA’s release notes are summarized in the description of the patch that updates Android’s Time Zone database, but I wanted to point out the detailed release notes for the sake of completeness.)

‍

Now that I know a way to find out what changes made it into a Mainline module release, I need to document the version code of each Mainline module on my device for a particular Google Play System Update patch level, find the AOSP tag that corresponds to each Mainline module version, and then generate a changelog.

The first step requires the use of an actual device to pull information from, as you need access to a device that actually has the updated Project Mainline modules installed on it. This is the only way to get Google’s signed, prebuilt APK and APEX modules if you aren’t an Android Partner (and even then, I don’t think Google updates the prebuilts they share with OEMs that frequently). 

If you have access to a device running Android 13, you can use Android’s binary transparency CLI to print information about the installed Mainline modules on the device. This command reveals most of the information you need, including the package name, version code, computed SHA-256 hash, visibility, and install location of all active modules. 

Another useful source of information is the apex-info-list.xml file that’s automatically generated by Android. This file, as its name suggests, contains information about APEX modules installed on the device. It lists the package name, version code, and install location of each module like the previous command, but it also lists additional information such as the last time each module was updated.

However, because this file only contains information about APEX modules, it can’t be relied on solely for documenting information on Project Mainline modules. The aforementioned binary transparency CLI can be used to detail the APK modules on Android 13, though there are also other useful shell commands that can be used to fill in the rest of the gaps (and which work on older Android versions). 

Once you have all the version codes, you can easily find the AOSP tag corresponding to each Mainline module release. To make things easier on myself, I put together a spreadsheet that catalogs relevant information on each Project Mainline module installed from my test device. I made two separate sheets, one documenting information on the Mainline modules installed on my test device when it had the October 2022 GPSU and another documenting the same information after my device received the November 2022 GPSU.

The final step to building a Google Play System Update changelog is to use the “git log” command to generate a list of commits between tags. You’ll need to have a checkout of the AOSP code for this to work, which requires a lot of space on your PC if you’re doing this locally. My local AOSP checkout takes up ~167GB of space, for example.

There’s a nifty tool called the AOSP Changelog Generator which not only helps you generate a changelog between two tags (using the aforementioned “git log” command) but also helps you set up email alerts for new git tags in AOSP and publish the changelog online using GitHub Pages. The tool was originally made by developer Al Sutton (who you may recognize from our Android Bytes podcast) before it was picked up by a company called Novoda and finally Michael Bestas of the LineageOS and CalyxOS projects, who uses it to create changelogs between the tags listed in this table.

Unfortunately, since Google hasn’t completed the rollout of the November 2022 GPSU (in fact I think the rollout was halted), they haven’t published the source code for all the updated modules I have on my test device. As a result, I can’t generate a changelog between the October 2022 and November 2022 GPSU just yet. When the source code is fully published for all updated modules, however, I will generate a changelog and publish it somewhere for everyone to see.

The unexpected problems of Project Mainline

In the meantime, I wanted to take a minute to talk about some of the problems that have popped up as a result of how Google handles Mainline releases. You’ll notice above that I have to track many different tags to get a complete picture of what’s new in a particular GPSU version. This is a headache for AOSP developers to deal with as it adds a lot of complexity to their build process. Whenever Google posts a new Android Security Bulletin, the corresponding commits in AOSP that patch the disclosed vulnerabilities and which bring the platform’s security patch level up are tagged in a pretty consistent way. 

For example, when Google posted the October 2022 ASB, they published the android-13.0.0_r7 and android-13.0.0_r8 tags for AOSP 13 and the android-security-10.0.0_r71, android-security-11.0.0_r60, and android-security-12.0.0_r40 tags for AOSP 10-12L. AOSP developers can fetch the commits under these tags by doing a git merge or, if necessary, pick the patches manually. A great example of this workflow is the LineageOS project, as you can see from how they handled the October 2022 ASB for their Android 10, 11, 12/12L, and 13 branches.

This workflow is how projects like LineageOS and our own Esper Foundation for Android keep up with monthly security updates. The problem, however, is that the Mainline modules that AOSP developers compile when building AOSP using these tags are wildly out of date compared to what Google is distributing. 

To give you an example, in the previous edition of Android Dessert Bites, I talked about a change that will let the Photo Picker take over the GET_CONTENT intent, essentially letting this feature work with existing apps. In that post, I said that enabling this requires that your device have MediaProvider version 330811020 or later because that’s when the requisite feature flag was introduced. If an AOSP developer were to compile the MediaProvider module are syncing android-13.0.0_r15 (the AOSP tag corresponding to the Android 13 November 2022 release for the Pixel 7 series), then that feature flag would be missing. Similarly if they were to compile the module using the AOSP master branch. Only the android-13-mainline-mediaprovider-release branch includes the commit introducing the feature flag, but even that would put the module slightly behind where the latest aml_mpr_331011070 tag is.

Despite succeeding at reducing fragmentation among Android OEM builds, Project Mainline has ironically increased fragmentation among custom ROMs and other non-GMS AOSP-based builds. (Given the miniscule number of custom ROM users out there, though, it’s easy to say that Mainline has had a significant net positive impact on fragmentation.) Most custom ROMs ship with outdated Mainline modules that are rarely updated and can only be updated through traditional OTA updates because they were compiled and signed by the developers themselves. In order to keep Mainline modules up to date, then, custom ROM developers would need to build modules separately and then include them in their builds or find some way to distribute them to users. Technically, the best way to keep modules up to date is to just ship Google’s signed, prebuilt modules so that Google Play System Updates can handle the heavy lifting, but many projects decline to do this out of principle. 

Google doesn’t have much incentive to address this issue since custom ROM users make up a tiny, tiny fraction of overall Android users. However, I think even OEMs would benefit from Google being more open with Project Mainline module updates. I’ve heard that OEMs get a bit more information about what’s going out in a Google Play System Update, but I’m pretty sure even they don’t have the kind of detailed changelog I’m envisioning. Simplifying how Mainline module releases are tagged and making sure changes are synced with platform branches would go a long way in making sure every Android device, GMS or not, has access to the same security fixes and functional updates introduced to Project Mainline modules.

* I’m not entirely sure how Google decides what to put for the Play System Update version. Initially, I assumed that they were always following the same logic behind the security patch level string, ie. a GPSU version indicates that the device is protected against all vulnerabilities in Project Mainline listed in the corresponding Android Security Bulletin. Here, a GPSU version of September 1, 2022 should mean that the device has received patches addressing these three vulnerabilities, which makes sense. Sometimes, though, no vulnerabilities affecting Project Mainline modules are addressed by an Android Security Bulletin, such as in the case of the October 2022 ASB, and yet the GPSU version may still be bumped. The GPSU version thus generally seems to denote which month’s ASB has been addressed.

Internally, I know that Google releases Mainline module updates as part of a “train”, ie. a versioned set of modules, and that these “trains” are named something like “T1009999” (this is not a real one, but it’s similar.) Google Play System Updates are not released on a fixed monthly cycle like the Android Security Bulletin, and a GPSU is not always guaranteed to bump the GPSU version. The update to the Time Zone Data module that I mentioned receiving on November 10th, for example, didn’t bump my device’s GPSU version to November 2022 from October 2022. The GPSU version is only incremented when Google bumps the version number of the ModuleMetadata module, and as I just mentioned, that’s usually done when patches addressing vulnerabilities in the corresponding month’s ASB have been implemented. As you can see in my spreadsheet, not every module is updated at the same time.

Android Device Management