Android is Adding Support for Updatable Root Certificates Amidst TrustCor Scare

The world’s biggest tech companies have lost confidence in one of the Internet’s behind-the-scenes gatekeepers. Microsoft, Mozilla, and Google are dropping TrustCor Systems as a root certificate authority in their products. Starting in Chrome version 111 for desktops, the browser will no longer trust certificates issued by TrustCor Systems. The same change is coming to Android, but unlike Chrome for desktops, Android’s root certificate store can’t be updated independently of the OS, meaning it’ll take some time for the certificate changes to roll out. Thankfully, that may no longer be the case in Android 14, as Google is preparing to implement updatable root certificates in the next release.

The root of Android's root certificate problem

The Internet is built on trust. Billions of people go online every day without realizing how many layers of software and services they have to trust to do the right thing. When you open a web browser and navigate to a website, for example esper.io, you have to trust that your network and device’s software are configured properly to point you to the right server hosting our website.

Once you connect to the right server, though, how do you know that connection is secure (HTTPS)? Your browser has to establish a private connection so that data can be encrypted in transit (TLS), but it can only establish that connection if it trusts the site’s (TLS) security certificate.

There are over a billion websites established on the Internet, though, so maintaining a comprehensive list of trusted security certificates is nigh impossible. Instead, web browsers and operating systems rely on a chain of trust to validate a site’s security certificate.

The security certificate for websites that end users connect to is validated by a service called a certificate authority (CA). That CA might only be an intermediate in the chain of trust, though, as the CA’s own certificate may have been validated by an entity even higher up the chain. At the highest level are root CAs, CAs whose security certificates establish the root of trust of the Internet.

The security certificates of these root CAs are usually trusted by major web browsers and operating systems because the entities that created them ostensibly guard their secrets well and at some point, the chain of validations has to end somewhere. For esper.io, the chain is as follows: our website’s security certificate was signed by Let’s Encrypt, a free, non-profit certificate authority, whose own certificate was signed by the Internet Security Research Group (ISRG).

Each operating system has its own built-in root store — a list of trusted root CA certificates — and Android is no different.Android’s system root store is what apps default to when trying to verify certificates, ie. when they try to establish a secure connection. With Android 9, Google made it so apps had to explicitly opt-in to enabling cleartext traffic (HTTP) for specific domains. This means that most apps today use TLS for all Internet connections, and they use the system root store to handle certificate verification.

Some apps on Android configure a custom set of trusted CAs on top of the OS’s built-in root store, limit which certificates to trust (certificate pinning), or trust additional certificates not included in the system root store. Firefox for Android notably uses its own root store, but Chrome for Android does not. Chrome recently introduced its own root store, but it’s only in use on Windows and macOS.

While it’s possible for users to manually install CA certificates through Android’s settings, these certificates are saved to the user certificate store and not Android’s system-wide one. Apps can opt into trusting certificates in the user certificate store, but they are not trusted by default. That means many apps by default rely on the operating system to verify certificates.

If there were to be a problem with a root CA — say hypothetically their certificates expired or they were discovered to be untrustworthy — then problems might arise as apps and users could no longer trust the intermediate CAs and sites whose certificates were signed by them. That’s a big deal because it means many web browsers and apps won’t be able to establish secure connections with certain websites anymore, breaking many services and rightfully scaring users away.

The solution seems simple: Just update Android’s built-in root store when a problem is found. However, the only way for Android’s built-in root store to be updated is through a firmware over-the-air (OTA) update. And if you know anything about Android, it’s that getting a necessary firmware update out to every device that needs it is a challenge.

A broken chain of trust

Both of the hypothetical problems I mentioned earlier have actually happened, and on numerous occasions too. We’ll look at two case studies that exemplify the problems with non-updatable root certificates on Android.

Case 1: How Let's Encrypt narrowly avoided disaster

When a certificate authority creates a new root certificate (ie. one that isn’t signed by another CA), they’re usually made to be valid for quite a while, often several years or even more than a decade. The reason is that when they expire, they can cause a cascade of certificate invalidations down the chain. Take the ISRG’s Root X1 certificate as an example. It’s valid until June 4, 2035, so by the time it expires, we’ll have run out of letters in the alphabet for Android to base its dessert name on!

New root certificates aren’t immediately accepted by most major operating systems and web browsers, as it’s up to each individual project to determine whether to accept the new certificate in its root store. For example, the ISRG Root X1 certificate was generated on June 4, 2015, but it wasn’t added to Android’s built-in root store until Android 7.1 released in October 2016.

That might not sound like that long of a gap, but the problem is that Let’s Encrypt would have had to wait nearly a year and a half for Android devices to come on the market that would trust any certificates signed by ISRG Root X1. Considering that Let’s Encrypt was created for the sole purpose of getting as many websites to support HTTPS as possible, it wouldn’t be viable for them to start off by issuing certificates signed by the ISRG Root X1. Instead, Let’s Encrypt did what many new CAs commonly do: get another CA to cross-sign their certificate.

In October 2015, Let’s Encrypt received cross-signatures from IdenTrust, a CA whose root certificate (DST Root CA X3) was already widely accepted by most major platforms, including Android since version 2.3.6. That let Let’s Encrypt immediately start issuing certificates that would be trusted by most Android devices. The problem, however, is that the DST Root CA X3 certificate was going to expire on September 30, 2021, meaning after that date, devices running Android 7.0 and lower would no longer trust certificates issued by Let’s Encrypt. That would’ve broken a lot of sites and services considering how popular Let’s Encrypt is (we use them as our own CA!) and how many users have outdated Android phones.

Let’s Encrypt raised alarm bells about this issue about a year before DST Root CA X3 expired. Fortunately, Let’s Encrypt quickly found a solution: They came to an agreement with IdenTrust to cross-sign the ISRG Root X1 certificate with DST Root CA X3. This means that any certificate issued by Let’s Encrypt that’s signed by ISRG Root X1 would also be trusted on devices that trust DST Root CA X3, including devices running Android versions 2.3.6 to 7.0. This only works because, according to Let’s Encrypt, Android, as opposed to other platforms, “intentionally does not enforce the expiration dates of certificates used as trust anchors.” As a result, Android devices with DST Root CA X3 in their root store would continue to trust certificates signed by it even past its expiration.

“But isn’t DST Root CA X3 expiring? The self-signed certificate which represents the DST Root CA X3 keypair is expiring. But browser and OS root stores don’t contain certificates per se, they contain “trust anchors”, and the standards for verifying certificates allow implementations to choose whether or not to use fields on trust anchors. Android has intentionally chosen not to use the notAfter field of trust anchors. Just as our ISRG Root X1 hasn’t been added to older Android trust stores, DST Root CA X3 hasn’t been removed. So it can issue a cross-sign whose validity extends beyond the expiration of its own self-signed certificate without any issues.”

Josh Aas and Aaron Gable, Let's Encrypt

This change allowed Let’s Encrypt to seamlessly support users on older Android devices after DST Root CA X3 expired last year, but this support will eventually end on September 2024 when their cross-sign agreement with IdenTrust ends. After that date, older Android devices will see certificate errors, though by that point, the impact on users will be miniscule given 90.4% of all Android devices are on Android 7.1 or later (as of August 2022) and that percentage will only get bigger as we approach September 2024.

While it’s great that Let’s Encrypt found a solution, they averted disaster only because of a quirk in the way Android handles expired root certificates. This would’ve never been a problem if Android had a mechanism to update the root store outside of OTA updates. Although the DST Root CA X3 expiration affects more platforms than just Android, Let’s Encrypt called out Android specifically because of how much more impactful the expiration would have been had they not found a solution.

Case 2: How TrustCor lost trust

TrustCor Systems is a certificate authority registered in Panama that recently came under fire after security researchers alleged it had ties to an organization that distributed a spyware SDK to U.S. intelligence agencies. The accusations, coupled with the organization’s lackluster responses to security researchers, caused trust in them to collapse, even though no specific allegation of wrongdoing was proven (such as issuing a certificate they should not have). Trust is paramount in this system, and losing that trust is a deathknell for any CA.

In response to this news, Microsoft, Mozilla, and Google announced that they are dropping TrustCor’s certificates from their respective root stores. Google in particular, however, only announced that they are removing TrustCor’s certificates from the Chrome Root Store which, as mentioned before, is only used on Chrome for Windows and macOS. They are dropping these certificates from Android’s root store, however, as shown by these AOSP code changes. However, removing these certificates from Android’s root store will require an OTA update to be rolled out, which means it could take considerable time for this change to arrive (if at all) on many devices.

The upside is that, according to security experts from the GrapheneOS project, the removal of the TrustCor CA should have “very little impact on web compatibility due to this CA barely being used by anyone other than a specific dynamic DNS provider.”

Still, the TrustCor scare shows that Android needs a better way to update its root store, especially since it’s not the first root certificate to be removed over suspicious activity. Other CAs that have been blocked in the past include CNNIC, WoSign, and StartCom. Fortunately, Google has been working on a solution to this problem, and it uses a familiar mechanism: Project Mainline.

Updatable root certificates coming soon

With the release of Android 10, Google introduced Project Mainline, an initiative that modularized system components into packages (.APK or .APEX) that could be updated independently of the rest of the OS. Android 10 launched with thirteen modular system components, and each subsequent release introduced additional modules. For example, in Android 13, Google turned Android’s Bluetooth and Ultra-wideband stacks into modules. One of the original modules that Project Mainline introduced was Conscrypt, a module that provides Android’s TLS implementation.

According to code changes recently submitted to AOSP, Android’s Conscrypt module will support updatable certificates. These code changes haven’t been merged yet, but they show how the feature will work. The Conscrypt module will contain a copy of Android’s root store, which the system will now default to reading from. Since APEX packages are updatable through Google Play, this means that the root certificates can be modified by updating the Conscrypt module. This will allow Google to bypass OEMs and carriers and push out updates to Android’s root store themselves, at least on devices with GMS.

It isn’t clear when this change will roll out, but it’s likely that it’ll first be widely available in Android 14. That’s because the change requires an update to the SystemCertificateSource API, which is not part of any modular system components and thus requires an OTA update to modify. Such a change could technically arrive in Android 13 QPR2 or QPR3 because modifications to non-public APIs are allowed after the API freeze, but I doubt Google will introduce this change before Android 14 considering that few devices outside of Pixel ever see QPR updates.

If you are looking for a tool to manage Android devices, give us at Esper a shout. To get an overview of mobile device management tools, check out our guide: 

Everything You Need to Know about MDM