What Does “AOSP Android” Really Mean?

What is AOSP?

AOSP is the Android Open Source Project and refers to the publicly available source code for the Android operating system. The first version of AOSP was released in November 2007, now referred to as the Android 1.0 beta. In every Android release since, Google has continued to publish the open source code of the operating system under the Apache 2.0 license. AOSP releases are identified by their API level, with each API level corresponding to a particular version of the Android operating system.

The AOSP codebase includes all of the components of the core Android operating system, but none of the proprietary software developed by Google or its partners that are found on most Android smartphones and tablets. For example, Google’s suite of mobile applications (aka GMS, or Google Mobile Services) is not included in AOSP. GMS includes applications like the Play Store (app store), Google’s cloud messaging framework, the Chrome browser, Google search, and more. Branded software and interface experiences of Android device OEMs (like those built by Samsung) are also not part of AOSP. Building a commercial smartphone or tablet using only AOSP code is not generally feasible, as most of the software drivers for components like the cellular modem, Bluetooth radio, and secure hardware components of the SoC (system-on-a-chip) powering a typical Android smartphone are closed source, too. 

None of the above is meant to say that AOSP is “useless” — far from it. Many device manufacturers use AOSP as a starting point for building products, even those without Google Mobile Services. Some examples of non-GMS products built from the AOSP codebase include automotive infotainment systems (now often using Android Automotive Open Source), smart TVs, smart appliances, point-of-sale systems, and smart home devices. Most of these products do require additional software to be fully-featured (such as user interface modifications and specific applications). And closed source drivers are usually necessary for critical hardware components to function, software that is provided under commercial agreements with a component vendor — typically, an SoC manufacturer (examples include Qualcomm, MediaTek, RockChip, Intel, and others).

While AOSP may be an open source codebase, not everyone gets to see that code at the same time. Android partners that work directly with Google are frequently given early access to pre-release Android source code, allowing them to develop products and OTA software updates before that code is publicly released. Generally speaking, early access to Android code requires an agreement with Google to place Google services on your Android devices. Still, this early access to the codebase isn’t the primary incentive for Android partners — Google Mobile Services is. GMS contains what many customers consider “core” smartphone functions, such as an app store, push messaging services, over-the-air updates, and popular applications like Google Maps, Gmail, Google Search, and YouTube.

Google has made no indication that it will ever stop publishing Android as an open source software project. While the value of AOSP as an “out of the box” operating system has admittedly diminished over the years, it remains extremely popular and relevant as the foundational component of countless consumer, enterprise, and even industrial devices.

Manage AOSP Devices

How to make Android (AOSP) without Google

The Android Open Source Project is a completely free and open operating system developed by Google. But as you might know, AOSP isn’t actually Android — it’s missing critical components like the Google Play Store, Google Play Services, and most Google applications. What you might not know is that solving this isn’t as simple as flashing a few apps, and that AOSP even contains its own GMS dependencies that may or may not be documented.

This renders AOSP largely unusable “out of the box” as an operating system. You need to add to it, build on it, and fill feature gaps. But how do you even figure out where those gaps are, let alone how to address them?

On this episode of Android Bytes, Mishaal Rahman and David Ruddock sit down with Chirayu Desai, who works on the CalyxOS project, a privacy-focused Android distribution based on the Android Open Source Project.

Note: The following is an edited transcript of an episode of our podcast, Android Bytes.

What is AOSP vs Android?

[00:00:30] Chirayu Desai

Thanks for having me. CalyxOS is a privacy and security-focused Android custom ROM, distribution, for OS however you want to call it. We basically take what Google provides as open-source, which is called the Android Open Source project, make modifications to it, include a bunch of other awesome open-source projects, and release it as a usable product for the end-user.

[00:00:59] Mishaal

This isn’t the first time we’ve talked about AOSP derivatives on Android Bytes, but we’ve never really dived into what actually makes AOSP less than usable by an end-user by default.

There are a lot of tiny modifications that need to be made – sometimes very major modifications to be made – in order to make AOSP a fully functional operating system for a smartphone. Because if you were to just build AOSP by default, and ship it on a device, the functionality you’d get could hardly be considered smartphone worthy. You’d be missing much that the average user comes to depend on nowadays that it’s not really worth just running pure AOSP, and nobody out there actually just runs AOSP, they all run some kind of derivative of it.

The main reason I wanted to do today’s episode is to basically explain that difference. What does it take to actually get AOSP into a usable state? What is the difference between AOSP builds with and without GMS, and what do projects like CalyxOS offer that actually make it usable for end-users?

To start off, I wanted to bring up a recent story from 9to5Google that said that Google wants to integrate Bluetooth tracker detection “directly into Android.” For those of you who don’t know, there’s a big privacy debacle right now with Bluetooth tile trackers and being able to use them to basically spy on people and track them. Companies like Apple and Google are working on ways to detect that a tracker is nearby and alert the user, “Hey, you might be being tracked right now.”

This is definitely a very welcome feature. But one thing that I took issue with was the claim this is being directly built into Android, which if you actually look past the headline, is false. This feature is being built into Google Play services, not the Android operating system itself. This means that any AOSP based derivative that ships without Google Play Services won’t have access to this feature. This is actually the case for a lot of features, and APIs that you take for granted, that are being part of Android, but they’re not actually in the open-source operating system.

Google is able to say that they’re building this feature directly into Android because Android is actually a trademarked term. They only consider devices that ship AOSP software with GMS to be running “Android.” If you don’t use GMS, you don’t use Android by Google’s definition. Colloquially, we all say, “We have Android. We run Android.” It’s a pretty much-accepted term, but there’s a distinction. Because of this I think there’s probably some confusion about which features are actually part of the open-source operating system and which aren’t.

Chirayu, what are some of the features that people commonly associate with Android but that are not actually part of the OS itself?

[00:04:13] Chirayu

A lot of the fancy features that you see on recent smartphones, some of them might be exclusive to the Pixels, and some of them might be available on multiple devices. A lot of those are implemented by Google either in Play Services or some other proprietary components that they create.

For example, Live Captions. The plumbing for that is all in AOSP. If somebody were to build an equivalent on their own, they wouldn’t have to do all of the work. The base is there, but the actual intelligence for the transcribing functionality is proprietary, that’s something Google has built on their own – it’s not open source.

There was also this simple thing, which is a text selection feature, where you are able to select text from websites, even from the recent app interface, and even select images. That is not part of AOSP, but the APIs, I believe are all present in AOSP.

[00:05:20] Mishaal

Yes. If you were to go on say the Android website for Android 12, there are a lot of features that Google doesn’t really talk about as being part of AOSP or not, which means they don’t tell you whether or not this feature is actually Pixel phone exclusive, or if it’s exclusive to devices with GMS, or if it’s available to all devices through AOSP. There’s not really a distinction being made there. It’s very difficult to tell unless you actually flash AOSP build onto your device and then do a comparison between what’s there and what isn’t on, say, a Pixel device.

If you were to actually go through and flash AOSP onto a device after downloading the source code and compiling it for whatever hardware you have, you would notice that there’s a lot missing in AOSP by default. A lot of features are also present but disabled by default, and there are a lot of things that are just straight-up missing. Most basic features, APIs that a lot of apps depend on and a lot of system apps that you would think are fundamental for a basic phone experience are just not there.

A lot of this stuff is included with Google Mobile Services, which is the suite of apps that Google licenses and distributes to its Android partners. A lot of those features depend on those applications and those APIs that are provided by those Google dependencies.

Of course, most users aren’t going to go out there and flash a pure AOSP build onto a device. But they have no idea what they’re missing if they were to do so.

What GMS and Google dependencies are contained in AOSP?

Chirayu, as someone who has a lot of experience actually building AOSP and going out of your way to include alternatives or workarounds for these missing APIs, can you talk about some of the Google dependencies that are included in fresh AOSP build that users may not be aware of?

[00:07:12] Chirayu

There are a few things that you might think Google would keep for themselves, but they’re actually open source. For example, Android needs to connect to a few internet servers for basic functionality. When you enter a Wi-Fi captive portal, such as at Starbucks, or on a plane, you are asked to log in. To detect that, Android needs to connect to some server. All devices need to connect to some server.

By default, AOSP is configured to connect to a Google server for this, as a fallback – they have to leave something in there. It’s better that it’s working, rather than not having anything there at all. It’s a simple one-line change, and you change it to something else, but you aren’t going to be as reliable as Google. If your server goes down, your users are affected. If Google’s server goes down, pretty much all of an Android is affected. There’s a lot more incentive on them to keep it working.

There’s also some other stuff – not strictly Google components – but things that rely on Google components. For example, we were surprised to see that in AOSP 12 (Android 12), a component relied on Google Play Services for push notifications, of all things. One day a notification just popped up, and I was really surprised; how is this in AOSP? I had to ask someone to double-check that, “am I looking at this right? Did they really add a proprietary dependency like that to AOSP?” And yes, they did.

There are also features that function with Play Services installed, but not with pure AOSP. A common problem with all open-source Android 12 derivatives is that link detection is broken. That is, when you go to a website and tap on a link, it opens in the app if you have it installed. That is not correctly working in AOSP because the thing responsible for it just crashes. Apparently, it’s handled by Play Services on stock, because somebody from Xiaomi, I think, uploaded a patch to Google saying that “This is broken without Play Services. This is how we reproduce it,” but the patch is incomplete. This happens because Google won’t have tested every single feature in AOSP. For them, it’s working with Play Services, so it becomes a lower priority thing.

The funniest indicator that I use to check if I’m running AOSP is holding the power button, which used to bring up the power menu by default. Now on stock Android, I it brings up Google Assistant – but that’s not a thing in AOSP. So what happens? Nothing. How do I know I’m running pure AOSP? You long-press the power button and nothing happens, that’s pure AOSP.

[00:10:00] Mishaal

I think it’s really interesting to see just how much of AOSP is designed around the assumption that you’re running it with GMS included. As you mentioned, the captive portal detection falls back to a Google server. There are various other things that also fall back to Google servers, like the time zone, updating server, whatever server they’re using for assisted GPS, and several other things.

Of course, there’s good reason for these things to exist. Without a captive portal detection server, your phone would just not prompt you to login to whatever Starbucks hotspot you’re trying to connect to. You really need that kind of thing to be there. It’s things like this, it just shows that AOSP is designed to be bundled with GMS. Not everything is really tested without GMS included because that’s not really what Google concerns itself with. It’s concerned only with Android, and Android by definition includes GMS. AOSP is just there to offer Android as an open source operating system. What they really care about is bundling it with GMS and licensing that out to OEMs.

[00:11:34] David

And it’s a matter of what Google has resources and time for. The Android team also bundles so much into GMS because Google does have valid business concerns about protecting certain proprietary features.

I think you see AOSP not necessarily get less attention, but that there’s just not time to try and create either open alternatives or achieve similar functionality where things won’t break. With this approach, Google can just say, “That’s the open-source version. You can create your own solution if you want to.” In the case of some of these features like Live Captions, they’ve left some breadcrumbs if you want to use the framework. But obviously recreating that feature yourself would just be completely impractical, and require a huge corporation with massive resources to accomplish.

[00:12:28] Mishaal

Yes, I think it’s definitely fair to say that for a lot of these new features, they do provide hooks to where OEMs could implement their own features on top of that. Live Caption is a very unique feature. However, that detection service, the machine learning, and making sure all of it is working properly obviously took a lot of work from Google’s teams. I think it’s fair that it’s not included as part of AOSP.

On the other hand, there are some areas in AOSP where it’s clear an open alternative would be in direct competition with Google’s own consumer products. The most notable example being the stock applications that are included in AOSP, like the music, calendar, and calculator apps. They’re just completely bare-bones compared to their GMS counterparts. It’s been a long time since I’ve run a pure AOSP build because I always flash GMS whenever I can.

Which features is AOSP missing?

Chirayu, what are some of the limitations or what are some of the features that are lacking in the pure AOSP builds of common apps like the dialer, the messenger, like a mail client, et cetera?

[00:13:35] Chirayu

One of the things that you will notice is there are just not a lot of apps. You flash your AOSP and I think you see like 10, 11, maybe 12 apps, that’s it. The app drawer is just half full. Then within those apps, you open up something like music, the UI is straight up from Android Gingerbread (2011). The only requirement they have for AOSP apps is that they pass the test suite all OEMs have to pass, which is called CTS. That is what they want. They want it to pass CTS and to serve as a reference to other OEMs. You can tell that all of the Google apps used to be based on AOSP.

There are some other distributions, such as Lineage OS, which try to work on these. They take the AOSP app and they make a lot of improvements to it, basically trying to make it look as close as possible to Google’s versions so that it matches the rest of the system. Then, there are some OEMs who contribute fixes to some of these apps. They all do it through the AOSP Gerrit, which is Google’s thing to accept patches from other people. If you just open that and browse to see whatever is available, a lot of the time you will see fixes from OEMs.

For example, we’ve had an issue where visual voicemail is not working for some people on CalyxOS. It works on some carriers but not all, and it definitely works fine on Google Dialer on the same device. One of our developers was browsing the AOSP Gerrit, and they found a fix from Motorola for visual voicemail issues on some devices. So even OEMs are using some of these apps as a base.

[00:16:07] Mishaal

There are also multiple AOSP apps that are pretty closely kept in sync with their Google counterparts. For example, Launcher3 is pretty close to what you get in Pixel Launcher, and necessary because a lot of OEMs depend on Launcher3. Because so much of Android is based on it – like the gesture navigation – the recent applications UI is integrated, and the taskbar in Android 12L and 13 is integrated into the Launcher3, too. System UI, settings, or other apps are also constantly updated in AOSP. With some apps like music, calendar, and mail, Google is clearly saying, “You know what? We’ll keep them updated so that they pass compatibility tests and that they aren’t completely broken when you compile, but they have much better implementations in GMS, so just go use those.”

And they do still update those apps from time to time. For example, the AOSP Dialer – even though it’s so bare-bones and lacks so many proprietary features. Just like all of the other bits of Android, Google does update it with the new APIs or hooks for the new APIs.

There was a recent video on Twitter where my friend Pierre posted a modification of the AOSP Dialer that basically uses the API that Google added to implement its proprietary call screen functionality. If you have a Pixel phone, you get a notification that says “screen call,” and that uses a whole bunch of Google proprietary magic to automatically interface with the dialer app and determine if the call is spam or not. He basically used that API and replaced the screen call button with a Rick Roll button. Obviously, no OEM is actually is going to do this. But it just goes to show that even though the base AOSP Dialer is so bare-bones, all the hooks are there for OEMs to use. I’m sure many OEMs do base their Dialer applications on the AOSP Dialer, even though it’s probably not ideal to do so because so much is missing. That’s why you see products like LineageOS as Chirayu mentioned go and extend them a whole bunch. They changed pretty much everything about it to make it actually usable by an end-user.

Calyx, on the other hand, seems like a different approach. It seems like Calyx ships a whole bunch of free open-source applications to serve as alternatives to the lacking applications in AOSP.

Chirayu, what are some of the apps that you guys include in Calyx to replace the basics?

[00:18:40] Chirayu

We include some apps from Lineage such as the music app because – well, I think I’ve said enough about it already. We have our own modifications to the dialer, which allows you to make a Signal or Whatsapp call directly from the dialer. It also warns you when you’re making a normal phone call that this call is not private, because normal phone calls that we make are not encrypted, whereas Signal and Whatsapp calls are end-to-end encrypted.

We actually had a big discussion around some of this where people were like, “Why are you promoting Whatsapp?” The thing is, we don’t want to promote anything not open source, but at the same time, Whatsapp has a huge user base – in the billions of people. We do a thing where if you don’t have Signal, we will prompt you to install Signal because it is free, open-source, and awesome. If you don’t have Whatsapp, we won’t show it. If you do have Whatsapp, you can also use this to make a call, and that’s more secure than a normal phone call. We try and make some minor improvements to stuff like this, then we also include F-Droid as an app store just to get open-source apps and also updates for all our apps, which is basically what Google does through the Play Store.

[00:21:34] Mishaal

You mentioned that it’s a necessity to include many applications to give users a full out of box experience, but in reality, it’s impossible to give users everything that they want. Because in order to do so, you’d have to include potentially hundreds of applications to appeal to every user’s use case, and doing so would massively bloat the system image that you’re shipping. What you end up doing is pointing users to other sources to download the applications that they actually need. You mentioned one of them being F-Droid, which is a popular open source repository for open-source applications. Another is Aurora Store.

There are many apps that you won’t find, but using Aurora Store plus F-Droid will cover most of your application needs, I would argue. The issue is that they’re not able to install and update apps in the same way that Google Play is. If you just pick up your Android phone with a Google Play Store, you open it, you download an app, it seamlessly downloads it and installs it in the background. You never get the “Do you want to install this app?” prompt like you do when you try to sideload something. If you were to side load F-Droid and Aurora Store, you would get those prompts every single time you want to install or update an app.

How do you add an app store to AOSP?

On the Calyx website, you guys claim that F-Droid and Aurora are able to seamlessly handle installs and updates, just like Google Play can. How is that possible? What did you do to give them special privileges to streamline this process?

[00:23:13] Chirayu

F-Droid’s had this thing for a long time now, which is what they call the privilege extension. The philosophy of that is, you don’t want to give F-Droid itself additional privileges to just reduce the security of the attack surface. What they have is a small helper app that talks to F-Droid that if you’re rooted, you can install. But if you’re running CalyxOS, we already have it preconfigured. That has the install permissions already given by the OS, so F-Droid talks to that, and that lets F-Droid install any apps it wants.

Aurora Store is also able to use that mechanism, so we had it configured with that. Then in Android 12, Google introduced this new API where apps are able to automatically install updates for themselves, or for other apps, given they meet certain criteria – and there were a few changes required. We contributed those to Aurora Store. We tend to do that, and we like to do that for any open-source project that we include. If we make any modifications or if there’s something useful we can do, we try to give back.

This is now possible with Aurora Store. You can install Aurora Store, the latest version on any Android 12 or above device. For installation, you will have to agree, but I think that is fine. It’s the first time we are installing. It’s a simple, yes. After that, it should be able to update apps without requiring confirmation for every single app from you.

[00:24:36] Mishaal

If I were listening to this, I might say “Okay, this sounds simple enough to bypass Google Play Store. I can just use a combination of F-Droid and Aurora to get whatever I need from either the open-source repositories or Google Play itself. Why do I need GMS? Why do OEMs ship it? What’s the downside to not having GMs, if I can just get all the apps I need through other sources?”

I think if users were to try to go this route, they would quickly discover that there are a lot of applications that would just straight up either not run or would have limited functionality. That’s because they rely on APIs that are only provided through Google Play Services, which is one of the core components of GMS. Some of the APIs that are critical to basic app functionality are only provided through Google Play Services. Functionality in Google Play Services includes the fused location provider API, which is the API that apps use to determine location. There’s also the Firebase Cloud Messaging API, which many apps use to implement push notifications. There’s the Maps API, which is a necessity to integrate Google Maps functionality. Google Maps being of course the best maps application out there, so of course, apps will want to use it.

There are also other services implemented through Play Services, such as the built-in backup transport, which handles backing up app data to your Google Drive account. A lot of apps rely on these Google Play APIs. If Google Play Services is not there, then a lot of them will just fail or just refuse to run or refuse to send you to push notifications, and so on. Obviously, this is not an ideal experience for users because it’s often not clear whether or not an app won’t run on their device.

How do you replace Google services with MicroG on AOSP?

What does CalyxOS provide in terms of alternatives to the APIs? What do you implement to ensure that a lot of these basic features and applications continue to work as expected?

[00:26:24] Chirayu

This is where yet another amazing open-source project comes in. It’s called MicroG. It’s basically an open source reimplementation of Google’s Play Services, APIs, and some of those services. We were talking about this before, that AOSP has APIs for everything. What MicroG does is it provides the implementation or everything that an app would expect from Google Play Services.

They have a page stating what is currently implemented, what is work in progress, and what will never be implemented, such as ads. The project is never going to implement ads for obvious reasons. But they implement push notifications. The way push notifications work is, say you are using WhatsApp. WhatsApp will send a notification to the Google servers, so MicroG has to talk to Google servers to get that notification, because that’s where WhatsApp sent it. However, Google’s server has no idea what’s in that notification – it just knows that I received a message. It doesn’t know from whom or what message did I exactly receive, but MicroG talks to Google servers for this.

The beauty of MicroG is this is all optional. There is simply a toggle for push notifications in there. If you decide that you are not using any apps with push notifications, or you don’t want your device talking to Google servers, just turn that off and it’s gone. What still remains are implementations of some other APIs. You mentioned Maps. For maps, MicroG does not rely on Google instead, it provides this alternative called Mapbox – it builds that in. If you open Uber or if you try to share a location with WhatsApp, you’ll get a map. You may notice the map looks slightly different – instead of Google Maps it will say Mapbox in the bottom left – but for the users, for the most part, it should work the same. This way, you also avoid talking to Google servers as much as possible.

There’s also some other stuff in MicroG that you wouldn’t expect to be in Play Services, such as fonts. Some apps rely on Google Fonts, and that implementation is provided by Play Services. What happens when you try to run those apps on AOSP, they crash. We actually contributed this to MicroG. We created a simple, very simple font implementation that just returns the default font available on the system. You want a font, you just get a font and that’s it. The app works with that.

[00:29:14] David

When it comes to MicroG, I guess I’m just curious, is Google actively trying to disable some of the reverse engineering that is happening there? Is Google try to block access to things that MicroG is attempting to use? We’re talking about the SafetyNet the other week and kind of the back and forth that happens there. MicroG, is that something Google seems to pay attention to or do they seem just, “Okay, letting it lie”?

[00:29:45] Chirayu

I cannot speak on behalf of the MicroG developer at all, so this is strictly my own views. It’s funny that you mention SafetNet, MicroG just recently gained SafetyNet support. SafetyNet is a binary that Google provides that keeps changing from time to time, and it’s heavily obfuscated, you can’t figure out what it’s doing. You don’t know at any point what it’s exactly doing, but MicroG now lets the same thing run and that actually works.

I was really surprised to find that, even if I believe it’s only basic support. But that is enough for a lot of apps that were previously broken to now continue working. I haven’t heard of Google having any issues with this. One thing it does provide is Google account login. This is also something that’s in Play Services just like you take your phone, when you log into a Google account but that is also provided by Play Services, and all of the stuff that comes with it like contacts sync, that is another proprietary APK.

You think you’d get contact sync but no, that’s also a separate thing. That’s also something MicroG provides, but in addition to that, it tries to use as little of Google as possible. The best example that I mentioned before was push notifications, you have to use Google – that’s where the notifications are, but for maps, you can use anything you want. Apps just want a map. Apps don’t specifically need Google Maps, other maps mostly work fine.

[00:31:13] David

Unlike Widevine, Google doesn’t have a specific reason to break something like this. If anything, it’s probably helpful as far as they’re concerned.

[00:31:22] Chirayu

Yes, I’d like to think so.

How do you get location services or data backup on AOSP?

[00:31:26] Mishaal

How about things like location services and backup? How does Calyx offer alternatives to those?

[00:31:33] Chirayu

MicroG has a very modular thing for location services, they call it UnifiedNLP, where NLP is Network Location Provider. What they have is, you are able to install various APKs. We include some of them, such as Mozilla, and one called DejaVu, which runs entirely on your device. There are other APKs available in F-Droid, such as Apple Wi-Fi Backend, which uses Apple’s Wi-Fi database to figure out your location on a Google device. So you don’t need to rely on Google for the location, you can just try and figure out the location from some other database. We use a combination of Mozilla, which has their own Wi-Fi database, then also DejaVu, which tries to build a local database on the device. That way location in most apps continue working.

Some things like data backup, I believe on stock, are part of Play Services simply because of convenience. You have to think like Google. If you add something, if Google adds something to Android 13 for example, that will only get to a small number of devices at the start. Even after a year or so it will still be a small percentage of the entire Android device population. Whereas with Play Services – and they recently did this with the permission auto-reset feature, they put it in Play Services – and it became available to billions of Android devices overnight, just like that. But they include backup in there and the backup to Google servers.

What we have instead is our own open source solution known as SeedVault. It already existed as a community project and then I think we came in and ended up basically rewriting the entire app. We funded that on our own and then we also got some external funding to work on some features for it. At this point, it’s able to back up your apps, the data of third party apps, such as your email application. It’s also able to back up your files, photos, documents, and everything. It does all this encrypted on the device. That way, your private data never leaves your device unprotected.

Then we use another Google API for choosing the backup location. You can backup your own device, which is just mostly for testing or if you are transferring it were some other way or the recommended option is you backup to a USB stick, which you can connect to the device with an adapter or even directly these days. Then we also support Nextcloud and other backends. This is all open source. It’s a separate project hosted on its own GitHub, and we are happy to see that it’s included in many of the custom ROMs. We also get reports that people are switching between custom ROMs using this backup tool, which is great to see and one of the intentions is to make switching phones easier now. If you’re doing CalyxOS or even if you’re using a different OS, that’s totally fine. We are glad that it works.

[00:35:02] David

That’s one where I think it’s a great example of the community having more “pure” motivations, I guess, than a Samsung or a Google or a OnePlus, where they’re really concerned about the data from their own applications and services first and foremost. That can become basically a technical blocker to a more modular or even just more robust backup and restore architecture within Android. Now I just do cloud restores when I set up a new phone, because the USB method is incredibly unreliable with Google’s solution, it doesn’t work half the time. The cloud restore is an incomplete solution, but it’s better than nothing.

It’s interesting to see, because when I used to flash custom ROMs way back in like the Nexus One days, it was fairly easy to make your data portable with a Nandroid backup and bring everything over. Granted, Android was a lot simpler back then, but it’s interesting to see that dynamic where, because you have this ecosystem of partners, nobody really is interested in a universal standard and you have competing business interests that really prevent it.

[00:36:08] Mishaal

Yes, back up and all these other APIs provided by Google certainly solve a lot of problems that fragmentation would otherwise create.

Because Google Play Services is incredibly obfuscated and closed source, it’s not possible to implement everything provided by Google Play Services immediately into MicroG, which is why it’s missing a lot of things that might make something like Android Auto work, for example. It provides most of the basic APIs that are needed to get most apps at least working on your device. In order to do so, I wanted to ask about the implementation, and how MicroG is actually integrated into a device build. Because that’s a very sticky question.

The way apps call Google APIs is they expect Google Play Services itself to be installed. They expect that it’s a legitimate version of Google Play Services provided by Google. Instead, MicroG kind of hijacks that process and pretends that it’s actually Google Play Services. It’s saying, “Hey, we’re the legit thing. Let us handle your API calls. Let us handle this data, and send things back to you.” It kind of spoofs the real Play Services and in order to do that, and custom ROMs have to implement something called signature spoofing, which is basically saying that the signature of MicroG is actually matching Google Play Services, even though it doesn’t.

This obviously is not ideal for security purposes, because you expect that when you’re installing one application on top of the other, the signatures match, therefore, and you rely on that to install that update.

Is MicroG safe? How does MicroG signature spoofing work?

With signature spoofing, you’d be able to bypass that signature matching and that wouldn’t be good for security. How does Calyx integrate MicroG while addressing this signature spoofing security concern?

[00:37:57] Chirayu

I want to correct you on that one, if that’s fine. Signature spoofing won’t let you install an app signed with a different key. If you take a step back, the way apps on Android work is every single app is signed with a private key that the developer of the app holds. Even in the operating system, we include apps that are signed by other people. Or we sign our own apps with our private keys, which you and I don’t have access to.

Updates to those apps can only be installed if they’re signed by the same key. What happens with MicroG pretending to be Play Services is that Aurora Store will think that there is an update available for Play Services, but the update won’t actually install – because they’re signed with different keys. That part of the OS which verifies that was, to my knowledge, even with all the other signature spoofing implementations, that never got modified.

Instead what’s modified is simple: When an app asks what signature this is signed with, we just reply with Google’s instead of ours (or rather, whatever MicroG signed with). What used to happen is MicroG had some patches released separately on their own, which were just meant for reference – and you are obviously need to restrict them in whatever way possible. Those patches were very open, but what we do is we take them and we heavily restrict it. The first thing is MicroG and only MicroG can spoof signature, no other app – none whatsoever. There are no exceptions, only MicroG.

The second is it can only spoof Google’s signature, not arbitrary signatures. That was one of the things people didn’t like about the old patches – which we never included by the way – that any app could pretend to be signed with any key. We don’t allow that. We are just like “MicroG needs to pretend to be Google to work, fine let’s do that. Anything else not allowed.”

What we like to do is rely on Google’s protections for this. We have guarded this behind a privileged permission that only system applications can get. Since on Calyx OS MicroG is a system application. It can get that, but no other app that you installed can get that permission. My thinking behind this is, if privileged permissions are broken, if there is a bug in that, that is a much bigger problem for Google than just for us at CalyxOS. They’ll deal with that. As long as we try and rely on Google security practices and try to work within those, we can still aim to have a good level of security.

[00:40:43] Mishaal

Good to know. Thanks for the correction there. I was mixing two concepts together where I used to in the past. There were some Xposed modules that would completely disable signature checking, and I confused that with signature spoofing. Good to know that the fundamental application security model of signature verification is not violated in any way with MicroG and signature spoofing. It’s also good to know that you guys limit the scope of what MicroG needs or what you need to include in order to make MicroG actually function on CalyxOS.

If you install CalyxOS, I’m sure, most users if you’re looking for a privacy and security-oriented operating system for your Android device, you probably get a lot of basic function covered by the stock applications included with Calyx. Then wherever you don’t find is fulfilled by those applications, you can grab from another source like F-Droid or Aurora Store.

Why would you use AOSP without GMS?

Why go to all this effort of customizing AOSP with all of these applications and apps? Why do this instead of just shipping AOSP with GMS?

[00:41:51] Chirayu

First of all, we can’t legally ship AOSP with GMS. I don’t think Google allows anyone to ship AOSP with GMS, apart from themselves. Secondly, we are all about free and open-source software, GMS is not that. Now some of the more technical among you may point out that we still need to rely on proprietary software to even get these devices to boot. I think that is acceptable.

What we do is – anything you need to get the device working or basic functionality – that’s fine if that’s proprietary, but anything else should be open source. Relying on GMS means you have to go through the entire Google approval process, their entire certification process which is called CTS, which we discussed earlier. You make a build and that build has to pass CTS, which can take days to run. Google’s partners or all the OEMs work around this by having early access to builds. We got Android 12 October last year, partners would’ve had Android 12 months before. They would be able to start working on it and get it all certified and everything, but we get it in October. If we want to get an update out quickly, we can’t possibly go through that entire process. What we do is we provide MicroG as an option.

When you first install CalyxOS, you are greeted with a screen to enable MicroG. If you would you like to use apps such as Google Maps or YouTube or Uber or ride sharing apps, then you probably want it on, but if you’re not planning to use any of these apps, you can turn that off. I don’t see Google letting people turn off Play Services, right from the start of install. They do let you disable it on stock, but then you’ll hit with these annoying notifications from every single Google app that this app will not work without Play Services. Some of them actually don’t.

Another thing we did – last month – was the Dirty Pipe vulnerability. We were able to get an update out in, I would say about six hours from when we first came to know about that vulnerability. I’m still thinking of ways to make that even faster. A bit of it is limited by the time it simply takes to build the OS, because Android is so huge, it can take up to an hour simply to compile the operating system, and then you have to sign it. You have to sign every single component in the OS, but even that can take a long time. Then one of the things we do at CalyxOS is we test every single build. You don’t get that with AOSP on its own. Every single download that we provide has been tested and is known working. There will always be some issues, of course, there are a lot of issues in AOSP itself. But at least the basic functionality, the device boots, it won’t die on you, phone calls will work and so on.

[00:44:45] David

I think this is a pretty good case in point, especially as Android has become more mature, for custom distros becoming much harder to build – in a way that’s usable, at least. Because Google has started to wall off so many things and put them behind Google Play Services, and other GMS components.

There’s also the question you kind of imply, Chirayu, which is one of trust. Why does Calyx decide that, for example, this is a good PDF viewer application to preload? Because it’s free and open-source and it looks like a trustworthy piece of software that’s maintained in the community. Asking users to identify that software, especially from the ground up would be a big ask for them. They don’t know what the trustworthy software is, what’s being kept up to date.

Just saying that, “it’s AOSP with a few things extra on top,” that’s not true. It is an entire set of choices that have been made very actively for this distro to work in this way. You have compatibilities like MicroG and letting the user choose if they want to have that or not. Because even having it architected in that way, where you have essentially two versions of the OS – it’s going to be fine without MicroG, and one that’s going to work with MicroG – that again, is a choice. We have two very different UXs there that people are going to have.

It’s interesting to see the challenges there and also just the amount of curation that still has to happen. It’s not just an open-source platform, it’s a series of choices that somebody a large group of people have made to present people with new options and new ways to use the device. I think that that can get overshadowed by just like, “Okay, do you have feature X, Y, or Z?” It’s in the aggregate that the distro itself becomes usable to people once you have all those choices taken together.

[00:46:32] Mishaal

I think that’s the beauty of AOSP, which is choice. When you’re building AOSP, you choose which providers to use for the basic captive portal detection, et cetera. You choose which applications to build and ship. You choose what security features to enable whether or not you want to harden Android with additional kernel, configs, whether you want to extend AOSP privacy features with more features. You choose when to ship your build.

As Chirayu mentioned, he’s able to include the kernel fixes for Dirty Pipe and ship them out before those fixes are even included in Google devices. Because when you’re on a contract with Google and you have to include GMS, and you have to do certification testing, and then you got to do carrier testing, et cetera, that can add weeks, maybe even months onto your ship time. When you just build from AOSP and you ship to X, Y, Z device that you control, you control when you get that software update out. That can be much quicker than if you had to go through a lot of these certification tests that are required to ship GMS. AOSP provides you much more control over the software that you’re delivering.

[00:47:41] David

And at Esper, obviously, we’re not working with consumer smartphones, we’re not working with any consumer tablets or laptops or things like that, that could run AOSP-based distros like Calyx OS. We’re working with things like point of sale terminals, or customer service kiosks, display signage, connected exercise equipment, two-way communication devices over cellular even. Systems that do very specific things – devices, I should say that do very specific things – do them all day, every day, we call them dedicated devices. Sometimes they’re known as “COSU”s in the computing world.

If you’re trying to build something with AOSP, whether that is a point of sale terminal, or whether it’s something you’re going to put on a customer’s wrist to take health metrics as a medical device, you’re probably wondering “What pieces of AOSP do I need? What pieces do I need to know about where should I be looking? Where do I even start making these decisions?” If you’re you’re deciding not just on a platform, but also how to use Android on your device, you should come talk to us at Esper! We do this. We have our own operating system distro called Foundation, which is based on AOSP.

We also offer robust services for things like deploying apps to devices. If you have GMS, you have an app deployment infrastructure, it’s called the Google Play Store. If you’re running AOSP, you don’t have GMS, so you don’t have an app deployment infrastructure! Being able to do these things that you would just take for granted on a smartphone on something like a cash register is a totally different scenario, and it does require a specialized approach.

If you’re in that world, and you’re building a device that could ostensibly run Android, but would probably need some changes or modifications, come talk to us at Esper. It’s esper.io and you can book a demo, and we’re happy to show you what we can do. We can manage fleets of devices in the dozens or hundreds. We can manage them to 10s or hundreds of thousands depending on your use case.

[00:49:47] Mishaal

Yes. Thanks, David and thank you, Chirayu, for joining us on today’s episode. I hope this has been illuminating for those of you who are listening who have considered whether or not you want to run a pure AOSP build on your device. The answer is no, you don’t want to run a pure AOSP build.

You want to run an AOSP derivative that takes care of a lot of the minor issues that come with just running AOSP and all of its lacking features. A project like CalyxOS is one such option you might turn to because they took a lot of legwork out by including a lot of stock applications. Some of which are developed by themselves, some of which they’re sourced from the open-source community. They also keep it up to date on the devices they support, so you don’t have to take care of merging patches on your own, making it much simpler.

I hope this episode has informed you about what it takes to actually make AOSP usable on a smartphone, which is a lot. Surprisingly, you think AOSP is built to support smartphones, tablets, automobiles, watches, and other form factors, but just building it and shipping it on a device is not enough to actually have something usable, especially not for an enterprise as David mentioned. Thank you, Chirayu, for joining us in today’s episode. If people want to follow you or your project, where can they find you?

[00:51:06] Chirayu

You can find CalyxOS at calyxos.org. We have a very active community and you’ll find links to that on the website. If you like to support our work, you can go to calyxinstitute.org and get a membership, and that directly supports what we’re doing. Thank you for having me.

Manage AOSP Devices with Esper