Certificate blacklisting in Jelly Bean

The last two posts introduced app encryption, the new system key store and a few other security related features introduced in Jelly Bean. Browsing the ASOP code reveals another new feature which sits higher in the security stack than the previously discussed ones -- certificate blacklisting. In this article we will present some details about its implementation and introduce a sample app that allows us to test how blacklisting works in practice.

Why blacklist certificates?

In a perfect world, a working Public Key Infrastructure (PKI) takes care of issuing, distributing and revoking certificates as necessary. All that a system needs to verify the identities of previously unknown machines and users are a few trust anchor certificates. In practice, though, there are number of issues. Those have been known for some time, but the recent breaches in top-level CAs have shown that the problems and their consequences are far from theoretical. Probably the biggest PKI issue is that revocation of root certificates is not really supported. Most OSes and browsers come with a pre-configured set of trusted CA certificates (dozens of them!) and when a CA certificate is compromised there are two main ways to handle it: 1. tell users to remove it from the trust store; or, 2. issue an emergency update that removes the affected certificate. Expecting users to handle this is obviously unrealistic, so that leaves the second option. Windows modifies OS trust anchors by distributing patches via Windows Update, and browser vendors simply release a new patch version. However, even if an update removes a CA certificate from the system trust store, a user can still install it again, especially when presented with a 'do this, or you can't access this site' ultimatum. To make sure removed trust anchors are not brought back, the hashes of their public keys are added to a blacklist and the OS/browser rejects them even if they are in the user trust store. This approach effectively revokes CA certificates (within the scope of the OS/browser, of course) and takes care of PKI's inability to handle compromised trust anchors. However, it's not exactly ideal: even an emergency update takes some time to prepare, and even after it is out some users won't update right away, no matter how often they are being nagged about it. CA compromises are relatively rare and widely publicized though, so it seems to work OK in practice (for now, at least).

While CA breaches are fairly uncommon, end entity (EE) key compromise occurs much more often. Whether due to a server breach, stolen laptop or a lost smart card, it happens daily. Fortunately, modern PKI systems have been designed with this in mind -- CAs can revoke certificates and publish revocation information in the form of CRLs, or provide online revocation status using OCSP. Unfortunately, this doesn't really work in the real world. Revocation checking generally requires network access to a machine different from the one we are trying to connect to, and as such has a fairly high failure rate. To mitigate this most browsers do their best to fetch fresh revocation information, but if this fails for some reason, they simply ignore the error (soft-fail), or at best show some visual indication that revocation information is not available. To solve this Google Chrome has opted to disable online revocation checks altogether, and now uses its online update mechanism to proactively push revocation information to browsers, without requiring an application update or restart. Thus Chrome can have an up-to-date local cache of revocation information which makes certificate validation both faster and more reliable. This is yet another blacklist (Chrome calls it a 'CRL set'), this time based on information published by each CA. The browser vendor effectively managing revocation data on the user's behalf is quite novel, and not everyone thinks it's a good idea, but it has worked well so far.

Android certificate blacklisting

In Android versions prior to 4.0 (Ice Cream Sandwich, ICS), the system trust store was a single Bouncy Castle key store file. Modifying it without root permissions was impossible and the OS didn't have a supported way to amend it. That meant that adding new trust anchors or removing compromised ones required an OS update. Since, unlike regular desktop OSes, updates are generally handled by carriers and not the OS vendor, they are usually few and far between. What's more, if a device doesn't sell well, it may never get an official update. In practice this means that there are thousands of devices that still trust compromised CAs, or don't trust newer CAs that have issued hundreds of web site certificates. ICS changed this by making the system trust store mutable and adding an UI, as well as an SDK API, that allows for adding and removing trust anchors. This didn't quite solve PKI's number one problem though -- aside from the user manually disabling a comprised trust anchor, an OS update was still required to blacklist a CA certificate. Additionally, Android does not perform online revocation checks when validating certificate chains, so there was no way to detect compromised end entity certificates, even if they have been revoked.

This finally leads us to the topic of the article -- Android 4.1 (Jelly Bean, JB) has taken steps to allow for online update of system trust anchors and revocation information by introducing certificate blacklists. There are now two system blacklists:

• a public key hash blacklist (to handle compromised CAs)
• a serial number blacklist (to handle compromised EE certificates)

The certificate chain validator component takes those two lists in consideration when verifying web site or user certificates. Let's look at how this implemented in a bit more detail.

Android uses a content provider to store OS settings in a system databases. Some of those settings can be modified by third party apps holding the necessary permissions, while some are reserved for the system and can only be changed by going through the system settings UI, or by another system application. The latter are known as 'secure settings'. Jelly Bean adds two new secure settings under the following URIs:

• content://settings/secure/pubkey_blacklist
• content://settings/secure/serial_blacklist

As the names imply, the first one stores public key hashes of compromised CAs and the second one a list of EE certificate serial numbers. Additionally, the system server now starts a CertiBlacklister component which registers itself as a ContentObserver for the two blacklist URIs. Whenever a new value is written to those, the CertBlacklister gets notified and writes the value to a file on disk. The format of the files is simple: a comma delimited list of hex-encoded public key hashes or certificate serial numbers. The actual files are:

• certificate blacklist: /data/misc/keychain/pubkey_blacklist.txt
• serial number blacklist: /data/misc/keychain/serial_blacklist.txt

Why write them to disk when they are already available in the settings database? Because the component that actually uses the blacklists is a standard Java CertPath API class that doesn't know anything about Android and it's system databases. The actual class, PKIXCertPathValidatorSpi, is part of the Bouncy Castle JCE provider, modified to handle certificate blacklists, which is an Android-specific feature and not defined in the standard CertPath API. The PKIX certificate validation algorithm the class implements is rather complex, but what Jelly Bean adds is fairly straightforward:

• when verifying an EE (leaf) certificate, check if it's serial number is in the serial number blacklist. If it is, return the same error (exception) as if the certificate has been revoked.
• when verifying a CA certificate, check if the hash of it's public key is in the public key blacklist. If it is, return the same error as if the certificate has been revoked.

The certificate path validator component is used throughout the whole system, so blacklists affect both applications that use HTTP client classes and the native Android browser and WebView. As mentioned above, modifying the blacklists requires system permissions, so only core system apps can use it. There are no apps in the AOSP source that actually call those APIs, but a good candidate to manage blacklists are the Google services components, available on 'Google experience' devices (i.e., devices with the Play Store client pre-installed). Those manage Google accounts, access to Google services and provide push-style notifications (aka, Google Client Messaging, GCM). Since GCM allows for real-time server-initiated push notifications, it's a safe bet that those will be used to trigger certificate blacklist updates (in fact, some source code comments hint at that). This all sounds good on paper (well, screen actually), but let's see how well it works on a real device. Enough theory, on to

Using Android certificate blacklisting

As explained above, the API to update blacklists is rather simple: essentially two secure settings keys, the values being the actual blacklists in hex-encoded form. Using them requires system permissions though, so our test application needs to either live in /system/app or be signed with the platform certificate. As usual, we choose the former for our tests. A screenshot of the app is shown below.

The app allows us to install a CA certificate to the system trust store (using the KeyChain API), verify a certificate chain (consisting of a the CA certificate and a single EE certificate), add either of the certificates to the system blacklist, and finally clear it so we can start over. The code is quite straightforward, see github repository for details. One thing to note is that it instantiates the low level org.bouncycastle.jce.provider.CertBlacklist class in order to check directly whether modifying the blacklist succeeded. Since this class is not part of the public API, it is accessed using reflection.

Some experimentation reveals that while the CertiBlacklister observer works as expected and changes to the blacklists are immediately written to the corresponding files in /data/misc/keychain, verifying the chain succeeds even after the certificates have been blacklisted. The reason for this is that, as all system classes, the certificate path validator class is pre-loaded and shared across all apps. Therefore it reads the blacklist files only at startup, and a system restart is needed to have it re-read the files. After a restart, validation fails with the expected error: 'Certificate revocation of serial XXXX'. Another issue is that while blacklisting by serial number works as expected, public key blacklisting doesn't appear to work in the current public build (JRO03C on Galaxy Nexus as of July 2012). This is a result of improper handling of the key hash format and will hopefully be fixed in a next JB maintenance release. Update: it is now fixed in AOSP master.

Summary

In Jelly Bean, Android takes steps to get on par with the Chrome browser with respect to managing certificate trust. It introduces features that allow for modifying blacklists dynamically: based on push notifications, and without requiring a system update. While the current implementation has some rough edges and does require a reboot to apply updates, once those are smoothed out, certificate blacklisting will definitely contribute to making Android more resilient to PKI-related attacks and vulnerabilities.