Certificate pinning in Android 4.2

A lot has happened in the Android world since our last post, with new devices being announced and going on and off sale.  Most importantly, however, Android 4.2 has been released and made its way to AOSP. It's an evolutionary upgrade, bringing various improvements and some new  user and developer features. This time around, security related enhancements made it into the what's new  list, and there is quite a lot of them. The most widely publicized one has been, as expected, the one users may actually see -- application verification. It recently got an in-depth analysis, so in this post we will look into something less visible, but nevertheless quite important -- certificate pinning. 

PKI's trust problems and proposed solutions

In the highly unlikely case that you haven't heard about it, the trustworthiness of the existing public CA model has been severely compromised in the recent couple of years. It has been suspect for a while, but recent high profile CA security breaches have brought this problem into the spotlight. Attackers managed to issue certificates for a wide range of sites, including Windows Update servers and Gmail. Not all of those were used (or at least not detected) in real attacks, but the incidents showed just how much of current Internet technology depends on certificates. Fraudulent ones can be used for anything from installing malware to spying to Internet communication, and all that while fooling users that they are using a secure channel or installing a trusted executable. And better security for CA's is not really a solution: major CA's have willingly issued hundreds of certificated for unqualified names such as localhost, webmail and exchange (here is a breakdown, by number of issued certificates). These could enable eavesdropping on internal corporate traffic by using the certificates for a man-in-the-middle (MITM) attack against any internal host accessed using an unqualified name. And of course there is also the matter of compelled certificate creation, where a government agency could compel a CA to issue a false certificate to be used for intercepting secure traffic (and all this may be perfectly legal). 

Clearly the current PKI system, which is largely based on a pre-selected set of trusted CA's (trust anchors), is problematic, but what are some of the actual problems? There are different takes on this one, but for starters, there are too many public CA's. As this map by the EFF's SSL Observatory project shows, there are more than public 650 CA's trusted by major browsers. Recent Android versions ship with over one hundred (140 for 4.2) trusted CA certificates and until ICS the only way to remote a trusted certificate was a vendor-initiated OS OTA. Additionally, there is generally no technical restriction to what certificates CA's can issue: as the Comodo and DigiNotar attack have shown, anyone can issue a certificate for *.google.com (name constraints don't apply to root CA's and don't really work for a public CA). Furthermore, since CA's don't publicize what certificates they have issued, there is no way for site operators (in this case Google) to know when someone issues a new, possibly fraudulent, certificate for one of their sites and take appropriate action (certificate transparency standards aims to address this). In short, with the current system if any of the built-in trust anchors is compromised, an attacker could issue a certificate for any site, and neither users accessing it, nor the owner of the site would notice. So what are some of the proposed solutions? 

Proposed solutions range from radical: scrape the whole PKI idea altogether and replace it with something new and better (DNSSEC is a usual favourite); and moderate: use the current infrastructure  but do not implicitly trust CA's; to evolutionary: maintain compatibility with the current system, but extend it in ways that limit the damage of CA compromise. DNSSEC is still not universally deployed, although the key TLD domains have already been signed. Additionally, it is inherently hierarchical and actually more rigid than PKI, so it doesn't really fit the bill too well. Other even remotely viable solutions have yet to emerge, so we can safely say that the radical path is currently out of the picture. Moving towards the moderate side, some people suggest the SSH model, in which no sites or CA's are initially trusted, and users decide what site to trust on first access. Unlike SSH however, the number of sites that you access directly or indirectly (via CDN's, embedded content, etc.) is virtually unlimited, and user-managed trust is quite unrealistic. Of a similar vein, but much more practical is Moxie Marlinspike's (of sslstrip and CloudCracker fame) Convergence. It is based on the idea of trust agility, a concept he introduced in his SSL And The Future Of Authenticity talk (and related blog post). It both abolishes the browser (or OS) pre-selected trust anchor set, and recognizes that users cannot possibly independently make trust decisions about all the sites they visit. Trust decisions are delegated to a set of notaries, that can vouch for a site by basically confirming that the certificate you receive from a site is one they have seen before. If multiple notaries point out the same certificate as correct, users can be reasonably sure that it is genuine and therefore trustworthy. Convergence is not a formal standard, but was released as actual working code including a Firefox plugin (client) and server-side notary software. While this system is promising, the number of available notaries is currently limited, and Google has publicly stated that it won't add it to Chrome, and it cannot currently be implemented as an extension either (Chrome lacks the necessary API's to let plugins override the default certificate validation module).

That leads us to the current evolutionary solutions, which have been deployed to a fairly large user base, mostly courtesy of the Chrome browser. One is certificate blacklisting, which is more of a band-aid solution: in addition to removing compromised CA certificates from the trust anchor set with a browser update, it also explicitly refuses to trust their public keys in order to cover the case where they are manually added to the trust store again. Chrome added blacklisting around the time Comodo was compromised, and Android has this feature since the original Jelly Bean release (4.1). The next one, certificate pinning (more accurately public key pinning), takes the converse approach: it whitelists the keys that are trusted to sign certificates for a particular site. Let's look at it in a bit more detail.

Certificate pinning

Pinning was introduced in Google Chrome 13 in order to limit the CA's that can issue certificates for Google properties. It actually helped discover the MITM attack against Gmail, which resulted from the DigiNotar breach. It is implemented by maintaining a list of public keys that are trusted to issue certificates for a particular DNS name. The list is consulted when validating the certificate chain for a host, and if the chain doesn't include at least one of the whitelisted keys, validation fails. In practice the browser keeps a list of SHA1 hashes of the SubjectPublicKeyInfo (SPKI) field of trusted certificates. Pinning the public keys instead of the actual certificates allows for updating host certificates without breaking validation and requiring pinning information update. You can find the current Chrome list here.

As you can see, the list now pins non-Google sites as well, such as twitter.com and lookout.com, and is rather large. Including more sites will only make it larger, and it is quite obvious that hard-coding pins doesn't really scale. A couple of new Internet standards have been proposed to help solve this scalability problem: Public Key Pinning Extension for HTTP (PKPE) by Google and Trust Assertions for Certificate Keys (TACK) by Moxie Marlinspike. The first one is simpler and proposes a new HTTP header (Public-Key-Pin, PKP) that holds pinning information including public key hashes, pin lifetime and whether to apply pinning to subdomains of the current host. Pinning information (or simply 'pins') is cached by the browser and used when making trust decisions until it expires. Pins are required to be delivered over a secure (TLS) connection, and the first connection that includes a PKP header is implicitly trusted (or optionally validated against pins built into the client). The protocol also supports an endpoint to report failed validations to via the report-uri directive and allows for a non-enforcing mode (specified with the Public-Key-Pins-Report-Only header), where validation failures are reported, but connections are still allowed. This makes it possible to notify host administrators about possible MITM attacks against their sites, so that they can take appropriate action. The TACK proposal, on the other header, is somewhat more complex and defines a new TLS extension (TACK) that carries pinning information signed with a dedicated 'TACK key'. TLS connections to a pinned hostname require the server to present a 'tack' containing the pinned key and a corresponding signature over the TLS server's public key. Thus both pinning information exchange and validation are carried out at the TLS layer. In contrast, PKPE uses the HTTP layer (over TLS) to send pinning information to clients, but also requires validation to be performed at the TLS layer, dropping the connection if validation against the pins fails. Now that we have an idea how pinning works, let's see how it's implemented on Android.

Certificate pinning in Android

As mentioned at beginning of the post, pinning is one of the many security enhancements introduced in Android 4.2. The OS doesn't come with any built-in pins, but instead reads them from a file in the /data/misc/keychain directory (where user-added certificates and blacklists are stored). The file is called, you guessed it, simply pins and is in the following format: hostname=enforcing|SPKI SHA512 hash, SPKI SHA512 hash,.... Here enforcing is either true or false and is followed by a list of SPKI hashes (SHA512) separated by commas. Note that there is no validity period, so pins are valid until deleted. The file is used not only by the browser, but system-wide by virtue of pinning being integrated in libcore. In practice this means that the default (and only) system X509TrustManager implementation (TrustManagerImpl) consults the pin list when validating certificate chains. However there is a twist: the standard checkServerTrusted() method doesn't consult the pin list. Thus any legacy libraries that do not know about certificate pinning would continue to function exactly as before, regardless of the contents of the pin list. This has probably been done for compatibility reasons, and is something to be aware of: running on 4.2 doesn't necessarily mean that you get the benefit of system-level certificate pins. The pinning functionality is exposed to third party libraries or SDK apps via the new X509TrustManagerExtensions SDK class. It has a single method, List<X509Certificate> checkServerTrusted(X509Certificate[] chain, String authType, String host) that returns a validated chain on success or throws a CertificateException if validation fails. Note the last parameter, host. This is what the underlying implementation (TrustManagerImpl) uses to search the pin list for matching pins. If one is found, the public keys in the chain being validated will be checked against the hashes in the pin entry for that host. If none of them matches, validation will fail and you will get a CertificateException. So what part of the system uses the new pinning functionality then? The default SSL engine (JSSE provider), namely the client handshake (ClientHandshakeImpl) and SSL socket (OpenSSLSocketImpl) implementations. They would check their underlying X509TrustManager and if it supports pinning, they will perform additional validation against the pin list. If validation fails, the connection won't be established, thus implementing pin validation on the TLS layer as required by the standards discussed in the previous section. We now know what the pin list is and who uses it, so let's find out how it is created and maintained.

First off, at the time of this writing, Google-managed (on Nexus devices) JB 4.2 installations have an empty pin list (i.e., the pins file doesn't exist). Thus certificate pinning on Android has not been widely deployed yet. Eventually it will be, but the current state of affairs makes it easier to play with, because restoring to factory state requires simply deleting the pins file and associated metadata (root access required). As you might expect, the pins file is not written directly by the OS. Updating it is triggered by a broadcast (android.intent.action.UPDATE_PINS) that contains the new pins in it's extras. The extras contain the path to the new pins file, its new version (stored in /data/misc/keychain/metadata/version), a hash of the current pins and a SHA512withRSA signature over all the above. The receiver of the broadcast (CertPinInstallReceiver) will then verify the version, hash and signature, and if valid, atomically replace the current pins file with new content (the same procedure is used for updating the premium SMS numbers list). Signing the new pins ensures that they can only by updated by whoever controls the private signing key. The corresponding public key used for validation is stored as a system secure setting under the "config_update_certificate" key (usually in the secure table of the
/data/data/com.android.providers.settings/databases/settings.db) Just like the pins file, this value currently doesn't exists, so its relatively safe to install your own key in order to test how pinning works. Restoring to factory state requires deleting the corresponding row from the secure table. This basically covers the current pinning implementation in Android, it's now time to actually try it out.

Using certificate pinning

To begin with, if you are considering using pinning in an Android app, you don't need the latest and greatest OS version. If you are connecting to a server that uses a self-signed or a private CA-issued certificate, chances you might already be using pinning. Unlike a browser, your Android app doesn't need to connect to practically every possible host on the Internet, but only to a limited number of servers that you know and have control over (limited control in the case of hosted services). Thus you know in advance who issued your certificates and only need to trust their key(s) in order to establish a secure connection to your server(s). If you are initializing a TrustManagerFactory with your own keystore file that contains the issuing certificate(s) of your server's SSL certificate, you are already using pinning: since you don't trust any of the built-in trust anchors (CA certificates), if any of those got compromised your app won't be affected (unless it also talks to affected public servers as well). If you, for some reason, need to use the default trust anchors as well, you can define pins for your keys and validate them after the default system validation succeeds. For more thoughts on this and some sample code (doesn't support ICS and later, but there is pull request with the required changes), refer to this post by Moxie Marlinspike. Update: Moxie has repackaged his sample pinning code in an easy to use standalone library. Update 2: His version uses a static, app-specific trust tore. Here's a fork that uses the system trust store, both on pre-ICS (cacerts.bks) and post-ICS (AndroidCAStore) devices.

Before we (finally!) start using pinning in 4.2 a word of warning: using the sample code presented below both requires root access and modifies core system files. It does have some limited safety checks, but it might break your system. If you decide to run it, make sure you have a full system backup and proceed with caution.

As we have seen, pins are stored in a simple text file, so we can just write one up and place it in the required location. It will be picked and used by the system TrustManager, but that is not much fun and is not how the system actually works. We will go through the 'proper' channel instead by creating and sending a correctly signed update broadcast. To do this, we first need to create and install a signing key. The sample app has one embedded so you can just use that or generate and load a new one using OpenSSL (convert to PKCS#8 format to include in Java code). To install the key we need the WRITE_SECURE_SETTINGS permission, which is only granted to system apps, so we must either sign our test app with the platform key (on a self-built ROM) or copy it to /system/app (on a rooted phone with stock firmware). Once this is done we can install the key by updating the "config_update_certificate" secure setting:

Settings.Secure.putString(ctx.getContentResolver(), "config_update_certificate", 
                          "MIICqDCCAZAC...");

If this is successful we then proceed to constructing our update request. This requires reading the current pin list version (from /data/misc/keychain/metadata/version) and the current pins file content. Initially both should be empty, so we can just start off with 0 and an empty string. We can then create our pins file, concatenate it with the above and sign the whole thing before sending the UPDATE_PINS broadcast. For updates, things are a bit more tricky since the metadata/version file's permissions don't allow for reading by a third party app. We work around this by launching a root shell to get the file contents with cat, so don't be alarmed if you get a 'Grant root?' popup by SuperSU or its brethren. Hashing and signing are pretty straightforward, but creating the new pins file merits some explanation.

To make it easier to test, we create (or append to) the pins file by connecting to the URL specified in the app and pinning the public keys in the host's certificate chain (we'll use www.google.com in this example, but any host accessible over HTTPS should do). Note that we don't actually pin the host's SSL certificate: this is to allow for the case where the host key is lost or compromised and a new certificate is issued to the host. This is introduced in the PKPE draft as a necessary security trade-off to allow for host certificate updates. Also note that in the case of one (or more) intermediate CA certificates we pin both the issuing certificate's key(s) and the root certificate's key. This is to allow for testing more variations, but is not something you might want to do in practice: for a connection to be considered valid, only one of the keys in the pin entry needs to be in the host's certificate chain. In the case that this is the root certificate's key, connections to hosts with certificates issued by a compromised intermediary CA will be allowed (think hacked root CA reseller). And above all, getting and creating pins based on certificates you receive from a host on the Internet is obviously pointless if you are already the target of a MITM attack. For the purposes of this test, we assume that this is not the case. Once we have all the data, we fire the update intent, and if it checks out the pins file will be updated (watch the logcat output to confirm). The code for this will look something like this (largely based on pinning unit test code in AOSP). With that, it is time to test if pinning actually works.

URL url = new URL("https://www.google.com");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.connect();

X509Certificate[] chain = (X509Certificate[])conn.getServerCertificates();
X509Certificate cert = chain[1];
String pinEntry = String.format("%s=true|%s", url.getHost(), getFingerprint(cert));
String contentPath = makeTemporaryContentFile(pinEntry);
String version = getNextVersion("/data/misc/keychain/metadata/version");
String currentHash = getHash("/data/misc/keychain/pins");
String signature = createSignature(content, version, currentHash);

Intent i = new Intent();
i.setAction("android.intent.action.UPDATE_PINS");
i.putExtra("CONTENT_PATH", contentPath);
i.putExtra("VERSION", version);
i.putExtra(REQUIRED_HASH", currentHash);
i.putExtra("SIGNATURE", signature);
sendBroadcast(i);

We have now pinned www.google.com, but how to test if the connection will actually fail? There are multiple ways to do this, but to make things a bit more realistic we will launch a MITM attack of sorts by using an SSL proxy. We will use the Burp proxy, which works by generating a new temporary (ephemeral) certificate on the fly for each host you connect to (if you prefer a terminal-based solution, try mitmproxy). If you install Burp's root certificate in Android's trust store and are not using pinning, browsers and other HTTP clients have no way of distinguishing the ephemeral certificate Burp generates from the real one and will happily allow the connection. This allows Burp to decrypt the secure channel on the fly and enables you to view and manipulate traffic as you wish (strictly for research purposes, of course). Refer to the Getting Started page for help with setting up Burp. Once we have Burp all set up, we need to configure Android to use it. While Android does support HTTP proxies, those are generally only used by the built-in browser and it is not guaranteed that HTTP libraries will use the proxy settings as well. Since Android is after all Linux, we can easily take care of this by setting up a 'transparent' proxy that redirects all HTTP traffic to our chosen host by using iptables. If you are not comfortable with iptables syntax or simply prefer an easy to use GUI, there's an app for that as well: Proxy Droid. After setting up Proxy Droid to forward packets to our Burp instance we should have all Android traffic flowing through our proxy. Open a couple of pages in the browser to confirm before proceeding further (make sure Burp's 'Intercept' button is off if traffic seems stuck).

Finally time to connect! The sample app allows you to test connection with both of Android's HTTP libraries (HttpURLConnection and Apache's HttpClient), just press the corresponding 'Check w/ ...' button. Since validation is done at the TLS layer, the connection shouldn't be allowed and you should see something like this (the error message may say 'No peer certificates' for HttpClient; this is due to the way it handles validation errors):

If you instead see a message starting with 'X509TrustManagerExtensions verify result: Error verifying chain...', the connection did go through but our additional validation using the X509TrustManagerExtensions class detected the changed certificate and failed. This shouldn't happen, right? It does though because HTTP clients cache connections (SSLSocket instances, which in turn each hold a X509TrustManager instance, which only reads pins when created). The easiest way to make sure pins are picked up is to reboot the phone after you pin your test host. If you try connecting with the Android browser after rebooting (not Chrome!), you will be greeted with this message:

As you can see the certificate for www.google.com is issued by our Burp CA, but it might as well be from DigiNotar: if the proper public keys are pinned, Android should detected the fraudulent host certificate and show a warning. This works because the Android browser is using the system trust store and pins via the default TrustManager, even though it doesn't use JSSE SSL sockets. Connecting with Chrome on the other hand works fine even though it does have built-in pins for Google sites: Chrome allows manually installed trust anchors to override system pins so that tools such as Burp or Fiddler continue to work (or pinning is not yet enabled on Android, which is somewhat unlikely).

So there you have it: pinning on Android works. If you look at the sample code, you will see that we have created enforcing pins and that is why we get connection errors when connecting through the proxy. If you set the enforcing parameter to false instead, connection will be allowed, but chains that failed validation will still be recorded to the system dropbox (/data/system/dropbox) in cert_pin_failure@timestamp.txt files, one for each validation failure.

Summary

Android adds certificate pinning by keeping a pin list with an entry for each pinned DNS name. Pin entries include a host name, an enforcing parameter and a list of SPKI SHA512 hashes of the of keys that are allowed to sign a certificate for that host. The pin list is updated by sending a broadcast with signed update data. Applications using the default HTTP libraries get the benefit of system-level pinning automatically or can explicitly check a certificate chain against the pin list by using the X509TrustManagerExtensions SDK class. Currently the pin list is empty, but the functionality is available now and once pins for major sites are deployed this will add another layer of defense against MIMT attacks that follow after a CA has been compromised.