Single sign-on to Google sites using AccountManager

In the first part of this series, we presented how the standard Android online account management framework works and explored how Google account authentication and authorization modules are implemented on Android. In this article we will see how to use the Google credentials stored on the device to log in to Google Web sites automatically. Note that this is different from using public Google API's, which generally only requires putting an authentication token (and possibly an API key) in a request header, and is quite well supported by the Google APIs Client Library. First, some words on what motivated this whole exercise (may include some ranting, feel free to skip to the next section).

Android developer console API: DIY

If you have ever published an application on the Android Market Google Play Store, you are familiar with the Android developer console. Besides letting you publish and update your apps, it also shows the number of total and active installs (notoriously broken and not too be taken too seriously, though it's been getting better lately), ratings and comments. Depending on how excited about the whole app publishing business you are, you might want to check it quite often to see how your app is doing, or maybe you just like hitting F5. Most people don't however, so pretty much every developer at some point comes up with the heretic idea that there must be a better way: you should be able to check your app's statistics on your Android device (obviously!), you should get notified about changes automatically and maybe even be able to easily see if today's numbers are better than yesterday's at a glance. Writing such a tool should be fairly easy, so you start looking for an API. If your search ends up empty it's not your search engine's fault: there is none! So before you start scraping those pretty Web pages with your favourite P-language, you check if someone has done this before -- you might get a few hits, and if you are lucky even find the Android app.

Originally developed by Timelappse, and now open source, Andlytics does all the things mentioned above, and more (and if you need yet another feature, consider contributing). So how does it manage to do all of this without an API? Through blood, sweat and a lot of protocol reversing guessing. You see, the current developer console is built on GWT which used to be Google's webstack-du-jour a few years back. GWT essentially consists of RPC endpoints at the server, called by a JavaScript client running in the browser. The serialization protocol in between is a custom one, and the specification is purposefully not publicly available (apparently, to allow for easier changes!?!). It has two main features: you need to know exactly how the transferred objects look like to be able to make any sense of it, and it was obviously designed by someone who used to write compilers for a living before they got into Web development ('string table' ring a bell?). Given the above, Andlytics was quite an accomplishment. Additionally, the developer console changing its protocol every other week and adding new features from time to time didn't really make it any easier to maintain. Eventually, the original developer had a bit too much GWT on his plate, and was kind enough to open source it, so others could share the pain.

But there is a bright side to all this: Developer Console v2. It was announced at this year's Google I/O to much applause, but was only made universally available a couple of weeks ago (sound familiar?). It is a work in progress, but is showing promise. And the best part: it uses perfectly readable (if a bit heavy on null's) JSON to transport data! Naturally, there was much rejoicing at the Andlytics Github project. It was unanimously decided that the sooner we obliterate all traces of GWT, the better, and the next version should use the v2 console 'API'. Deciphering the protocol didn't take long, but it turned out that while to log in to the v1 console all you needed was a ClientLogin (see the next section for an explanation) token straight out of Android's AccountManger, the new one was not so forgiving and the login flow was somewhat more complex. Asking the user for their password and using it to login was obviously doable, but no one would like that, so we needed to figure out how to log in using the Google credentials already cached on the device. Android browser and Chrome are able to automatically log you in to the developer console without requiring your password, so it was clearly possible. The process is not really documented though, and that prompted this (maybe a bit too wide-cast) investigation. Which finally leads us to the topic of this post: to show how to use cached Google account credentials for single sign-on. Let's first see what standard ways are available to authenticate to Google's public services and API's.

Google services authentication and authorization

The official place to start when selecting an auth mechanism is the Google Accounts Authentication and Authorization page. It lists quite a few protocols, some open and some proprietary. If you research further you will find that currently all but OAuth 2.0 and Open ID are considered deprecated, and using the proprietary ones is not recommended. However, a lot of services are still using older, proprietary protocols, so we will look into some of those as well. Most protocols also have two variations: one for Web applications and one for the so called, 'installed applications'. Web applications run in a browser, and are expected to be able to take advantage of all standard browser features: rich UI, free-form user interaction, cookie store and ability to follow redirects. Installed applications, on the other hand, don't have a native way to preserve session information, and may not have the full Web capabilities of a browser. Android native applications (mostly) fall in the 'installed applications' category, so let's see what protocols are available for them.

ClientLogin

The oldest and most widely used till now authorization protocol for installed applications is ClientLogin. It assumes the application has access to the user's account name and password and lets you get an authorization token for a particular service, that can be saved and used for accessing that service on behalf of the user. Services are identified by proprietary service names, for example 'cl' for Google Calendar and 'ah' for Google App engine. A (non-exhaustive) list of supported service names can be found in the Google Data API reference. Here are a few Android-specific ones, not listed in the reference: 'ac2dm', 'android', 'androidsecure', 'androiddeveloper', 'androidmarket' and 'youngandroid' (probably for the discontinued App Inventor). The token can be fairly long-lived (up to two weeks), but cannot be refreshed and the application needs to obtain a new token when it expires. Additionally, there is no way to validate the token short of accessing the associated service: if you get an OK HTTP status (200), it is still valid, if 403 is returned you need to consult the additional error code and retry or get a new token. Another limitation is that ClientLogin tokens don't offer fine grained access to a service's resources: access is all or nothing, you cannot specify read-only access or access to a particular resource only. The biggest drawback for use in mobile apps though is that ClientLogin requires access to the actual user password. Therefore, if you don't want to force users to enter it each time a new token is required, it needs to be saved on the device, which poses various problems. As we saw in the previous post, in Android this is handled by GLS and the associated online service by storing an encrypted password or a master token on the device. Getting a token is as simple as calling the appropriate AccountManger method, which either returns a cached token or issues an API request to fetch a fresh one. Despite it's many limitations, the protocol is easy to understand and straightforward to implement, so it has been widely used. It has been officially deprecated since April 2012 though, and apps using it are encouraged to migrate to OAuth 2.0, but this hasn't quite happened yet. 

OAuth 2.0

No one likes OAuth 1.0 (except Twitter) and AuthSub is not quite suited for native applications, so we will only look at the currently recommended OAuth 2.0 protocol. OAuth 2.0 has been in the works for quite some time, but it only recently became an official Internet standard. It defines different authorization 'flows', aimed at different use cases, but we will not try to present all of them here. If you are unfamiliar with the protocol, refer to one of the multiple posts that aim to explain it at a higher level, or just read the RFC if you need the details.  And, of course, you can watch for this for a slightly different point of view. We will only discuss how OAuth 2.0 relates to native mobile applications.

The OAuth 2.0 specification defines 4 basic flows for getting an authorization token for a resource, and the two ones that don't require the client (in our scenario an Android app) to directly handle user credentials (Google account user name and password), namely the authorization code grant flow and the implicit grant flow, both have a common step that needs user interaction. They both require the authorization server (Google's) to authenticate the resource owner (the user of the our Android app) and establish whether they grant or deny the access request for the specified scope (e.g., read-only access to profile information). In a typical Web application that runs in a browser, this is very straightforward to do: the user is redirected to an authentication page, then to a access grant page that basically says 'Do you allow app X to access data Y and Z?', and if they agree, another redirect, which includes an authorization token, takes them back to the original application. The browser simply needs to pass on the token in the next request to gain access to the target resource. Here's an official Google example that uses the implicit flow: follow this link and grant access as requested to let the demo Web app display your Google profile information. With a native app things are not that simple. It can either

• use the system browser to handle the permission grant step, which would typically involve the following steps:
• launch the system browser and hope that the user will finish the authentication and permission grant process
• detect success or failure and extract the authorization token from the browser on success (from the window title, redirect URL or the cookie store)
• ensure that after granting access, the user ends up back in your app
• finally, save the token locally and use it to issue the intended Web API request
• embed a WebView or a similar control in the apps's UI. Getting a token would generally involve these steps:
• in the app's UI, instruct the user what to do and load the login/authorization page
• register for a 'page loaded' callback, and check for the final success URL each time it's called
• when found, extract the token from the redirect URL or the WebView's cookie jar and save it locally
• finally use the token to send the intended API request

Neither is ideal, both are confusing to the user and to implement the first one on Android you might event have to (temporarily) start a Web server (redirect_uri is set to http://localhost in the API console, so you can't just use a custom scheme). The second one is generally preferable, if not pretty: here's an (somewhat outdated) overview of what needs to be done and a more recent example with full source code. This integration complexity and UI impedance mismatch are the problems that OAuth 2.0 support via the AccountManager initially, and recently Google Play Services aim to solve. When using either of those, user authentication is implemented transparently by passing the saved master token (or encrypted password) to the server side component, and instead of a WebView with a permission grant page, you get the Android native access grant dialog. If you approve, a second request is sent to convey this and the returned access token is directly delivered to the requesting app. This is essentially the same flow as for Web applications, but has the advantages that it doesn't require context switching from native to browser and back, and is much more user friendly. Of course, it only works for Google accounts, so if you wanted to write, say, a Facebook client, you still have to use a WebView to process the access permission grant and get an authorization token.

Now that we have an idea what authentication methods are available, let's see if we can use them to access an online Google service that doesn't have a dedicated API.

Google Web properties single sign-on

Being able to access multiple related, but separate services without needing to authenticate to each one individually is generally referred to as single sign-on (SSO). There are multiple standard ways to accomplish this for different contexts, ranging from Kerberos to SAML-based solutions. We will use the term here in a narrower meaning: being able to use different Google services (Web sites or API's) after having authenticated to only one of them (including the Android login service). If you have a fairly fast Internet connection, you might not even notice it, but after you log in to, say, Gmail, clicking on YouTube links will take you to a completely different domain, and yet you will be able to comment on that neat cat video without having to log in again. If you have a somewhat slower connection and a wide display though, you may notice that there is a lot of redirecting and long parameter passing, with the occasional progress bar going on. What happens behind the scenes is that your current session cookies and authentication tokens are being exchanged for yet other tokens and more cookies, to let you seamlessly log in to that other site. If you are curious, you can observe the flow with Chrome's built-in developer tools (or similar plugins for other browsers), or check out our sample. All of those requests and responses are essentially a proprietary SSO protocol (Google's), which is not really publicly documented anywhere, and, of course, is likely to change fairly often as Google rolls out upgrades to their services. With that said, there is a distinct pattern, and on a higher level you only have two main cases. We are deliberately ignoring the persistent cookie ('Stay signed in')  scenario for simplicity's sake.

• Case 1: you haven't authenticated to any of the Google properties. If you access, for example, mail.google.com in that state you will get a login screen originating at https://accounts.google.com/ServiceLogin with parameters specifying the service you are trying to access ('mail' for Gmail) and where to send you after you are authenticated. After you enter your credentials, you will generally get redirected a few times around the accounts.google.com, which will set a few session cookies, common (Domain=.google.com) for all services (always SID and LSID, plus a few more). The last redirect will be to the originally requested service and include an authentication token in the redirected location (usually specified with the auth parameter, e.g.: https://mail.google.com/mail/?auth=DQAAA...). The target service will validate the token and set a few more service-specific sessions cookies, restricted by domain and path, and with the Secure and HttpOnly flags set. From there, it might take a couple of more redirects before you finally land at an actual content page.
• Case 2: you have already authenticated to at least one service (Gmail in our example). In this state, if you open, say, Calendar, you will go through https://accounts.google.com/ServiceLogin again, but this time the login screen won't be shown. The accounts service will modify your SID and LSID cookies, maybe set a few new ones and finally redirect you the original service, adding an authentication token to the redirect location. From there the process is similar: one or more service-specific cookies will be set and you will finally be redirected to the target content.

Those flows obviously work well for browser-based logins, but since we are trying to do this from an Android app, without requiring user credentials or showing WebView's, we have a different scenario. We can easily get a ClientLogin or an OAuth 2.0 token from the AccountManager, but since we are not preforming an actual Web login, we have no cookies to present. The question becomes: is there a way to log in with a standard token alone? Since tokens can be used with the data APIs (where available) of each service, they obviously contain enough information to authenticate us and grant access to the service's resources. What we need is an Web endpoint, that will take our token and give us a set of cookies we could use to access the corresponding Web site in exchange. Clues and traces of such a service are scattered around the Internet, mostly in the code of unofficial Google client libraries and applications. Once we know it is definitely possible, the next problem becomes getting it to work with Android's AccountManger.

Logging in using AccountManager

The only real documentation we could find, besides code comments and READMEs of the unofficial Google client applications mentioned above, is a short Chromium OS design document. It tells us that the standard (at the time) login API for installed applications, ClientLogin, alone is not enough to accomplish Web SSO, and outlines a three step process that lets us exchange ClientLogin tokens for session cookies valid for a particular service:

1. Get a ClientLogin token (this we can do via the AccountManager)
2. Pass it to https://www.google.com/accounts/IssueAuthToken, to get a one-time use, short-lived token that will authenticate the user to any service (the so called, 'ubertoken')
3. Finally, pass the ubertoken to https://www.google.com/accounts/TokenAuth, to exchange it for the full set of browser cookies we need to do SSO

This outlines the process, but is a little light on the details. Fortunately, those can be found in the Chromium OS source code, as well as a few other projects. After a fair bit of digging, here's what we uncovered:

1. To get the mythical ubertoken, you need to pass the SID and LSID cookies to the IssueAuthToken endpoint like this:
   

   https://www.google.com/accounts/IssueAuthToken?service=gaia&Session=false&SID=sid&LSID=lsid
   

2. The response will give you the ubertoken, which you pass to the TokenAuth endpoint along with the URL of the service you want to use:
   

   https://www.google.com/accounts/TokenAuth?source=myapp&auth=ubertoken&continue=service-URL
   

3. If the token check out OK, the response will give you a URL to load. If your HTTP client is set up to follow redirects automatically, once you load it, needed cookies will be set automatically (just as in a browser), and you will finally land on the target site. As long as you keep the same session (which usually means the same HTTP client instance) you will be able to issue multiple requests, without needing to go through the authentication flow again.

What remains to be seen is, can we implement this on Android. As usual, it turns out that there is more than one way to do it:

The hard way

The straightforward way would be to simply implement the flow outlined above using your favourite HTTP client library. We choose to use Apache HttpClient, which supports session cookies and multiple requests using a single instance out of the box. The first step calls for the SID and LSID cookies though, not an authentication token: we need cookies to get a token, in order to get more cookies. Since Android's AccountManager can only give us authentication tokens, and not cookies, this might seem like a hopeless catch-22 situation. However, while browsing the authtokens table of the system's accounts database earlier, we happened to notice that it actually had a bunch of tokens with type SID and LSID. Our next step is, of course, to try to request those tokens via the AccountManager interface, and this happens to work as expected:

String sid = am.getAuthToken(account, "SID", null, activity, null, null)
    .getResult().getString(AccountManager.KEY_AUTHTOKEN);
String lsid = am.getAuthToken(account, "LSID", null, activity, null, null)
    .getResult().getString(AccountManager.KEY_AUTHTOKEN);

Having gotten those, the rest is just a matter of issuing two HTTP requests (error handling omitted for brevity):

String TARGET_URL = "https://play.google.com/apps/publish/v2/";
Uri ISSUE_AUTH_TOKEN_URL = 
 Uri.parse("https://www.google.com/accounts/IssueAuthToken?service=gaia&Session=false");
Uri TOKEN_AUTH_URL = Uri.parse("https://www.google.com/accounts/TokenAuth");

String url = ISSUE_AUTH_TOKEN_URL.buildUpon().appendQueryParameter("SID", sid)
     .appendQueryParameter("LSID", lsid)
     .build().toString();
HttpPost getUberToken = new HttpPost(url);
HttpResponse response = httpClient.execute(getUberToken);
String uberToken = EntityUtils.toString(entity, "UTF-8");
String getCookiesUrl = TOKEN_AUTH_URL.buildUpon()
     .appendQueryParameter("source", "android-browser")
     .appendQueryParameter("auth", authToken)
     .appendQueryParameter("continue", TARGET_URL)
     .build().toString();
HttpGet getCookies = new HttpGet(getCookiesUrl);
response = httpClient.execute(getCookies);

CookieStore cookieStore = httpClient.getCookieStore();
// check for service-specific session cookie
String adCookie = findCookie(cookieStore.getCookies(), "AD");
// fail if not found, otherwise get page content
String responseStr = EntityUtils.toString(entity, "UTF-8");

This lets us authenticate to the Android Developer Console (version 2) site without requiring user credentials and we can easily proceed to parse the result and use it in a native app (warning: work in progress!) from here. The downside is that for this to work, the user has to grant access twice, for two cryptically looking token types (SID and LSID).

Of course, after writing all of this, it turns out that the stock Android browser already has code that does it, which we could have used or at least referenced from the very beginning. Better yet, this find leads us to an yet easier way to accomplish our task. 

The easy way

The easy way is found right next to the Browser class referenced above, in the DeviceAccountLogin class, so we can't really take any credit for this. It is hardly anything new, but some Googling suggests that it is neither widely known nor used much. You might have noticed that the Android browser is able to silently log you in to Gmail and friends, when you use the mobile site. The way this is implemented is via the 'magic' token type 'weblogin:'. If you use it along with the service name and URL of the site you want to access, it will do all of the steps listed above automatically and instead of a token will give you a full URL you can load to get automatically logged in to your target service. This magic URL is in the format shown below, and includes both the ubertoken and the URL of the target site, as well as the service name (this example is for the Android Developer Console, line is broken for readability):

https://accounts.google.com/MergeSession?args=service%3Dandroiddeveloper%26continue
%3Dhttps://play.google.com/apps/publish/v2/&uberauth=APh...&source=AndroidWebLogin

Here's how to get the MergeSession URL:

String tokenType = "weblogin:service=androiddeveloper&"
+ "continue=https://play.google.com/apps/publish/v2/";
String loginUrl = accountManager.getAuthToken(account,tokenType, false, null, null)
                   .getResult().getString(AccountManager.KEY_AUTHTOKEN);

This is again for the Developer Console, but works for any Google site, including Gmail, Calendar and even the account management page. The only problem you might have is finding the service name, which is hardly obvious in some cases (e.g., 'grandcentral' for Google Voice and 'lh2' for Picasa).

It takes only a single HTTP request form Android to get the final URL, which tells us that the token issuing flow is implemented on the server side. This means that you can also use the Google Play Services client library to issue a weblogin: 'token' (see screenshot below and note that unlike for OAuth 2.0 scopes, it shows the 'raw' token type). Probably goes without saying, but it also means that if you happen to come across someone's accounts.db file, all it takes to log in into their Google account(s) is two HTTPS requests: one to get the MergeSession URL, and one to log in to their accounts page. If you are thinking 'This doesn't affect me, I use Google two-factor authentication (2FA)!', you should know that in this case 2FA doesn't really help. Why? Because since Android doesn't support 2FA, to register an account with the AccountManager you need to use an application specific password (Update: On ICS and later, GLS will actually show a WebView and let you authenticate using your password and OTP. However, the OTP is not required once you get the master token). And once you have entered one, any tokens issued based on it, will just work (until you revoke it), without requiring entering an additional code. So if you value your account, keep your master tokens close and revoke them as soon as you suspect that your phone might be lost or stolen. Better yet, consider a solution that lets you wipe it remotely (which might not work after your revoke the tokens, so be sure to check how it works before you actually need it).

As we mentioned above, this is all ClientLogin based, which is officially deprecated, and might be going away soon (EOL scheduled for April 2013). But some of the Android Google data sync feeds still depend on ClientLogin, so if you use it you would probably OK for a while. Additionally, since the weblogin: implementation is server-based, it might be updated to conform with the latest (OAuth 2.0-based?) infrastructure without changing the client-side interface. In any case, watch the Android Browser and Chormium code to keep up to date.

Summary

Google offers multiple online services, some with both a traditional browser-based interface and a developer-oriented API. Consequently, there are multiple ways to authenticate to those, ranging from form-based username and password login to authentication API's such as ClientLogin and OAuth 2.0. It is relatively straightforward to get an authentication token for services with a public API on Android, either using Android's native AccountManager interface or the newer Google Play Services extension. Getting the required session cookies to login automatically to the Web sites of services that do not offer an API is however neither obvious, nor documented. Fortunately, it is possible and very easy to do if you combine the special 'weblogin:' token type with the service name and the URL of the site you want to use. The best available documentation about this is the Android Browser source code, which uses the same techniques to automatically log you in to Google sites using the account(s) already registered on your device.

Moral of the story: interoperability is so much easier when you control all parties involved.