Multi-Screen iOS Apps with PhoneGap

Posted: January 12th, 2012 | Author: Andrew | Filed under: Adobe, Apps, Development, HTML5, iOS, JavaScript, Mobile, open source, PhoneGap | Tags: adobe, development, HTML5, ios, JavaScript, mobile, open source, phonegap | 25 Comments »

Did you know that apps built on top of iOS can have a multi-screen workflow? For example in Keynote, you can have an external screen show a presentation while you control it on your iOS device. In the Jimi Hendrix app, you can view the audio player on an external screen, and in Real Racing HD, you can view the game on an external screen while the iOS device becomes your controller. (among others)

This is all made possible by the UIWindow and UIScreen APIs in iOS. Even better, on the iPad 2 and iPhone 4Gs, this can be done wirelessly using Airplay with an Apple TV device. On other iOS devices, you can have a second screen using a VGA output.

One of the benefits of using a cross platform solution like PhoneGap or Flex/Air is that you can build apps with an easier to use/more familiar paradigm.  However, cross platform runtimes don’t always offer access to every API feature that native development enables.

Out of the box, PhoneGap apps are confined to a single screen.  You can use screen mirroring to mirror content on an external screen, but you can’t have a second screen experience.  It’s a good thing you can write native plugins/extensions to enable native functionality within your applications.

ExternalScreen Native Plugin For PhoneGap

I recently did exactly that… I created a PhoneGap native plugin that enables second screen capability for PhoneGap applications.   The plugin listens for external screen connection notifications, and if an additional screen is available, it creates a new UIWebView for HTML-based content in the external screen – complete with functions for injecting HTML, JavaScript, or URL locations.

Why?

You might be wondering “Why?” you would want this plugin within PhoneGap…  this plugin enables the multi-screen experiences described in the apps mentioned above.  They extend the interactions and capabilities of the mobile hardware.   With this PhoneGap native plugin, you can create rich multi-screen experiences with the ease of HTML and JavaScript.   Here are a few ideas of the types of apps that you can build with this approach (scroll down for source code):

Fleet Manager

Let’s first consider a simple Fleet Manager application which allows you monitor vehicles in a mobile app.  This is a similar concept which I’ve used in previous examples.   The basic functionality allows you to see information on the tablet regarding your fleet.   What if this app connected to a larger screen and was able to display information about your vehicles for everyone to see?   Watch the video below to see this in real life.

This application example is powered by Google Maps, and all of the data is randomly generated on the client.

Law Enforcement

Let’s next consider a mobile law enforcement application application which gives you details to aid in investigations and apprehension of criminals.  Let’s pretend that you are a detective who is searching for a fugitive, and you walk into a crowded bar near the last known location of that fugitive.  You connect to the bar’s Apple TV on their big screen TV, pull up images and videos of the suspect, then say “Have you seen this person?”.   This could be incredibly powerful.  Check out the video below to see a prototype in real life.

This law enforcement demo scenario is a basic application powered by the FBI’s most wanted RSS data feeds.

Tip Of The Iceberg

There are lots of use cases where a second screen experience could be beneficial and create a superior product or application.   Using PhoneGap allows you to build those apps faster & with the ease of HTML and JavaScript, using traditional web development paradigms.

How It Works

Now, let’s review what makes this all work…   The client interfaces for both of these samples are written in HTML & JavaScript, and utilize jQuery, iScroll, and Modernizr, with a trick for removing link click delay on iOS devices.

The PhoneGap native plugin is written in Objective C, with a JavaScript interface to integrate with the client application. PhoneGap plugins are actually very easy to develop.  Basically, you have to write the native code class, write a corresponding JS interface, and add a mapping in your PhoneGap.plist file to expose the new functionality through PhoneGap.  There is a great reference on the PhoneGap wiki for native plugins which includes architecture & structure, as well as platform specific authoring and installation of those plugins.    Here are quick links to the iOS-specific native plugin content authoring and installation.

The ExternalScreen plugin creates a UIWebView for the the external screen, and exposes methods for interacting with the UIWebView.   Note: This is just a normal UIWebView, it does not have support for all PhoneGap libraries… just a standard HTML container.

You can read up on multi-screen programming at iOS from these useful tutorials:

• iOS View Programming Guide
• iPad VGA Output

Now let’s first examine the native code:

PGExternalScreen.h

The header file shows the method signatures for the native functionality.  The corresponding PGExternalScreen.m contains all of the actual code to make it all work.   Note: If you are using ARC (Automatic Reference Counting), you will need to remove the retain/release calls in PGExternalScreen.m.

@interface PGExternalScreen : PGPlugin {

    NSString* callbackID;
    UIWindow* externalWindow;
    UIScreen* externalScreen;
    UIWebView* webView;
    NSString* baseURLAddress;
    NSURL* baseURL;
}

@property (nonatomic, copy) NSString* callbackID;

//Public Instance Methods (visible in phonegap API)
- (void) setupScreenConnectionNotificationHandlers:(NSMutableArray*)arguments withDict:(NSMutableDictionary*)options ;
- (void) loadHTMLResource:(NSMutableArray*)arguments withDict:(NSMutableDictionary*)options;
- (void) loadHTML:(NSMutableArray*)arguments withDict:(NSMutableDictionary*)options;
- (void) invokeJavaScript:(NSMutableArray*)arguments withDict:(NSMutableDictionary*)options;
- (void) checkExternalScreenAvailable:(NSMutableArray*)arguments withDict:(NSMutableDictionary*)options;

//Instance Methods
- (void) attemptSecondScreenView;
- (void) handleScreenConnectNotification:(NSNotification*)aNotification;
- (void) handleScreenDisconnectNotification:(NSNotification*)aNotification;
@end

PGExternalScreen.js

The PGExternalScreen.js file defines the native methods that are exposed through PhoneGap.   You invoke the function, and can add success/fail callback function references.

var PGExternalScreen = {

    setupScreenConnectionNotificationHandlers: function (success, fail) {
        return PhoneGap.exec(success, fail, "PGExternalScreen", "setupScreenConnectionNotificationHandlers", []);
    },

    loadHTMLResource: function (url, success, fail) {
        return PhoneGap.exec(success, fail, "PGExternalScreen", "loadHTMLResource", [url]);
    },

    loadHTML: function (html, success, fail) {
        return PhoneGap.exec(success, fail, "PGExternalScreen", "loadHTML", 1);
    },

    invokeJavaScript: function (scriptString, success, fail) {
        return PhoneGap.exec(success, fail, "PGExternalScreen", "invokeJavaScript", [scriptString]);
    },

    checkExternalScreenAvailable: function (success, fail) {
        return PhoneGap.exec(success, fail, "PGExternalScreen", "checkExternalScreenAvailable", []);
    }

};

The Client

You can call any of these functions from within your PhoneGap application’s JavaScript just by referencing the exposed method on the PGExternalScreen instance.

// check if an external screen is available
PGExternalScreen.checkExternalScreenAvailable( resultHandler, errorHandler );

//load a local HTML resource
PGExternalScreen.loadHTMLResource( 'secondary.html', resultHandler, errorHandler );

//load a remote HTML resource (requires the URL to be white-listed in PhoneGap)
PGExternalScreen.loadHTMLResource( 'http://www.tricedesigns.com', resultHandler, errorHandler );

//load a HTML string
PGExternalScreen.loadHTML('</pre>
<h1>HTML</h1>
<pre>this is html content', resultHandler, errorHandler );

//invoke a JavaScript (passed as a string)
PGExternalScreen.invokeJavaScript('document.write(\'hello world\')', resultHandler, errorHandler );

The full code for the ExternalScreen PhoneGap native plugin, as well as both client applications and a basic usage example is available on github at:

• https://github.com/triceam/phonegap-plugins/tree/master/iPhone/ExternalScreen

Be sure to read the README for additional setup information.

(Update: source code link changed)

---