Multi-Screen iOS Apps with PhoneGap

Did you know that apps built on top of iOS can have a multi-screen workflow? For example in Keynote, you can have an external screen show a presentation while you control it on your iOS device. In the Jimi Hendrix app, you can view the audio player on an external screen, and in Real Racing HD, you can view the game on an external screen while the iOS device becomes your controller. (among others)

Real Racing HD

This is all made possible by the UIWindow and UIScreen APIs in iOS. Even better, on the iPad 2 and iPhone 4Gs, this can be done wirelessly using Airplay with an Apple TV device. On other iOS devices, you can have a second screen using a VGA output.

One of the benefits of using a cross platform solution like PhoneGap or Flex/Air is that you can build apps with an easier to use/more familiar paradigm.  However, cross platform runtimes don’t always offer access to every API feature that native development enables.

Out of the box, PhoneGap apps are confined to a single screen.  You can use screen mirroring to mirror content on an external screen, but you can’t have a second screen experience.  It’s a good thing you can write native plugins/extensions to enable native functionality within your applications.

ExternalScreen Native Plugin For PhoneGap

I recently did exactly that… I created a PhoneGap native plugin that enables second screen capability for PhoneGap applications.   The plugin listens for external screen connection notifications, and if an additional screen is available, it creates a new UIWebView for HTML-based content in the external screen – complete with functions for injecting HTML, JavaScript, or URL locations.

Why?

You might be wondering “Why?” you would want this plugin within PhoneGap…  this plugin enables the multi-screen experiences described in the apps mentioned above.  They extend the interactions and capabilities of the mobile hardware.   With this PhoneGap native plugin, you can create rich multi-screen experiences with the ease of HTML and JavaScript.   Here are a few ideas of the types of apps that you can build with this approach (scroll down for source code):

Fleet Manager

Let’s first consider a simple Fleet Manager application which allows you monitor vehicles in a mobile app.  This is a similar concept which I’ve used in previous examples.   The basic functionality allows you to see information on the tablet regarding your fleet.   What if this app connected to a larger screen and was able to display information about your vehicles for everyone to see?   Watch the video below to see this in real life.

This application example is powered by Google Maps, and all of the data is randomly generated on the client.

Law Enforcement

Let’s next consider a mobile law enforcement application application which gives you details to aid in investigations and apprehension of criminals.  Let’s pretend that you are a detective who is searching for a fugitive, and you walk into a crowded bar near the last known location of that fugitive.  You connect to the bar’s Apple TV on their big screen TV, pull up images and videos of the suspect, then say “Have you seen this person?”.   This could be incredibly powerful.  Check out the video below to see a prototype in real life.

This law enforcement demo scenario is a basic application powered by the FBI’s most wanted RSS data feeds.

Tip Of The Iceberg

There are lots of use cases where a second screen experience could be beneficial and create a superior product or application.   Using PhoneGap allows you to build those apps faster & with the ease of HTML and JavaScript, using traditional web development paradigms.

How It Works

Now, let’s review what makes this all work…   The client interfaces for both of these samples are written in HTML & JavaScript, and utilize jQuery, iScroll, and Modernizr, with a trick for removing link click delay on iOS devices.

The PhoneGap native plugin is written in Objective C, with a JavaScript interface to integrate with the client application. PhoneGap plugins are actually very easy to develop.  Basically, you have to write the native code class, write a corresponding JS interface, and add a mapping in your PhoneGap.plist file to expose the new functionality through PhoneGap.  There is a great reference on the PhoneGap wiki for native plugins which includes architecture & structure, as well as platform specific authoring and installation of those plugins.    Here are quick links to the iOS-specific native plugin content authoring and installation.

The ExternalScreen plugin creates a UIWebView for the the external screen, and exposes methods for interacting with the UIWebView.   Note: This is just a normal UIWebView, it does not have support for all PhoneGap libraries… just a standard HTML container.

You can read up on multi-screen programming at iOS from these useful tutorials:

Now let’s first examine the native code:

PGExternalScreen.h

The header file shows the method signatures for the native functionality.  The corresponding PGExternalScreen.m contains all of the actual code to make it all work.   Note: If you are using ARC (Automatic Reference Counting), you will need to remove the retain/release calls in PGExternalScreen.m.

@interface PGExternalScreen : PGPlugin {

    NSString* callbackID;
    UIWindow* externalWindow;
    UIScreen* externalScreen;
    UIWebView* webView;
    NSString* baseURLAddress;
    NSURL* baseURL;
}

@property (nonatomic, copy) NSString* callbackID;

//Public Instance Methods (visible in phonegap API)
- (void) setupScreenConnectionNotificationHandlers:(NSMutableArray*)arguments withDict:(NSMutableDictionary*)options ;
- (void) loadHTMLResource:(NSMutableArray*)arguments withDict:(NSMutableDictionary*)options;
- (void) loadHTML:(NSMutableArray*)arguments withDict:(NSMutableDictionary*)options;
- (void) invokeJavaScript:(NSMutableArray*)arguments withDict:(NSMutableDictionary*)options;
- (void) checkExternalScreenAvailable:(NSMutableArray*)arguments withDict:(NSMutableDictionary*)options;

//Instance Methods
- (void) attemptSecondScreenView;
- (void) handleScreenConnectNotification:(NSNotification*)aNotification;
- (void) handleScreenDisconnectNotification:(NSNotification*)aNotification;
@end

PGExternalScreen.js

The PGExternalScreen.js file defines the native methods that are exposed through PhoneGap.   You invoke the function, and can add success/fail callback function references.

var PGExternalScreen = {

    setupScreenConnectionNotificationHandlers: function (success, fail) {
        return PhoneGap.exec(success, fail, "PGExternalScreen", "setupScreenConnectionNotificationHandlers", []);
    },

    loadHTMLResource: function (url, success, fail) {
        return PhoneGap.exec(success, fail, "PGExternalScreen", "loadHTMLResource", [url]);
    },

    loadHTML: function (html, success, fail) {
        return PhoneGap.exec(success, fail, "PGExternalScreen", "loadHTML", 1);
    },

    invokeJavaScript: function (scriptString, success, fail) {
        return PhoneGap.exec(success, fail, "PGExternalScreen", "invokeJavaScript", [scriptString]);
    },

    checkExternalScreenAvailable: function (success, fail) {
        return PhoneGap.exec(success, fail, "PGExternalScreen", "checkExternalScreenAvailable", []);
    }

};

The Client

You can call any of these functions from within your PhoneGap application’s JavaScript just by referencing the exposed method on the PGExternalScreen instance.

// check if an external screen is available
PGExternalScreen.checkExternalScreenAvailable( resultHandler, errorHandler );

//load a local HTML resource
PGExternalScreen.loadHTMLResource( 'secondary.html', resultHandler, errorHandler );

//load a remote HTML resource (requires the URL to be white-listed in PhoneGap)
PGExternalScreen.loadHTMLResource( 'http://www.tricedesigns.com', resultHandler, errorHandler );

//load a HTML string
PGExternalScreen.loadHTML('</pre>
<h1>HTML</h1>
<pre>this is html content', resultHandler, errorHandler );

//invoke a JavaScript (passed as a string)
PGExternalScreen.invokeJavaScript('document.write(\'hello world\')', resultHandler, errorHandler );

The full code for the ExternalScreen PhoneGap native plugin, as well as both client applications and a basic usage example is available on github at:

Be sure to read the README for additional setup information.

(Update: source code link changed)

  • http://www.raymondcamden.com Raymond Camden

    Fraking epic!

  • http://www.tegdesign.com Tegan Snyder

    I literally can’t wait to get to work tomorrow and start playing with this. Excellent stuff!

  • Dawesi

    now all we need is android, bb, etc… ;-)

    • http://www.tricedesigns.com Andrew

      Do any of the other platforms support multiple screens? I’ve never seen a demo of it. I’ll have to look into it and see if its even possible.

  • Nicolas BUI

    Awesome :)

  • Pingback: Multi-Screen iOS Apps with PhoneGap | HTML5 Game Development

  • http://movl.com Alan

    I’m running your basic sample on iPhone4S, and an AppleTV on same network, but the UIScreen count is always 1. I can share photos to the tv tho. Is there anything I’m missing? maybe a setting somewhere?

  • Alan

    I sorted it out. Works great! So I’m having an issue with entering text once a second screen is in place. Have you run into this or found a solution?

    • http://www.tricedesigns.com Andrew

      What issue ar you seeing? You won’t be able to set input focus on the second screen.

    • Brent

      I’m seeing a similar issue. I’ve got two screens, #1 is on an iPad, #2 is on an external screen. If I init the second screen and add in an input field on screen #1 the keyboard does not appear. If I remove the second screen the keyboard works fine.

      • Brent

        Figured it out, in PGExternalScreen.m, – (void) attemptSecondScreenView, the second last line “[externalWindow makeKeyAndVisible];” can be commented out / removed. Since you can’t interact with the window, there’s no point in grabbing the interaction.

        • http://www.tricedesigns.com Andrew

          Awesome, thanks for sharing!

  • http://www.andrewthorp.com Andrew Thorp

    Working great in loading the content, but external assets (css, images, etc) do not seem to be appearing correctly.

    I am using the following:

    But the styles aren’t making it. If i copy them to inline styles they work, but the url’s to images do not work (assets/img/img-here.png).

    Any ideas?

  • http://www.andrewthorp.com/ Andrew Thorp

    Followup to the previous question, I am trying to load a ‘local’ html file.

    This is a HTML file that needs to have a background image and needs to be able to execute jQuery. Can I not use external assets in a ‘local’ html file that is loaded via loadHTMLResource();

    Thanks!

    • http://www.tricedesigns.com Andrew

      Hi Andrew, I think it might be a bug in the plugin… I’ll have to look into it further. This is an issue w/ the root for the UIWebView. I’ll look into it and get back to you.

    • http://www.tricedesigns.com Andrew

      It has been fixed, and the latest version is now in github at: https://github.com/triceam/phonegap-plugins/tree/master/iPhone/ExternalScreen I’ve initiated a “pull request” to have the external screen plugin updated in the main phonegap-plugins repo.

  • http://www.andrewthorp.com/ Andrew Thorp

    Andrew,

    I have been searching for it today and have come up with similar findings. baseURL might not be set correctly.

    I really like PhoneGap and love your plugin – and am glad that I won’t have to abandon it!

    I have a demo coming up very soon and am looking forward to showing off the abilities of these tools.

    Thanks again!
    Andrew

    p.s. Sorry for bugging you about it a couple of times. :)

  • http://www.andrewthorp.com/ Andrew Thorp

    Thanks! Works great!

    One question: Do I have any control over the resolution that it is displayed on the second screen at?

    Thanks for all of your help!
    Andrew

  • http://www.andrewthorp.com Andrew Thorp

    Gotcha – so would it be possible for me to tie into your plugin and/or to PhoneGap to accomplish this without having to write a native iPad application?

    Thanks!
    Andrew

    • http://www.tricedesigns.com Andrew

      You would have to modify the native code within the plugin to detect & change screen size. It’s not that complicated, and you wouldn’t have to write an entirely new native app from scratch.

      • Adam

        You would have to modify the native code within the plugin to detect & change screen size.

        Very good statement, +100500

        http://www.plagiarismdetect.com

  • http://www.tricedesigns.com Andrew

    The latest bug fix, which fixes relative paths of local content (due to an incorrect base url) has been merged in the main phonegap-plugins repository.

  • http://www.andrewthorp.com/ Andrew Thorp

    Hey,

    I have the VGA output working, but it is only allowing me to choose 720×480 when I connect it to my HDTV.

    If I don’t do anything with resolution picking, your webView is filling up the whole screen at 16:9. Any idea why availableModes is only returning a single resolution?

    (I realize you are not the person who developed that iPadVGAOutputTest, but you have just been so helpful!)

    Cheers!
    Andrew

  • http://www.andrewthorp.com/ Andrew Thorp

    Yeah, I do understand that it is returning what is supported by the screen, but it’s odd that if I don’t set it (just use your plugin code) – it seems to use the screens ‘native’ resolution, which is much larger than 720×480.

    It actually outputs at 16:9, and 720×480 is 3:2, so I know it CAN feed it out at 16:9.

    I’m connecting with VGA…

    Thanks!

  • Pingback: Andrew Trice » Blog Archive » PhoneGap Native Plugins

  • http://www.andrewthorp.com/ Andrew Thorp

    Hey Andrew,

    I have another question.

    The ‘handleScreenConnectNotification’ method. Wouldn’t it be more useful if that sent a notification back to the JavaScript that I could listen for?

    If the display is disconnected and reconnected mid application, I am not aware of it in the app itself. Am I supposed to test every second to see if there is a second screen? Or am I missing something?

    My code looks like this: https://gist.github.com/2023751

    When I first load the application, whether there is a second screen or not, I see: “External Screen Notification Handlers initialized.”

    However, if I plug my second screen in after that, I see a white screen with “loading…” on it, but the resultHandler is never fired again.

    Thanks!

    • http://www.tricedesigns.com Andrew

      You can add a call to [self writeJavascript: @"put javascript here"]; inside of both handleScreenConnectNotification and handleScreenDisconnectNotification, and it will do this for you. If I have time, I can add them later this week, but it should be pretty easy to add yourself to get started w/o waiting on me. The writeJavascript function will execute any JS string within the PhoneGap web view.

  • http://www.pixelagent.co.uk Nigel

    Is there an issue with the ‘makeKeyAndVisible’ when using this plugin? Once synced the keyboard on my ipad won’t load.

    • http://www.tricedesigns.com Andrew

      I’ll have to test it here and let you know… I haven’t noticed any errors before. I’ll double check, and see if I can find anything.

  • Blake

    Are you sure this supports AirPlay? I’ve been playing around with it but can’t seem to get AirPlay to connect. So far it only works over HDMI+VGA.

    • http://www.tricedesigns.com Andrew

      Yes it supports Airplay, but you need to be sure that the devices supports Airplay mirroring – This works on iPhone4S, iPad2+, but not on iPad 1 or iPhone 4 and earlier b/c those devices do not support Airplay mirroring.

  • grim

    Anyone give this a shot with a newer version of Phonegap? I had some issues with the name changes from PG to CDV in the plugins, and am successfully building in XCode, but things don’t seem to work. Awesome plugin – any target version of Phonegap you’d recommend?

    • http://www.tricedesigns.com Andrew

      Yes, this does work with newer versions of PhoneGap. All you need to do is update the class name references. You can use it on any version of PhoneGap after 1.3. Just make sure that the device you are using supports AirPlay mirroring. The iPhone 4 (and earlier), and iPad 1 do not support it.

  • http://www.creeostudio.it NoriSte

    Hi Andrew, do you know if something similar will be possible with AIR? I believe we can’t but do you know if is Adobe planning to support it? It could be very useful.
    Now I’ve to create a native app only on selected smarttvs… http://www.youtube.com/watch?v=0e8cmy1Vmic

    • http://www.tricedesigns.com Andrew

      I am not actively engaged with the AIR roadmap, so I can’t say with any certainty. However, I am not aware of any plans. You could create an ANE that does exactly what this PhoneGap native plugin does (or just port this one). However, it would be a web view, and not Flash/AIR content in the second screen.

  • Pingback: Andrew Trice » Blog Archive » Connected Second-Screen App Experiences with PhoneGap & Audio Watermarks

  • Stefan

    This is beyond epic. I was waiting for someone to create this plugin. Can’t wait to use this!

    • http://www.tricedesigns.com Andrew

      Thanks! Feel free to share whatever you create!

  • http://www.appelsiini.net/ Mika Tuupola

    Excellent stuff. I have been working with couple of similar projects. However these are HTML5 + JS applications and not PhoneGap. Content on bigger screens is controlled by iPad websockets. This is not screen mirroring although some views look almost the same. Here is video of the first version of the app (ignore the cheesy music).

    http://vimeo.com/36902847

    After reading this article I think I will give ExternalScreen plugin also a try. Possibility to use Apple TV for displaying content screen seems great to me.

    • http://www.tricedesigns.com Andrew

      Very cool. This plugin just gives you the ability to control both screens inside of the phonegap app’s web view (via JavaScript).

  • Kevin

    Hi Andrew, yesterday i want to test your external screen plugin, but i dont get it to work.
    I use Phonegap 2.5.0 and install your sample “basic usage” in my app.

    I use a iPhone5 with ATV3 and Reflector for Mac mirroring.

    Both devices are ready and i can use them to watch f.e. image from iphone on atv and mac.

    But when i test you example i get the error: SUCCESS: NO or External Webview Unavailable

    What do i wrong

    Greets from germany

    Kevin

    • http://www.appzer.de Kevin

      today i could test it with a vga cable, this works fine, but not over airplay (apple tv3 or reflector for mac)

      Kevin

      • http://www.tricedesigns.com Andrew

        Make sure that Airplay mirroring is enabled.

    • http://www.tricedesigns.com Andrew

      Make sure that you have screen mirroring turned “ON”. See details here: http://support.apple.com/kb/HT5209 IF you don’t have mirroring turned “ON”, it will not recognize the second screen, even though AirPlay is enabled. The really odd part is that you aren’t really mirroring your screen in this case.

  • peter

    great work! – just cannot get the initial viewport of the appletv to fullscreen without displaying a black border around the content of loadhtmlresource. after a video file has been played once, the viewport would stick to fullscreen. any ideas, how set the initial view to ‘true’ fullscreen?

  • Julian Montoya

    how i can config to show the video out the mode airview clasic?

    i have some problems whit the app return after y play some video.
    Thanks!
    /* excuse but…my english is basic level*/

  • octalmage

    Hey Andrew! I know you’re probably busy and have no interest in this plugin, but it would be amazing if you could update this and submit it to the PhoneGap:Build repository.

    https://build.phonegap.com/plugins

    The possibilities for this are insane, and myself and I’m sure the rest of the community would LOVE the change to play with it.