Back in the summer, I was lucky enough to get my hands on some early builds of Stage3D for mobile. I built some simple examples, including basic geometric shapes and simple 3D bubble charts inside of mobile Flex/AIR applications. I have been asked numerous times for the source code, and I’ve finally given in, and am sharing some source code.
I am not posting the full mobile application source code, since Stage3D for mobile is not yet available. However, I have ported the 3D bubble chart example to run in a Flex application targeting the desktop (Flash Player 11). The bubble chart example extends the concepts explored in the basic geometric shapes example.
Before you say “shoot, he didn’t give us the mobile code”, let me explain… When I ported the code from the mobile project to the desktop Flex project, all I changed was code specific to the mobile Flex framework. I changed <s:ViewNavigatorapplication> to <s:Application> and the corresponding architecture changes that were required, and I changed the list item renderers to Spark item renderers based on <s:Group> instead of mobile item renderers. In the mobile item renderers, all my drawing logic was done using the ActionScript drawing API. For simplicity in the port, I just used <s:Rect> to add the colored regions in the desktop variant.
That is all I changed!
The stage3D code between the desktop and mobile implementations is identical. You can see the desktop port in action in the video below:
The source code was intended to be exploratory at best… I was simply experimenting with hardware accelerated content, and how it can be used within your applications. There is one big “gotcha” that you will have to watch out for if you want Stage3D content within a Flex application… Stage3D content shows up behind Flex content on the display list. By default, Flex apps have a background color, and they will hide the Stage3D content. If you want to display any Stage3D content within a Flex application (regardless of web, desktop AIR, or mobile), you must set the background alpha of the Flex application to zero (0). Otherwise you will pull out some hair trying to figure out why it doesn’t show up.
The source code for the web/Flex port of this example is available at:
If you have ever tried to develop any kind of application using HTML5 audio that is widely supported, then you have likely pulled all the hair from your head. In its current state, HTML5 Audio is wrought with issues… lack of consistent codec support across browsers & operating systems, no polyphony (a single audio clip can not be played on top of itself), and lack of concurrency (on some of the leading mobile browsers you can only play one audio file at a time, if at all). Even the leading HTML5 games for desktop browsers don’t even use HTML5 audio (they use Flash). Don’t believe me? Just take a look at Angry Birds, Cut the Rope, or Bejeweled in a proxy/resource monitor…
The Problem
You want fast & responsive audio for your mobile applications. This is especially the case for multimedia intensive and/or gaming applications.
HTML5 audio is not *yet* ready for prime-time. There are some great libraries like SoundManager, which can help you try to use HTML5 audio with a failover to Flash, but you are still limited without polyphony or concurrency. In desktop browsers, Flash fixes these issues, and Flash is still vastly superior to HTML5 for audio programming.
If you are building mobile applications, you can have great audio capabilities by developing apps with AIR. However, what if you aren’t using AIR? In native applications, you can access the underlying audio APIs and have complete control.
If you are developing mobile applications with PhoneGap, you can use the Media class, which works great. If you want polyphony, then you will have to do some work managing audio files for yourself, which can get tricky. You can also write native plugins that integrate with the audio APIs for the native operating systems, which is what i will be covering in this post.
Before continuing further, let’s take a minute to understand what I am talking about when I refer to concurrency, polyphony, and low-latency…
Concurrency
Concurrency in audio programming refers to the ability to play multiple audio resources simultaneously. HTML5 in most mobile devices does not support this – not in iOS, not in Android. In fact, HTML5 Audio does not work *at all* in Android 2.x and earlier. Native APIs do support this, and so does PhoneGap’s Media class, which is based on Android MediaPlayer and iOS AVAudioPlayer.
In this case, polyphony is the production of multiple sounds simultaneously (I’m not referring to the concept of polyphany in music theory). In describing concurrency, I refered to the ability to play 2 separate sounds at the same time, where with polyphony I refer to the ability to play the same sound “on top” of itself. There can be multiple “voices” of the same sound. In the most literal of definitions concurrency could be considered a part of polyphony, and polyphony a part of concurrency… Hopefully you get what I’m trying to say. In its current state, HTML5 audio supports neither concurrency or polyphony. The PhoneGap Media class does not support polyphony, however you can probably manage multiple media instances via javascript to achieve polyphonic behavior – this requires additional work in the JavaScript side of things to juggle resources.
Low Latency
Low latency refers to “human-unnoticeable delays between an input being processed and the corresponding output providing real time characteristics” according to wikipedia. In this case, I refer to low latency audio, meaning that there is an imperceptible delay between when a sound is triggered, and when it actually plays. This means that sounds will play when expected, not after a wait. This means a bouncing ball sound should be heard as you see the ball bouncing on the screen. Not after it has already bounced.
In HTML5, you can auto-load a sound so that it is ready when you need it, but don’t expect to play more than one at a time. With the PhoneGap Media class, the audio file isn’t actually requested until you invoke “play”. This occurs inside “startPlaying” on Android, and “play” on iOS. What I wanted was a way to preload the audio so that it is immediately ready for use at the time it is needed.
The Solution
PhoneGap makes it really easy to build natively installed applications using a familiar paradim: HTML & JavaScript. Luckily, PhoneGap also allows you to tie into native code using the native plugin model. This enables you to write your own native code and expose that code to your PhoneGap application via a JavaScript interface… and that is exactly what I did to enable low-latency, concurrent, and polyphonic audio in a PhoneGap experience.
I created PhoneGap native plugins for Android and iOS that allow you to preload audio, and playback that audio quickly, with a very simple to use API. I’ll get into details how this works further in the post, but you can get a pretty good idea of what I mean by viewing the following two videos.
The first is a basic “Drum Machine”. You just tap the pads to play an audio sample.
The second is a simple user interface that allows you to layer lots of complex audio, mimicking scenarios that may occur within a video gaming context.
Assets used in this example from freesound.org. See README for specific links & attribution.
You may have noticed a slight delay in this second video between the tap and the actual sounds. This is because I am using “touchStart” events in the first example, and just using a normal <a href=”javascript:foo()”> link in the second. There is always a delay for “normal” links in all multi-touch devices/environments because there has to be time for the device to detect a gesture event. You can bypass this delay in mobile web browsers by using touch events for all input.
Side Note: I have also noticed that touch events are slightly slower to be recognized on Android devices than iOS. My assumption is that this is related to specific device capabilities – this is more noticeable on the Amazon Kindle Fire than the Motorola Atrix. The delay does not appear to be a delay in the actual audio playback.
How it works
The native plugins expose a very simple API for hooking into native Audio capabilities. The basic usage is:
Preload the audio asset
Play the audio asset
When done, unload the audio asset to conserve resources
The basic components of a PhoneGap native plugin are:
A JavaScript interface
Corresponding Native Code classes
You can learn more about getting started with native plugins on the PhoneGap wiki.
Let’s start by examining the native plugin’s JavaScript API. You can see that it just hands off the JavaScript calls to the native layer via PhoneGap:
Next, let’s examine some intricacies of the plugin… One thing to keep in mind is that I do not have callbacks to the phonegap app once a media asset is loaded. If you need “loaded” callbacks, you will need to add those yourself.
preloadFX: function ( id, assetPath, success, fail)
params:
id – string unique ID for the audio file assetPath – the relative path to the audio asset within the www directory success – success callback function fail – error/fail callback function
detail:
The preloadFX function loads an audio file into memory. These are lower-level audio methods and have minimal overhead. These assets should be short (less than 5 seconds). These assets are fully concurrent and polyphonic.
On Android, assets that are loaded using preloadFX are managed/played using the Android SoundPool class. Sound files longer than 5 seconds may have errors including (not playing, clipped content, not looping) – all will fail silently on the device (debug output will be visible if connected to debugger).
On iOS, assets that are loaded using preloadFX are managed/played using System Sound Services from the AudioToolbox framework. Audio loaded using this function is played using AudioServicesPlaySystemSound. These assets should be short, and are not intended to be looped or stopped.
preloadAudio: function ( id, assetPath, voices, success, fail)
params:
id – string unique ID for the audio file assetPath – the relative path to the audio asset within the www directory voices – the number of polyphonic voices available success – success callback function fail – error/fail callback function
detail:
The preloadAudio function loads an audio file into memory. These have more overhead than assets laoded via preloadFX, and can be looped/stopped. By default, there is a single “voice” – only one instance that will be stopped & restarted when you hit play. If there are multiple voices (number greater than 0), it will cycle through voices to play overlapping audio. You must specify multiple voices to have polyphonic audio – keep in mind, this takes up more device resources.
On Android, assets that are loaded using preloadAudio are managed/played using the Android MediaPlayer.
On iOS, assets that are loaded using preloadAudio are managed/played using AVAudioPlayer.
play: function (id, success, fail)
params:
id – string unique ID for the audio file success – success callback function fail – error/fail callback function
detail:
Plays an audio asset. You only need to pass the audio ID, and the native plugin will determine the type of asset and play it.
loop: function (id, success, fail)
params:
id – string unique ID for the audio file success – success callback function fail – error/fail callback function
detail:
Loops an audio asset infinitely. On iOS, this only works for assets loaded via preloadAudio. This works for all asset types for Android, however it is recommended to keep usage consistent between platforms.
stop: function (id, success, fail)
params:
id – string unique ID for the audio file success – success callback function fail – error/fail callback function
detail:
Stops an audio file. On iOS, this only works for assets loaded via preloadAudio. This works for all asset types for Android, however it is recommended to keep usage consistent between platforms.
unload: function (id, success, fail)
params:
id – string unique ID for the audio file success – success callback function fail – error/fail callback function
detail:
Unloads an audio file from memory. DO NOT FORGET THIS! Otherwise, you will cause memory leaks.
I’m not just doing this for myself, the audio is completely open source for you to take advantage of as well. You can download the full code, as well as all examples from github at github:
Do you want to learn more about the future of Flex?
Do you want to learn more about the Flex transition to the Apache Software Foundation?
How can you contribute and help make Flex thrive?
Do you have questions that you would like to voice to Adobe?
As promised, Adobe is kicking off The Flex User Group Tour to discuss recent events surrounding Flex and the Flash Platform. These meetings are intended to help you understand the changes happening with Flex and Flash, the impact to related tools, as well as to educate about the process & transition to Apache. You can learn more about the user group tour and get an up-to-date listing of dates & cities from the Flex Team blog - be sure to check back periodically for updates. Initial cities include New York, Boston, Denver, Seattle, Los Angeles, Sand Diego, and Dallas. Expect more cities & countries to be announced at a later date.
We hope to see you at one of the upcoming events. I’m scheduled to speak at the Dallas event in April, and I hop to see you there!
Did you know that apps built on top of iOS can have a multi-screen workflow? For example in Keynote, you can have an external screen show a presentation while you control it on your iOS device. In the Jimi Hendrix app, you can view the audio player on an external screen, and in Real Racing HD, you can view the game on an external screen while the iOS device becomes your controller. (among others)
Real Racing HD
This is all made possible by the UIWindow and UIScreen APIs in iOS. Even better, on the iPad 2 and iPhone 4Gs, this can be done wirelessly using Airplay with an Apple TV device. On other iOS devices, you can have a second screen using a VGA output.
One of the benefits of using a cross platform solution like PhoneGap or Flex/Air is that you can build apps with an easier to use/more familiar paradigm. However, cross platform runtimes don’t always offer access to every API feature that native development enables.
Out of the box, PhoneGap apps are confined to a single screen. You can use screen mirroring to mirror content on an external screen, but you can’t have a second screen experience. It’s a good thing you can write native plugins/extensions to enable native functionality within your applications.
ExternalScreen Native Plugin For PhoneGap
I recently did exactly that… I created a PhoneGap native plugin that enables second screen capability for PhoneGap applications. The plugin listens for external screen connection notifications, and if an additional screen is available, it creates a new UIWebView for HTML-based content in the external screen – complete with functions for injecting HTML, JavaScript, or URL locations.
Why?
You might be wondering “Why?” you would want this plugin within PhoneGap… this plugin enables the multi-screen experiences described in the apps mentioned above. They extend the interactions and capabilities of the mobile hardware. With this PhoneGap native plugin, you can create rich multi-screen experiences with the ease of HTML and JavaScript. Here are a few ideas of the types of apps that you can build with this approach (scroll down for source code):
Fleet Manager
Let’s first consider a simple Fleet Manager application which allows you monitor vehicles in a mobile app. This is a similar concept which I’ve used in previous examples. The basic functionality allows you to see information on the tablet regarding your fleet. What if this app connected to a larger screen and was able to display information about your vehicles for everyone to see? Watch the video below to see this in real life.
This application example is powered by Google Maps, and all of the data is randomly generated on the client.
Law Enforcement
Let’s next consider a mobile law enforcement application application which gives you details to aid in investigations and apprehension of criminals. Let’s pretend that you are a detective who is searching for a fugitive, and you walk into a crowded bar near the last known location of that fugitive. You connect to the bar’s Apple TV on their big screen TV, pull up images and videos of the suspect, then say “Have you seen this person?”. This could be incredibly powerful. Check out the video below to see a prototype in real life.
This law enforcement demo scenario is a basic application powered by the FBI’s most wanted RSS data feeds.
Tip Of The Iceberg
There are lots of use cases where a second screen experience could be beneficial and create a superior product or application. Using PhoneGap allows you to build those apps faster & with the ease of HTML and JavaScript, using traditional web development paradigms.
The PhoneGap native plugin is written in Objective C, with a JavaScript interface to integrate with the client application. PhoneGap plugins are actually very easy to develop. Basically, you have to write the native code class, write a corresponding JS interface, and add a mapping in your PhoneGap.plist file to expose the new functionality through PhoneGap. There is a great reference on the PhoneGap wiki for native plugins which includes architecture & structure, as well as platform specific authoring and installation of those plugins. Here are quick links to the iOS-specific native plugin content authoring and installation.
The ExternalScreen plugin creates a UIWebView for the the external screen, and exposes methods for interacting with the UIWebView. Note: This is just a normal UIWebView, it does not have support for all PhoneGap libraries… just a standard HTML container.
You can read up on multi-screen programming at iOS from these useful tutorials:
The header file shows the method signatures for the native functionality. The corresponding PGExternalScreen.m contains all of the actual code to make it all work. Note: If you are using ARC (Automatic Reference Counting), you will need to remove the retain/release calls in PGExternalScreen.m.
The PGExternalScreen.js file defines the native methods that are exposed through PhoneGap. You invoke the function, and can add success/fail callback function references.
You can call any of these functions from within your PhoneGap application’s JavaScript just by referencing the exposed method on the PGExternalScreen instance.
// check if an external screen is available
PGExternalScreen.checkExternalScreenAvailable( resultHandler, errorHandler );
//load a local HTML resource
PGExternalScreen.loadHTMLResource( 'secondary.html', resultHandler, errorHandler );
//load a remote HTML resource (requires the URL to be white-listed in PhoneGap)
PGExternalScreen.loadHTMLResource( 'http://www.tricedesigns.com', resultHandler, errorHandler );
//load a HTML string
PGExternalScreen.loadHTML('</pre>
<h1>HTML</h1>
<pre>this is html content', resultHandler, errorHandler );
//invoke a JavaScript (passed as a string)
PGExternalScreen.invokeJavaScript('document.write(\'hello world\')', resultHandler, errorHandler );
The full code for the ExternalScreen PhoneGap native plugin, as well as both client applications and a basic usage example is available on github at:
After spending some time playing around sketching with the HTML5 canvas element earlier this week, I figured “why not add some ‘enterprise’ concepts to this example?”… Next thing you know we’ve got a multi-device shared sketching/collaboration experience.
To keep things straightforward, I chose to demonstrate the near-realtime collaboration using a short-interval HTTP poll. HTTP polling is probably the simplest form of near-realtime data in web applications, however you may experience lag when compared to a socket connection of equivalent functionality. I’ll discuss the various realtime data options you have in Flex/Flash and HTML/JS and their pros & cons further in this post.
What you’ll see in the video below is the sketching example with realtime collaboration added using short-interval data polling of a ColdFusion application server. The realtime collaboration is shown between an iPad 2, a Kindle Fire, and a Macbook Pro.
Before we get into the code for this example, let’s first review some realtime data basics…
First, why/when would you need realtime data in your applications? Here are just a few:
Time sensitive information, where any delay could have major repercussions
Realtime financial information
Emergency services (medical, fire, police)
Military/Intelligence scenarios
Business critical efficiency/performance metrics
Collaboration
Realtime audio/video collaboration
Shared experience (presentations/screen sharing)
Entertainment
Streaming media (audio/video)
Gaming
Regardless of whether you are building applications for mobile, the web, or desktop, using any technology (Flex/Flash, HTML/JS, Java, .NET, Objective C, or C/C++ (among others)), there are basically 3 methods for streaming/realtime data:
Socket Connection
HTTP Polling
HTTP Push
Socket Connections
Socket connectionss are basically end-to-end communications channels between two computer processes. Your computer (a client) connects to a server socket and establishes a persistent connection that is used to pass data between the client and server in near-realtime. Persistent socket connections are generally based upon TCP or UDP and enable asynchronus bidirectional communication. Binary or Text-based messages can be sent in either direction at any point in time, in any sequence, as data is available. In HTML/JS applications you can use web sockets, which I recently discussed, or use a plugin that handles realtime socket communication. Did you also know that the next version of ColdFusion will even have web socket support built in? In Flash/Flex/AIR, this can be achieved using the RTMP protocol (LCDS, Flash Media Server, etc…) or raw sockets (TCP or UDP).
Direct Socket Communications
In general, direct socket based communication is the most efficient means of data transfer for realtime application scenarios. There is less back and forth handshaking and less packet encapsulation required by various protocols (HTTP, etc…), and you are restricted by fewer network protocol rules. However, socket based communications often run on non-standard or restricted ports, so they are more likely to be blocked by IT departments or stopped by network firewalls. If you are using socket based communication within your applications, which are running on non-standard ports, and you don’t govern the network, you may want a fallback to another realtime data implementation for failover cases.
HTTP Polling
HTTP Polling is the process of using standard HTTP requests to periodically check for data updates on the server. The client application requests information from the server. Generally, the client will send a timestamp indicating the last data update time. If there is information available on the server that is newer than the timestamp, that data will be immediately sent back to the client (and the client’s timestamp will be updated). After a period of time, another request will be made, and so forth until the polling is stopped within the application. Using this approach, the application is more-or-less “phoning home” periodically to the server to see if there are any updates. You can achieve near-realtime performance by setting a very short polling interval (less than one second).
Basic Data Poll Sequence
HTTP polling uses standard web protocols and ports, and generally will not be blocked by firewalls. You can poll on top of standard HTTP (port 80) or HTTPS (port 443) without any issue. This can be achieved by polling JSON services, XML Services, AMF, or any other data format on top of a HTTP request. HTTP polling will generally be slower than a direct socket method, and will also utilize more network bandwidth b/c of request/response encapsulation and the periodic requests to the server. It is also important to keep in mind that the HTTP spec only allows for 2 concurrent connections to a server at any point in time. Polling requests can consume HTTP connections, thus slowing load time for other portions of your application. HTTP polling can be employed in HTML/JS, Flex/Flash/AIR, desktop, server, or basically any other type of application using common libraries & APIs.
HTTP Push
HTTP Push technologies fall into 2 general categories depending upon the server-side technology/implementation. This can refer to HTTP Streaming, where a connection is opened between the client and server and kept open using keep-alives. As data is ready to send to the client, it will be pushed across the existing open HTTP connection. HTTP Push can also refer to HTTP Long Polling, where the client will periodically make a HTTP request to the server, and the server will “hold” the connection open until data is available to send to the client (or a timeout occurs). Once that request has a complete response, another request is made to open another connection to wait for more data. Once Again, with HTTP Long Poll there should be a very short polling interval to maintain near-realtime performance, however you can expect some lag.
HTTP Long Poll Sequence
HTTP Streaming & HTTP Long polling can be employed in HTML/JS applications using the Comet approach (supported by numerous backend server technologies) and can be employed in Flex/Flash/AIR using BlazeDS or LCDS.
Collaborative Applications
Now back to the collaborative sketching application shown in the video above… the application builds off of the sketching example from previous blog posts. I added logic to monitor the input sketches and built a HTTP poll-based monitoring service to share content between sessions that share a common ID.
Realtime Collaborative Sketches
In the JavaScript code, I created an ApplicationController class that acts as an observer to the input from the Sketcher class. The ApplicationController encapsulates all logic handling data polling and information sharing between sessions. When the application loads, it sets up the polling sequence.
The polling sequence is setup so that a new request will be made to the server 250MS after receiving a response from the previous request. Note: this is very different from using a 250MS interval using setInterval. This approach guarantees 250MS from response to the next request. If you use a 250MS interval using setInterval, then you are only waiting 250MS between each request, without waiting for a response. If your request takes more than 250 MS, you will can end up have stacked, or “concurrent” requests, which can cause serious performance issues.
When observing the sketch input, the start and end positions and color for each line segment get pushed into a queue of captured transactions that will be pushed to the server. (The code supports multiple colors, even though there is no method to support changing colors in the UI.)
When a poll happens, the captured transactions are sent to the server (a ColdFusion CFC exposed in JSON format) as a HTTP post.
ApplicationController.prototype.poll = function () {
this.pendingTransactions = this.capturedTransactions;
this.capturedTransactions = [];
var data = { "method":"synchronize",
"id":this.id,
"timestamp":this.lastTimeStamp,
"transactions": JSON.stringify(this.pendingTransactions),
"returnformat":"json" };
var url = "services/DataPollGateway.cfc";
$.ajax({
type: 'POST',
url: url,
data:data,
success: this.getRequestSuccessFunction(),
error: this.getRequestErrorFunction()
});
}
The server then stores the pending transactions in memory (I am not persisting these, they are in-ram on the server only). The server checks the transactions that are already in memory against the last timestamp from the client, and it will return all transactions that have taken place since that timestamp.
<cffunction name="synchronize" access="public" returntype="struct">
<cfargument name="id" type="string" required="yes">
<cfargument name="timestamp" type="string" required="yes">
<cfargument name="transactions" type="string" required="yes">
<cfscript>
var newTransactions = deserializeJSON(transactions);
if( ! structkeyexists(this, "id#id#") ){
this[ "id#id#" ] = ArrayNew(1);
}
var existingTransactions = this[ "id#id#" ];
var serializeTransactions = ArrayNew(1);
var numberTimestamp = LSParseNumber( timestamp );
//check existing tranactions to return to client
for (i = 1; i lte ArrayLen(existingTransactions); i++) {
var item = existingTransactions[i];
if ( item.timestamp GT numberTimestamp ) {
ArrayAppend( serializeTransactions, item.content );
}
}
var newTimestamp = GetTickCount();
//add new transactions to server
for (i = 1; i lte ArrayLen(newTransactions); i++) {
var item = {};
if ( structkeyexists( newTransactions[i], "clear" )) {
serializeTransactions = ArrayNew(1);
existingTransactions = ArrayNew(1);
}
item.timestamp = newTimestamp;
item.content = newTransactions[i];
ArrayAppend( existingTransactions, item );
}
var result = {};
result.transactions = serializeTransactions;
result.timestamp = newTimestamp;
this[ "id#id#" ] = existingTransactions;;
</cfscript>
<cfreturn result>
</cffunction>
When a poll request completes, any new transactions are processed and a new poll is requested.
ApplicationController.prototype.getRequestSuccessFunction = function() {
<pre> var self = this;
return function( data, textStatus, jqXHR ) {
var result = eval( "["+data+"]" );
if ( result.length > 0 )
{
var transactions = result[0].TRANSACTIONS;
self.lastTimeStamp = parseInt( result[0].TIMESTAMP );
self.processTransactions( transactions );
}
self.pendingTransactions = [];
self.requestPoll();
}
}
You can access the full client and server application source on Github at:
Apache Flex allows developers to target a variety of platforms, initially Apple iOS, Google Android, RIM BlackBerry, Microsoft Windows, and Mac OS X with a single codebase. Flex provides a compiler, skinnable user-interface components and managers to handle styling, skinning, layout, localization, animation, module-loading and user interaction management.
Just a bit of extra detail for you, all of which is available through Apache:
One growing trend that I have seen in mobile & tablet applications is the creation of tools that enable your workforce to perform their job better. This can be in the case of mobile data retrieval, streamlined sales process with apps for door-to-door sales, mobile business process efficiency, etc…
One of the topics that comes up is how do you capture a signature and store it within your application? This might be for validation that the signer is who they say they are, or for legal/contractual reasons. Imagine a few scenarios:
Your cable TV can’t be installed until you sign the digital form on the installation tech’s tablet device
You agree to purchase a service from a sales person (door to door, or in-store kiosk) – your signature is required to make this legally binding.
Your signature is required to accept an agreement before confidential data is presented to you.
These are just a few random scenarios, I’m sure there are many more. In this post, I will focus on 2 (yes, I said two) cross-platform solutions to handle this task – one built with Adobe Flex & AIR, and one built with HTML5 Canvas & PhoneGap.
Watch the video below to see this in action, then we’ll dig into the code that makes it work.
The basic flow of the application is that you enter an email address, sign the interface, then click the green “check” button to submit to the signature to a ColdFusion server. The server then sends a multi-part email to the email address that you provided, containing text elements as well as the signature that was just captured.
If you’d like to jump straight to specific code portions, use the links below:
Let’s first examine the server component of the sample application. The server side is powered by ColdFusion. There’s just a single CFC that is utilized by both the Flex/AIR and HTML/PhoneGap front-end applications. The CFC exposes a single service that accepts two parameters: the email address, and a base-64 encoded string of the captured image data.
<cffunction name="submitSignature" access="remote" returntype="boolean">
<cfargument name="email" type="string" required="yes">
<cfargument name="signature" type="string" required="yes">
<cfmail SUBJECT ="Signature"
FROM="#noReplyAddress#"
TO="#email#"
username="#emailLoginUsername#"
password="#emailLoginPassword#"
server="#mailServer#"
type="HTML" >
<p>This completes the form transaction for <strong>#email#</strong>.</p>
<p>You may view your signature below:</p>
<p><img src="cid:signature" /></p>
<p>Thank you for your participation.</p>
<cfmailparam
file="signature"
content="#toBinary( signature )#"
contentid="signature"
disposition="inline" />
</cfmail>
<cfreturn true />
</cffunction>
Note: I used base-64 encoded image data so that it can be a single server component for both user interfaces. In Flex/AIR you can also serialize the data as a binary byte array, however binary serialization isn’t quite as easy with HTML/JS… read on to learn more.
The Flex/AIR Solution
The main user interface for the Flex/AIR solution is a simple UI with some form elements. In that UI there is an instance of my SignatureCapture user interface component. This is a basic component that is built on top of UIComponent (the base class for all Flex visual components), which encapsulates all logic for capturing the user signature. The component captures input based on mouse events (single touch events are handled as mouse events in air). The mouse input is then used to manipulate the graphics content of the component using the drawing API. I like to think of the drawing API as a language around the childhood game “connect the dots”. In this case, you are just drawing lines from one point to another.
When the form is submitted, the graphical content is converted to a base-64 encoded string using the Flex ImageSnapshot class/API, before passing it to the server.
You can check out a browser-based Flex version of this in action at http://tricedesigns.com/portfolio/sigCaptureFlex/ – Just enter a valid email address and use your mouse to sign within the signature area. When this is submitted, it will send an email to you containing the signature.
package
{
import flash.display.DisplayObject;
import flash.display.Graphics;
import flash.display.Sprite;
import flash.events.MouseEvent;
import flash.geom.Point;
import mx.core.UIComponent;
import mx.graphics.ImageSnapshot;
import mx.managers.IFocusManagerComponent;
import spark.primitives.Graphic;
public class SignatureCapture extends UIComponent
{
private var captureMask : Sprite;
private var drawSurface : UIComponent;
private var lastMousePosition : Point;
private var backgroundColor : int = 0xEEEEEE;
private var borderColor : int = 0x888888;
private var borderSize : int = 2;
private var cornerRadius :int = 25;
private var strokeColor : int = 0;
private var strokeSize : int = 2;
public function SignatureCapture()
{
lastMousePosition = new Point();
super();
}
override protected function createChildren():void
{
super.createChildren();
captureMask = new Sprite();
drawSurface = new UIComponent();
this.mask = captureMask;
addChild( drawSurface );
addChild( captureMask );
this.addEventListener( MouseEvent.MOUSE_DOWN, onMouseDown );
}
protected function onMouseDown( event : MouseEvent ) : void
{
lastMousePosition = globalToLocal( new Point( stage.mouseX, stage.mouseY ) );
stage.addEventListener( MouseEvent.MOUSE_MOVE, onMouseMove );
stage.addEventListener( MouseEvent.MOUSE_UP, onMouseUp );
}
protected function onMouseMove( event : MouseEvent ) : void
{
updateSegment();
}
protected function onMouseUp( event : MouseEvent ) : void
{
updateSegment();
stage.removeEventListener( MouseEvent.MOUSE_MOVE, onMouseMove );
stage.removeEventListener( MouseEvent.MOUSE_UP, onMouseUp );
}
protected function updateSegment() : void
{
var nextMousePosition : Point = globalToLocal( new Point( stage.mouseX, stage.mouseY ) );
renderSegment( lastMousePosition, nextMousePosition );
lastMousePosition = nextMousePosition;
}
public function clear() : void
{
drawSurface.graphics.clear();
}
override public function toString() : String
{
var snapshot : ImageSnapshot = ImageSnapshot.captureImage( drawSurface );
return ImageSnapshot.encodeImageAsBase64( snapshot );
}
override protected function updateDisplayList(w:Number, h:Number):void
{
super.updateDisplayList(w,h);
drawSurface.width = w;
drawSurface.height = h;
var g : Graphics = this.graphics;
//draw rectangle for mouse hit area
g.clear();
g.lineStyle( borderSize, borderColor, 1, true );
g.beginFill( backgroundColor, 1 );
g.drawRoundRect( 0,0,w,h, cornerRadius, cornerRadius );
//fill mask
g.clear();
g = captureMask.graphics;
g.beginFill( 0, 1 );
g.drawRoundRect( 0,0,w,h, cornerRadius, cornerRadius );
}
protected function renderSegment( from : Point, to : Point ) : void
{
var g : Graphics = drawSurface.graphics;
g.lineStyle( strokeSize, strokeColor, 1 );
g.moveTo( from.x, from.y );
g.lineTo( to.x, to.y );
}
}
}
The HTML5/PhoneGap Solution
The main user interface for the HTML5/PhoneGap solution is also a simple UI with some form elements. In that UI there is a Canvas element that is used to render the signature. I created a SignatureCapture JavaScript class that encapsulates all logic for capturing the user signature. In browsers that support touch events (mobile browsers), this is based on the touchstart, touchmove and touchend events. In browsers that don’t support touch (aka desktop browsers), the signature input is based on mousedown, mousemove and mouseup events. The component captures input based on touch or mouse events, and that input is used to manipulate the graphics content of the Canvas tag instance. The canvas tag also supports a drawing API that is similar to the ActionScript drawing API. To read up on Canvas programmatic drawing basics, check out the tutorials at http://www.adobe.com/devnet/html5/html5-canvas.html
When the form is submitted, the graphical content is converted to a base-64 encoded string using the Canvas’s toDataURL() method. The toDataURL() method returns a base-64 encoded string value of the image content, prefixed with “data:image/png,”. Since I’ll be passing this back to the server, I don’t need this prefix, so it is stripped, then sent to the server for content within the email.
You can check out a browser-based version of this using the HTML5 Canvas in action at http://tricedesigns.com/portfolio/sigCapture/ – Again, just enter a valid email address and use your mouse to sign within the signature area. When this is submitted, it will send an email to you containing the signature. However, this example requires that your browser supports the HTML5 Canvas tag.
You can check out the SignatureCapture code below, or check out the full project at https://github.com/triceam/Mobile-Signature-Capture/tree/master/html%20client. This class will also work in desktop browser applications that support the HTML5 canvas. I used Modernizr to determine whether touch events are supported within the client container (PhoneGap or desktop browser). The main application workflow is within application.js.
Also a note for Android users, the Canvas toDataURL() method does not work in Android versions earlier than 3.0. However, you can implement your own toDataURL() method for use in older OS versions using the technique in this link: http://jimdoescode.blogspot.com/2011/11/trials-and-tribulations-with-html5.html (I did not update this example to support older Android OS versions.)
While working with Kevin’s code, I started tinkering… “what if I change this, what if I tweak that?” Next thing you know, I put together a sample scenario showing subscription-based realtime data streaming to multiple web clients using web sockets. Check out the video below to see it in action.
You are seeing 9 separate browser instances getting realtime push-based updates from a local server using web sockets. When the browser loads, the html-based client makes a web socket connection, then requests all symbols from the server. The server then sends the stock symbol definitions back to the client and displays them within the HTML user interface. From there, the user can click on a stock symbol to subscribe to updates for that particular symbol. DISCLAIMER: All that data is randomly generated!
I put together this example for experimentation, but also to highlight a few technical scenarios for HTML-based applications. Specifically:
Realtime/push data in HTML-based apps
Per-client subscriptions for realtime data
Multi-series realtime data visualization in HTML-based apps
The server is an AIR app started by Kevin, based on the web sockets draft protocol. It is written in JavaScript, and the client is a HTML page to be viewed in the browser.
If you don’t feel like reading the full web sockets protocol reference, you can get a great overview from websocket.org or Wikipedia.
One thing to keep in mind is that web sockets are not widely supported in all browsers yet. There is a great reference matrix for web socket support from caniuse.com:
If you still aren’t sure if your browser supports web sockets, you can also check simply by visiting websocketstest.com/. If you want to test for web socket support within your own applications, you can easily check for support using Modernizr. Note: I didn’t add the Modernizr test in this example… I only tested in Chrome on OSX.
OK, now back to the sample application. All of the source code for this example is available on github at: https://github.com/triceam/Websocket-Streaming-Example. To run it yourself, you first have to launch the server. You can do this on the command line by invoking ADL (part of the AIR SDK):
cd "/Applications/Adobe Flash Builder 4.6/sdks/4.6.0/bin"
./adl ~/Documents/dev/Websocket-Streaming-Example/server/application.xml
You’ll know the server is started b/c an air window will popup (you can ignore this, just don’t close it), and you will start seeing feed updates in the console output.
Once the server is running, open “client/client.html” in your browser. It will connect to the local server, and then request the list of symbols. If you click on a symbol, it will subscribe to that feed. Just click on the symbol name again to unsubscribe. You’ll know the feed is subscribed b/c the symbol will show up in a color (matching the corresponding feed on the chart). Again, let me reiterate that I only tested this in Chrome.
You can open up numerous client instances, and all will receive the same updates in real time for each subscribed stock symbol.
The “meat” of code for the server starts in server/scripts/server/server.js. Basically, the server loads a configuration file for the socket server, then creates a ConnectionManager and DataFeed (both of these are custom JS classes). The ConnectionManager class encapsulates all logic around socket connections. This includes managing the ServerSocket as well as all client socket instances and events. The DataFeed class handles data within the app. First, it generates random data, then sets up an interval to generate random data updates. For every data update, the ConnectionManager instance’s dispatch() method is invoked to send updates to all subscribed clients. Rather than trying to put lots of code snippets inline in this post (which would just be more confusing), check out the full source at: https://github.com/triceam/Websocket-Streaming-Example/tree/master/server
The client code all starts in client.html, with the application logic inside of client/scripts/client.js. Once the client interface loads, it connects to the web socket and adds the appropriate event handlers. Once subscribed to a data feed, realtime data will be returned via the web socket instance, transformed slightly to fit the data visualization structure, then rendered in an HTML canvas using the RGraph data visualization library. RGraph is free to get started with, however if you want to deploy a production app with it, you’ll need a license. You’ll notice that each feed updates independently, based upon the client subscriptions. Note: The data visualization is not temporally aligned… if you want the updates in time-sequence, there is a litte bit more work involved in the client-side data transformation.
This example is intended to get your minds rolling with the concepts; it is not *yet* an all-encompassing enterprise solution. You can expect to see a few more data push scenarios here in the near future, based on different enterprise server technologies.
In case you had not seen on the Flex team blog, twitter or through some other medium, Flex SDK 4.6 and Flash Builder 4.6 were released today! Go get them, if you have not done so already. Flex 4.6 marks a huge advancement for the Flex SDK, especially regarding mobile applications.
Flash Builder 4.6 is a FREE update for Flash Builder 4.5 users. From the Flex team blog:
A lot is included in this update, so much so that we couldn’t deliver it in the Adobe Application Manager. This means Flash Builder 4.5 users won’t automatically be notified about the update and will have to download the full Flash Builder 4.6 installer and enter their Flash Builder 4.5 serial number.
Coinciding with the Flex & Flash Builder releases, new content around Flex and Flash Builder 4.6 have been posted on Adobe TV. There is a bunch of great new content worth checking out, including fellow evangelist Michael Chaize’s adaptive UI for different platforms and device form factors. In addition, here I am speaking out the new Captive Runtime feature introduced in AIR 3…
Very often I see an argument against Adobe technology claiming that Adobe isn’t “open” or is based on “closed” technology. This is far from the the truth, and I’d like to highlight some of Adobe’s involvement in open source initiatives and open standards.
Webkit
First, let’s focus on Webkit, the open source web browser engine used in Chrome, Safari, iOS, BlackBerry, Android, and other systems… Adobe has contributed code for both CSS Shaders and CSS Regions to webkit (among others), and is taking steps to become a more active contributor to the webkit browser engine.
CSS Shaders
CSS Shaders will enable rich & high quality visual effects within HTML-based content. Check out the video below to get a better idea of what will be possible with CSS Shaders.
CSS Regions will enable rich, non-rectangular layout, with overflow between region containers. Imagine text wrapping area of an image (not based on the bounding rectangle), or text flowing between containers in a non-linear fashion. CSS Regions will enable significantly richer layout in content across the web. Check out the next video to see CSS Regions in action.
PhoneGap is “an HTML5 app platform that allows you to author native applications with web technologies and get access to APIs and app stores.” PhoneGap has always been open source. Adobe acquired Nitobi, the creators of PhoneGap, and with that acquisition the PhoneGap project was submitted to the Apache Software Foundation as Apache Callback.
CQ/CRX
The CQ content management system not only is built on top of open source software, but the development team also contributes back to open source in many projects, and is deeply involved with the Apache Software Foundation.
In addition to driving industry standards, Day Software has been a leading force in the open source community. Day Software’s Roy Fielding was co-founder of the Apache Software Foundation, author of the Apache Software license, and creator of the Apache web server.
Day’s R&D team are strong contributors to the open source world, with a development model based on building true open source communities around key technology advancements that originate in Day R&D through the sponsorship of new projects via the Apache Software Foundation.
Top-level projects such as Apache Jackrabbit (the reference implementation of the JSR-170 standard), Apache Sling (an industry-first REST-based web application development framework), and Apache Felix (an OSGi R4 Service Platform implementation) are examples of Day’s unique commitment to open source.
In total, Day Software contributes to over 12 Apache projects and 25 open source projects. www.ohloh.org, an independent website that tracks open source contributions, shows that over 75% of Day engineers are active committers to open source projects, with over 75% of those engineers being ranked in the top 1% of open source developers worldwide.
Flex has been open source and available since version 3.0 (around 2009 I think). Recently, it was announced that Flex will be contributed towards an open source software foundation. This move will give the Flex framework direction and tooling to the community for further management and development. Be sure to read the official statement and FAQ for further detail.
As of November 2011, Adobe is in the process of preparing two proposals for incubating Flex SDK and BlazeDS at the Apache Software Foundation.
In addition to contributing the core Flex SDK (including automation and advanced data visualization components), Adobe also plans to donate the following:
Complete, but yet-to-be-released, Spark components, including ViewStack, Accordion, DateField, DateChooser and an enhanced DataGrid.
BlazeDS, the server-based Java remoting and web messaging technology that enables developers to easily connect to back-end distributed data and push data in real-time to Flex applications.
Falcon, the next-generation MXML and ActionScript compiler that is currently under development (this will be contributed when complete in 2012)
Falcon JS, an experimental cross-compiler from MXML and ActionScript to HTML and JavaScript.
Flex testing tools, as used previously by Adobe, so as to ensure successful continued development of Flex with high quality
Adobe will also have a team of Flex SDK engineers contributing to those new Apache projects as their full-time responsibility. Adobe has in-development work already started, including additional Spark-based components.
Open @ Adobe on SourceForge
Open @ Adobe is another collection of open source projects that are contributed by Adobe. This contains OSMF (Open Source Media Framework), FlexPMD (code coverage), CSS Regions, CSS Shaders, the Text Layout Framework (TLF), Flex Unit, and many others. Be sure to also check out the Open @ Adobe blog.
RIAForge.org
RIAForge is an open source online community sponsored by Adobe, and developed/maintained by fellow evangelist Raymond Camden. RIAForge hosts over 1225 active open source projects, with over 19 million project page views, and with each project you get a blog, issue tracker, subversion hosting, forums, and statistics.
opensource.adobe.com
Of course, don’t forget opensource.adobe.com, where you can read about additional open source projects at Adobe.
This is not meant to be a definitive list, and it doesn’t even touch on the subject of open file formats or protocols. If you didn’t already, be sure to check out the Open @ Adobe blog.
If you hadn’t heard already, 
First, let’s focus on 
Next, let’s take a look at 
The 
Flex has been open source and available since version 3.0 (around 2009 I think). Recently, it was announced that Flex will be contributed towards an open source software foundation. This move will give the Flex framework direction and tooling to the community for further management and development. Be sure to read the 
Recent Comments