Realtime Data & Your Applications

After spending some time playing around sketching with the HTML5 canvas element earlier this week, I figured “why not add some ‘enterprise’ concepts to this example?”…  Next thing you know we’ve got a multi-device shared sketching/collaboration experience.

To keep things straightforward, I chose to demonstrate the near-realtime collaboration using a short-interval HTTP poll.  HTTP polling is probably the simplest form of near-realtime data in web applications, however you may experience lag when compared to a socket connection of equivalent functionality.   I’ll discuss the various realtime data options you have in Flex/Flash and HTML/JS and their pros & cons further in this post.

What you’ll see in the video below is the sketching example with realtime collaboration added using short-interval data polling of a ColdFusion application server.  The realtime collaboration is shown between an iPad 2, a Kindle Fire, and a Macbook Pro.

Before we get into the code for this example, let’s first review some realtime data basics…

First, why/when would you need realtime data in your applications?  Here are just a few:

  • Time sensitive information, where any delay could have major repercussions
    • Realtime financial information
    • Emergency services (medical, fire, police)
    • Military/Intelligence scenarios
    • Business critical efficiency/performance metrics
  • Collaboration
    • Realtime audio/video collaboration
    • Shared experience (presentations/screen sharing)
  • Entertainment
    • Streaming media (audio/video)
    • Gaming

Regardless of whether you are building applications for mobile, the web, or desktop, using any technology (Flex/Flash, HTML/JS, Java, .NET, Objective C, or C/C++ (among others)), there are basically 3 methods for streaming/realtime data:

  • Socket Connection
  • HTTP Polling
  • HTTP Push

Socket Connections

Socket connectionss are basically end-to-end communications channels between two computer processes.   Your computer (a client) connects to a server socket and establishes a persistent connection that is used to pass data between the client and server in near-realtime.   Persistent socket connections are generally based upon TCP or UDP and enable asynchronus bidirectional communication.   Binary or Text-based messages can be sent in either direction at any point in time, in any sequence, as data is available.   In HTML/JS applications you can use web sockets, which I recently discussed, or use a plugin that handles realtime socket communication. Did you also know that the next version of ColdFusion will even have web socket support built in?  In Flash/Flex/AIR, this can be achieved using the RTMP protocol (LCDS, Flash Media Server, etc…) or raw sockets (TCP or UDP).

Direct Socket Communications

In general, direct socket based communication is the most efficient means of data transfer for realtime application scenarios.  There is less back and forth handshaking and less packet encapsulation required by various protocols (HTTP, etc…), and you are restricted by fewer network protocol rules.  However, socket based communications often run on non-standard or restricted ports, so they are more likely to be blocked by IT departments or stopped by network firewalls.    If you are using socket based communication within your applications, which are running on non-standard ports, and you don’t govern the network, you may want a fallback to another realtime data implementation for failover cases.

HTTP Polling

HTTP Polling is the process of using standard HTTP requests to periodically check for data updates on the server.  The client application requests information from the server.  Generally, the client will send a timestamp indicating the last data update time.  If there is information available on the server that is newer than the timestamp, that data will be immediately sent back to the client (and the client’s timestamp will be updated).   After a period of time, another request will be made, and so forth until the polling is stopped within the application.  Using this approach, the application is more-or-less “phoning home” periodically to the server to see if there are any updates.  You can achieve near-realtime performance by setting a very short polling interval (less than one second).

Basic Data Poll Sequence

HTTP polling uses standard web protocols and ports, and generally will not be blocked by firewalls.  You can poll on top of standard HTTP (port 80) or HTTPS (port 443) without any issue.  This can be achieved by polling JSON services, XML Services, AMF, or any other data format on top of a HTTP request.   HTTP polling will generally be slower than a direct socket method, and will also utilize more network bandwidth b/c of request/response encapsulation and the periodic requests to the server.  It is also important to keep in mind that the HTTP spec only allows for 2 concurrent connections to a server at any point in time.  Polling requests can consume HTTP connections, thus slowing load time for other portions of your application.   HTTP polling can be employed in HTML/JS, Flex/Flash/AIR, desktop, server, or basically any other type of application using common libraries & APIs.

HTTP Push

HTTP Push technologies fall into 2 general categories depending upon the server-side technology/implementation.  This can refer to HTTP Streaming, where a connection is opened between the client and server and kept open using keep-alives.   As data is ready to send to the client, it will be pushed across the existing open HTTP connection.   HTTP Push can also refer to HTTP Long Polling, where the client will periodically make a HTTP request to the server, and the server will “hold” the connection open until data is available to send to the client (or a timeout occurs).   Once that request has a complete response, another request is made to open another connection to wait for more data.  Once Again, with HTTP Long Poll there should be a very short polling interval to maintain near-realtime performance, however you can expect some lag.

HTTP Long Poll Sequence

HTTP Streaming & HTTP Long polling can be employed in HTML/JS applications using the Comet approach (supported by numerous backend server technologies) and can be employed in Flex/Flash/AIR using BlazeDS or LCDS.

Collaborative Applications

Now back to the collaborative sketching application shown in the video above… the application builds off of the sketching example from previous blog posts.   I added logic to monitor the input sketches and built a HTTP poll-based monitoring service to share content between sessions that share a common ID.

Realtime Collaborative Sketches

In the JavaScript code, I created an ApplicationController class that acts as an observer to the input from the Sketcher class.   The ApplicationController encapsulates all logic handling data polling and information sharing between sessions.   When the application loads, it sets up the polling sequence.

The polling sequence is setup so that a new request will be made to the server 250MS after receiving a response from the previous request.  Note: this is very different from using a 250MS interval using setInterval.  This approach guarantees 250MS from response to the next request.  If you use a 250MS interval using setInterval, then you are only waiting 250MS between each request, without waiting for a response.  If your request takes more than 250 MS, you will can end up have stacked, or “concurrent” requests, which can cause serious performance issues.

When observing the sketch input, the start and end positions and color for each line segment get pushed into a queue of captured transactions that will be pushed to the server.  (The code supports multiple colors, even though there is no method to support changing colors in the UI.)

[js]ApplicationController.prototype.observe = function(start, end, color) {
this.capturedTransactions.push( {"sx":start.x, "sy":start.y, "ex":end.x, "ey":end.y, "c":color} );
}[/js]

When a poll happens, the captured transactions are sent to the server (a ColdFusion CFC exposed in JSON format) as a HTTP post.

[js]ApplicationController.prototype.poll = function () {
this.pendingTransactions = this.capturedTransactions;
this.capturedTransactions = [];

var data = { "method":"synchronize",
"id":this.id,
"timestamp":this.lastTimeStamp,
"transactions": JSON.stringify(this.pendingTransactions),
"returnformat":"json" };

var url = "services/DataPollGateway.cfc";
$.ajax({
type: ‘POST’,
url: url,
data:data,
success: this.getRequestSuccessFunction(),
error: this.getRequestErrorFunction()
});
}[/js]

The server then stores the pending transactions in memory (I am not persisting these, they are in-ram on the server only).   The server checks the transactions that are already in memory against the last timestamp from the client, and it will return all transactions that have taken place since that timestamp.

[cf]<cffunction name="synchronize" access="public" returntype="struct">
<cfargument name="id" type="string" required="yes">
<cfargument name="timestamp" type="string" required="yes">
<cfargument name="transactions" type="string" required="yes">

<cfscript>

var newTransactions = deserializeJSON(transactions);

if( ! structkeyexists(this, "id#id#") ){
this[ "id#id#" ] = ArrayNew(1);
}

var existingTransactions = this[ "id#id#" ];
var serializeTransactions = ArrayNew(1);
var numberTimestamp = LSParseNumber( timestamp );

//check existing tranactions to return to client
for (i = 1; i lte ArrayLen(existingTransactions); i++) {
var item = existingTransactions[i];
if ( item.timestamp GT numberTimestamp ) {
ArrayAppend( serializeTransactions, item.content );
}
}

var newTimestamp = GetTickCount();

//add new transactions to server
for (i = 1; i lte ArrayLen(newTransactions); i++) {
var item = {};

if ( structkeyexists( newTransactions[i], "clear" )) {
serializeTransactions = ArrayNew(1);
existingTransactions = ArrayNew(1);
}

item.timestamp = newTimestamp;
item.content = newTransactions[i];
ArrayAppend( existingTransactions, item );
}

var result = {};
result.transactions = serializeTransactions;

result.timestamp = newTimestamp;
this[ "id#id#" ] = existingTransactions;;

</cfscript>

<cfreturn result>
</cffunction>[/cf]

When a poll request completes, any new transactions are processed and a new poll is requested.

[js]ApplicationController.prototype.getRequestSuccessFunction = function() {
<pre> var self = this;
return function( data, textStatus, jqXHR ) {

var result = eval( "["+data+"]" );
if ( result.length > 0 )
{
var transactions = result[0].TRANSACTIONS;
self.lastTimeStamp = parseInt( result[0].TIMESTAMP );
self.processTransactions( transactions );
}

self.pendingTransactions = [];
self.requestPoll();
}
}[/js]

You can access the full client and server application source on Github at:

I used ColdFusion in this example, however the server side could be written in any server-side language… Java, PHP, .NET, etc…

If you were building this application using web sockets, you could simply push the data across the socket connection without the need for queueing.

Sketching with HTML5 Canvas and “Brush Images”

In a previous post on capturing user signatures in mobile applications, I explored how you capture user input from mouse or touch events and visualize that in a HTML5 Canvas.  Inspired by activities with my daughter, I decided to take this signature capture component and make it a bit more fun & exciting.   My daughter and I often draw and sketch together… whether its a magnetic sketching toy, doodling on the iPad, or using a crayon and a placemat at a local pizza joint, there is always something to draw. (Note: I never said I was actually good at drawing.)

Olivia & the iPad

You can take that exact same signature capture example, make the canvas bigger, and then combine it with a tablet and a stylus, and you’ve got a decent sketching application.   However, after doodling a bit you will quickly notice that your sketches leave something to be desired.   When you are drawing on a canvas using moveTo(x,y) and lineTo(x,y), you are somewhat limited in what you can do. You have lines which can have consisten thickness, color, and opacity. You can adjust these, however in the end, they are only lines.

If you switch your approach away from moveTo and lineTo, then things can get interesting with a minimal amount of changes. You can use images to create “brushes” for drawing strokes in a HTML5 canvas element and add a lot of style and depth to your sketched content.  This is an approach that I’ve adapted to JavaScript from some OpenGL drawing applications that I’ve worked on in the past.  Take a look at the video below to get an idea what I mean.

Examining the sketches side by side, it is easy to see the difference that this makes.   The variances in stroke thickness, opacity & angle add depth and style, and provide the appearance of drawing with a magic marker.

Sketches Side By Side

It’s hard to see the subtleties in this image, so feel free to try out the apps on your own using an iPad or in a HTML5 Canvas-capable browser:

Just click/touch and drag in the gray rectangle area to start drawing.

Now, let’s examine how it all works.   Both approaches use basic drawing techniques within the HTML5 Canvas element.   If you aren’t familiar with the HTML5 Canvas, you can quickly get up to speed from the tutorials from Mozilla.

moveTo, lineTo

The first technique uses the canvas’s drawing context moveTo(x,y) and lineTo(x,y) to draw line segments that correspond to the mouse/touch coordinates.   Think of this as playing “connect the dots” and drawing a solid line between two points.

The code for this approach will look something like the following:

[js]var canvas = document.getElementById(‘canvas’);
var context = canvas.getContext(‘2d’);

context.beginPath();
context.moveTo(a.x, a.y);
context.lineTo(b.x, b.y);
context.lineTo(c.x, c.y);
context.closePath();
context.stroke();[/js]

The sample output will be a line from point A, to point B, to point C:

lineTo(x,y) Stroke Sample

Brush Images

The technique for using brush images is identical in concept to the previous example – you are drawing a line from point A to point B.  However, rather than using the built-in drawing APIs, you are programmatically repeating an image (the brush) from point A to point B.

First, take a look at the brush image shown below at 400% of the actual scale.  It is a simple image that is a diagonal shape that is thicker and more opaque on the left side.   By itself, this will just be a mark on the canvas.

Brush Image (400% scale)

When you repeat this image from point A to point B, you will get a “solid” line.  However the opacity and thickness will vary depending upon the angle of the stroke.   Take a look at the sample below (approximated, and zoomed).

Brush Stroke Sample (simulated)

The question is… how do you actually do this in JavaScript code?

First, create an Image instance to be used as the brush source.

[js]brush = new Image();
brush.src = ‘assets/brush2.png’;[/js]

Once the image is loaded, the image can be drawn into the canvas’ context using the drawImage() function. The trick here is that you will need to use some trigonometry to determine how to repeat the image. In this case, you can calculate the angle and distance from the start point to the end point. Then, repeat the image based on that distance and angle.

[js]var canvas = document.getElementById(‘canvas’);
var context = canvas.getContext(‘2d’);

var halfBrushW = brush.width/2;
var halfBrushH = brush.height/2;

var start = { x:0, y:0 };
var end = { x:200, y:200 };

var distance = parseInt( Trig.distanceBetween2Points( start, end ) );
var angle = Trig.angleBetween2Points( start, end );

var x,y;

for ( var z=0; (z<=distance || z==0); z++ ) {
x = start.x + (Math.sin(angle) * z) – halfBrushW;
y = start.y + (Math.cos(angle) * z) – halfBrushH;
context.drawImage(this.brush, x, y);
}[/js]

For the trigonometry functions, I have a simple utility class to calculate the distance between two points, and the angle between two points. This is all based upon the good old Pythagorean theorem.

[js]var Trig = {
distanceBetween2Points: function ( point1, point2 ) {

var dx = point2.x – point1.x;
var dy = point2.y – point1.y;
return Math.sqrt( Math.pow( dx, 2 ) + Math.pow( dy, 2 ) );
},

angleBetween2Points: function ( point1, point2 ) {

var dx = point2.x – point1.x;
var dy = point2.y – point1.y;
return Math.atan2( dx, dy );
}
}[/js]

The full source for both of these examples is available on github at:

This example uses the twitter bootstrap UI framework, jQuery, and Modernizr.  Both the lineTo.html and brush.html apps use the exact same code, which just uses a separate rendering function based upon the use case.    Feel free to try out the apps on your own using an iPad or in a HTML5 Canvas-capable browser:

Just click/touch and drag in the gray rectangle area to start drawing.

Stylistic Sketchy
Stylistic Sketchy - Click to Get Started

Flex Accepted by Apache Software Foundation

In case you did not see the post on the Flex Team blog on Dec. 31st, it was announced that Flex has officially been accepted by the Apache Software Foundation.   You can view the Apache Flex proposal on the Apache incubator wiki at http://wiki.apache.org/incubator/FlexProposal.

Apache Flex allows developers to target a variety of platforms, initially Apple iOS, Google Android, RIM BlackBerry, Microsoft Windows, and Mac OS X with a single codebase. Flex provides a compiler, skinnable user-interface components and managers to handle styling, skinning, layout, localization, animation, module-loading and user interaction management.

Just a bit of extra detail for you, all of which is available through Apache:

Initial goals of the Apache Flex incubation project:

  • Donate all Adobe Flex SDK source code and documentation to the Apache Software Foundation.
  • Setup and standardize the open governance of the Apache Flex project.
  • Rename all assets from Adobe Flex SDK to Apache Flex in project source code, docs, tests and related infrastructure.

Interesting App Store Statistics

Here are some interesting and quite surprising statistics for the US Census Browser HTML/PhoneGap showcase application that I released in December, which I wanted to share. The app is a browser for US Census data, full detail available here: http://www.tricedesigns.com/2010-census/. The Census Browser application was intended as a showcase app for enterprise-class data visualization in HTML-based applications, and all source code is freely available to the public.

What is really surprising is the “health” of my app within the given ecosystems. I offered the app as a free download in each market. The app is focused on Census data, so there is obviously not a ton of consumer demand, however the data is still interesting to play around with. I would not expect the same results for all types of apps in all markets.

Here are a few observations from the data:

  • Barnes & Noble Nook downloads far exceeded all other markets combined (69% of all downloads)
  • BlackBerry Playbook downloads were in 3rd, just behind iOS (BB is 11% of all downloads)
  • Android traffic was minimal (2% of all downloads)

The general public perception/assumption that I encounter is that the iOS market is strongest, followed by Android, and that BB is dead. These numbers show a conflicting reality. Barnes & Noble was the strongest, with iOS in second place, and BlackBerry just behind iOS.

Here is the full data for downloads in December:

Market Release Date # Downloads Link Notes
iOS 12/4/11 1151 link (iPad only)
Android (Google) 12/6/11 58 link (large-xlarge screens only)
Android (Amazon) 12/6/11 63 link (includes Kindle Fire)
BlackBerry 12/14/11 752 link (PlayBook only)
Barnes & Noble 12/20/11 4508 link (Nook)

Other Observations

Here are a few other observations from analyzing the download statistics for the various app markets…

Lots of people got Nook devices for Christmas this year:

BlackBerry Playbook downloads spiked from the BerryReview.com app review:

iOS traffic peaked just after the inital release with an increase after the winter holidays, but has been more-or-less consistent with no “spike”:

Amazon Market only had 8 downloads on Christmas day – this is likely the result of the fact that the Kindle Fire is branded as a consumer media device, not an analytics/computing device:

Know what else is interesting?   The charting/analytics for Amazon, Google, and Nook markets are all built with Adobe Flash, with both Amazon and Nook built using Adobe Flex.