Capturing User Signatures in Mobile Applications

One growing trend that I have seen in mobile & tablet applications is the creation of tools that enable your workforce to perform their job better. This can be in the case of mobile data retrieval, streamlined sales process with apps for door-to-door sales, mobile business process efficiency, etc…

One of the topics that comes up is how do you capture a signature and store it within your application? This might be for validation that the signer is who they say they are, or for legal/contractual reasons. Imagine a few scenarios:

  • Your cable TV can’t be installed until you sign the digital form on the installation tech’s tablet device
  • You agree to purchase a service from a sales person (door to door, or in-store kiosk) – your signature is required to make this legally binding.
  • Your signature is required to accept an agreement before confidential data is presented to you.

These are just a few random scenarios, I’m sure there are many more.   In this post, I will focus on 2 (yes, I said two) cross-platform solutions to handle this task – one built with Adobe Flex & AIR, and one built with HTML5 Canvas & PhoneGap.  

Source for both solutions is available at: https://github.com/triceam/Mobile-Signature-Capture

Watch the video below to see this in action, then we’ll dig into the code that makes it work.

The basic flow of the application is that you enter an email address, sign the interface, then click the green “check” button to submit to the signature to a ColdFusion server.  The server then sends a multi-part email to the email address that you provided, containing text elements as well as the signature that was just captured.

If you’d like to jump straight to specific code portions, use the links below:



The Server Solution

Let’s first examine the server component of the sample application.   The server side is powered by ColdFusion. There’s just a single CFC that is utilized by both the Flex/AIR and HTML/PhoneGap front-end applications.   The CFC exposes a single service that accepts two parameters: the email address, and a base-64 encoded string of the captured image data.

<cffunction name="submitSignature" access="remote" returntype="boolean">
    <cfargument name="email" type="string" required="yes">
    <cfargument name="signature" type="string" required="yes">
    
    <cfmail SUBJECT ="Signature"
        FROM="#noReplyAddress#"
        TO="#email#"
        username="#emailLoginUsername#"
        password="#emailLoginPassword#"
        server="#mailServer#" 
        type="HTML" >
        
        <p>This completes the form transaction for <strong>#email#</strong>.</p>
        
        <p>You may view your signature below:</p>
        <p><img src="cid:signature" /></p>
        
        <p>Thank you for your participation.</p>
        
        <cfmailparam
            file="signature"
            content="#toBinary( signature )#"
            contentid="signature"
            disposition="inline" />
    
    </cfmail>

    <cfreturn true />
</cffunction>

Note: I used base-64 encoded image data so that it can be a single server component for both user interfaces. In Flex/AIR you can also serialize the data as a binary byte array, however binary serialization isn’t quite as easy with HTML/JS… read on to learn more.


The Flex/AIR Solution

The main user interface for the Flex/AIR solution is a simple UI with some form elements. In that UI there is an instance of my SignatureCapture user interface component. This is a basic component that is built on top of UIComponent (the base class for all Flex visual components), which encapsulates all logic for capturing the user signature. The component captures input based on mouse events (single touch events are handled as mouse events in air). The mouse input is then used to manipulate the graphics content of the component using the drawing API. I like to think of the drawing API as a language around the childhood game “connect the dots”. In this case, you are just drawing lines from one point to another.

When the form is submitted, the graphical content is converted to a base-64 encoded string using the Flex ImageSnapshot class/API, before passing it to the server.

You can check out a browser-based Flex version of this in action at http://tricedesigns.com/portfolio/sigCaptureFlex/ – Just enter a valid email address and use your mouse to sign within the signature area. When this is submitted, it will send an email to you containing the signature.

You can check out the SignatureCapture component code below, or check out the full project at https://github.com/triceam/Mobile-Signature-Capture/tree/master/flex%20client. This class will also work in desktop AIR or browser/based Flex applications. The main application workflow and UI is contained with Main.mxml.

package
{
	import flash.display.DisplayObject;
	import flash.display.Graphics;
	import flash.display.Sprite;
	import flash.events.MouseEvent;
	import flash.geom.Point;
	
	import mx.core.UIComponent;
	import mx.graphics.ImageSnapshot;
	import mx.managers.IFocusManagerComponent;
	
	import spark.primitives.Graphic;
	
	public class SignatureCapture extends UIComponent
	{
		private var captureMask : Sprite;
		private var drawSurface : UIComponent;
		private var lastMousePosition : Point;
		
		private var backgroundColor : int = 0xEEEEEE;
		private var borderColor : int = 0x888888;
		private var borderSize : int = 2;
		private var cornerRadius :int = 25;
		private var strokeColor : int = 0;
		private var strokeSize : int = 2;
		
		public function SignatureCapture()
		{
			lastMousePosition = new Point();
			super();
		}
		
		override protected function createChildren():void 
		{
			super.createChildren();
			
			captureMask = new Sprite();
			drawSurface = new UIComponent();
			this.mask = captureMask;
			addChild( drawSurface );
			addChild( captureMask );
			
			this.addEventListener( MouseEvent.MOUSE_DOWN, onMouseDown );
		}
		
		protected function onMouseDown( event : MouseEvent ) : void
		{
			lastMousePosition = globalToLocal( new Point( stage.mouseX, stage.mouseY ) );
			stage.addEventListener( MouseEvent.MOUSE_MOVE, onMouseMove );
			stage.addEventListener( MouseEvent.MOUSE_UP, onMouseUp );
		}
		
		protected function onMouseMove( event : MouseEvent ) : void
		{
			updateSegment();
		}
		
		protected function onMouseUp( event : MouseEvent ) : void
		{
			updateSegment();
			stage.removeEventListener( MouseEvent.MOUSE_MOVE, onMouseMove );
			stage.removeEventListener( MouseEvent.MOUSE_UP, onMouseUp );
		}
		
		protected function updateSegment() : void
		{
			var nextMousePosition : Point = globalToLocal( new Point( stage.mouseX, stage.mouseY ) );
			renderSegment( lastMousePosition, nextMousePosition );
			lastMousePosition = nextMousePosition;
		}
		
		
		public function clear() : void
		{
			drawSurface.graphics.clear();
		}
		
		override public function toString() : String
		{
			var snapshot : ImageSnapshot = ImageSnapshot.captureImage( drawSurface );
			return ImageSnapshot.encodeImageAsBase64( snapshot );
		}
		
		override protected function updateDisplayList(w:Number, h:Number):void
		{
			super.updateDisplayList(w,h);
			
			drawSurface.width = w;
			drawSurface.height = h;
			
			var g : Graphics = this.graphics;
			
			//draw rectangle for mouse hit area
			g.clear();
			g.lineStyle( borderSize, borderColor, 1, true );
			g.beginFill( backgroundColor, 1 );
			g.drawRoundRect( 0,0,w,h, cornerRadius, cornerRadius );
		
			
			//fill mask
			g.clear();
			g = captureMask.graphics;
			g.beginFill( 0, 1 );
			g.drawRoundRect( 0,0,w,h, cornerRadius, cornerRadius );
		}
		
		protected function renderSegment( from : Point, to : Point ) : void
		{
			var g : Graphics = drawSurface.graphics;
			g.lineStyle( strokeSize, strokeColor, 1 );
			g.moveTo( from.x, from.y );
			g.lineTo( to.x, to.y );
		}
	}
}



The HTML5/PhoneGap Solution

The main user interface for the HTML5/PhoneGap solution is also a simple UI with some form elements. In that UI there is a Canvas element that is used to render the signature. I created a SignatureCapture JavaScript class that encapsulates all logic for capturing the user signature. In browsers that support touch events (mobile browsers), this is based on the touchstart, touchmove and touchend events. In browsers that don’t support touch (aka desktop browsers), the signature input is based on mousedown, mousemove and mouseup events. The component captures input based on touch or mouse events, and that input is used to manipulate the graphics content of the Canvas tag instance. The canvas tag also supports a drawing API that is similar to the ActionScript drawing API. To read up on Canvas programmatic drawing basics, check out the tutorials at http://www.adobe.com/devnet/html5/html5-canvas.html

When the form is submitted, the graphical content is converted to a base-64 encoded string using the Canvas’s toDataURL() method. The toDataURL() method returns a base-64 encoded string value of the image content, prefixed with “data:image/png,”. Since I’ll be passing this back to the server, I don’t need this prefix, so it is stripped, then sent to the server for content within the email.

You can check out a browser-based version of this using the HTML5 Canvas in action at http://tricedesigns.com/portfolio/sigCapture/ – Again, just enter a valid email address and use your mouse to sign within the signature area. When this is submitted, it will send an email to you containing the signature. However, this example requires that your browser supports the HTML5 Canvas tag.

You can check out the SignatureCapture code below, or check out the full project at https://github.com/triceam/Mobile-Signature-Capture/tree/master/html%20client. This class will also work in desktop browser applications that support the HTML5 canvas. I used Modernizr to determine whether touch events are supported within the client container (PhoneGap or desktop browser). The main application workflow is within application.js.

Also a note for Android users, the Canvas toDataURL() method does not work in Android versions earlier than 3.0. However, you can implement your own toDataURL() method for use in older OS versions using the technique in this link: http://jimdoescode.blogspot.com/2011/11/trials-and-tribulations-with-html5.html (I did not update this example to support older Android OS versions.)

function SignatureCapture( canvasID ) {
	this.touchSupported = Modernizr.touch;
	this.canvasID = canvasID;
	this.canvas = $("#"+canvasID);
	this.context = this.canvas.get(0).getContext("2d");	
	this.context.strokeStyle = "#000000";
	this.context.lineWidth = 1;
	this.lastMousePoint = {x:0, y:0};
	
	this.canvas[0].width = this.canvas.parent().innerWidth();
    
	if (this.touchSupported) {
		this.mouseDownEvent = "touchstart";
		this.mouseMoveEvent = "touchmove";
		this.mouseUpEvent = "touchend";
	}
	else {
		this.mouseDownEvent = "mousedown";
		this.mouseMoveEvent = "mousemove";
		this.mouseUpEvent = "mouseup";
	}
	
	this.canvas.bind( this.mouseDownEvent, this.onCanvasMouseDown() );
}

SignatureCapture.prototype.onCanvasMouseDown = function () {
	var self = this;
	return function(event) {
		self.mouseMoveHandler = self.onCanvasMouseMove()
		self.mouseUpHandler = self.onCanvasMouseUp()

		$(document).bind( self.mouseMoveEvent, self.mouseMoveHandler );
		$(document).bind( self.mouseUpEvent, self.mouseUpHandler );
		
		self.updateMousePosition( event );
		self.updateCanvas( event );
	}
}

SignatureCapture.prototype.onCanvasMouseMove = function () {
	var self = this;
	return function(event) {

		self.updateCanvas( event );
     	event.preventDefault();
    	return false;
	}
}

SignatureCapture.prototype.onCanvasMouseUp = function (event) {
	var self = this;
	return function(event) {

		$(document).unbind( self.mouseMoveEvent, self.mouseMoveHandler );
		$(document).unbind( self.mouseUpEvent, self.mouseUpHandler );
		
		self.mouseMoveHandler = null;
		self.mouseUpHandler = null;
	}
}

SignatureCapture.prototype.updateMousePosition = function (event) {
 	var target;
	if (this.touchSupported) {
		target = event.originalEvent.touches[0]
	}
	else {
		target = event;
	}

	var offset = this.canvas.offset();
	this.lastMousePoint.x = target.pageX - offset.left;
	this.lastMousePoint.y = target.pageY - offset.top;

}

SignatureCapture.prototype.updateCanvas = function (event) {

	this.context.beginPath();
	this.context.moveTo( this.lastMousePoint.x, this.lastMousePoint.y );
	this.updateMousePosition( event );
	this.context.lineTo( this.lastMousePoint.x, this.lastMousePoint.y );
	this.context.stroke();
}

SignatureCapture.prototype.toString = function () {

	var dataString = this.canvas.get(0).toDataURL("image/png");
	var index = dataString.indexOf( "," )+1;
	dataString = dataString.substring( index );
	
	return dataString;
}

SignatureCapture.prototype.clear = function () {

	var c = this.canvas[0];
	this.context.clearRect( 0, 0, c.width, c.height );
}

Source for the ColdFusion server, as well as Flex/AIR and HTML5/PhoneGap clients is available at: https://github.com/triceam/Mobile-Signature-Capture

Toying with Realtime Data & Web Sockets

Recently I was acting as a “second set of eyes” to help out fellow Adobe Evangelist Kevin Hoyt track down a quirk with a websockets example that he was putting together. Kevin has a great writeup to familiarize yourself with web sockets & streaming communication that I highly recommend checking out.

While working with Kevin’s code, I started tinkering… “what if I change this, what if I tweak that?” Next thing you know, I put together a sample scenario showing subscription-based realtime data streaming to multiple web clients using web sockets. Check out the video below to see it in action.

You are seeing 9 separate browser instances getting realtime push-based updates from a local server using web sockets. When the browser loads, the html-based client makes a web socket connection, then requests all symbols from the server. The server then sends the stock symbol definitions back to the client and displays them within the HTML user interface. From there, the user can click on a stock symbol to subscribe to updates for that particular symbol. DISCLAIMER: All that data is randomly generated!

I put together this example for experimentation, but also to highlight a few technical scenarios for HTML-based applications. Specifically:

  • Realtime/push data in HTML-based apps
  • Per-client subscriptions for realtime data
  • Multi-series realtime data visualization in HTML-based apps

The server is an AIR app started by Kevin, based on the web sockets draft protocol. It is written in JavaScript, and the client is a HTML page to be viewed in the browser.

If you don’t feel like reading the full web sockets protocol reference, you can get a great overview from websocket.org or Wikipedia.

One thing to keep in mind is that web sockets are not widely supported in all browsers yet. There is a great reference matrix for web socket support from caniuse.com:

If you still aren’t sure if your browser supports web sockets, you can also check simply by visiting websocketstest.com/. If you want to test for web socket support within your own applications, you can easily check for support using Modernizr. Note: I didn’t add the Modernizr test in this example… I only tested in Chrome on OSX.

OK, now back to the sample application. All of the source code for this example is available on github at: https://github.com/triceam/Websocket-Streaming-Example.  To run it yourself, you first have to launch the server. You can do this on the command line by invoking ADL (part of the AIR SDK):

cd "/Applications/Adobe Flash Builder 4.6/sdks/4.6.0/bin"
./adl ~/Documents/dev/Websocket-Streaming-Example/server/application.xml

You’ll know the server is started b/c an air window will popup (you can ignore this, just don’t close it), and you will start seeing feed updates in the console output.

Once the server is running, open “client/client.html” in your browser. It will connect to the local server, and then request the list of symbols. If you click on a symbol, it will subscribe to that feed. Just click on the symbol name again to unsubscribe. You’ll know the feed is subscribed b/c the symbol will show up in a color (matching the corresponding feed on the chart). Again, let me reiterate that I only tested this in Chrome.

You can open up numerous client instances, and all will receive the same updates in real time for each subscribed stock symbol.

The “meat” of code for the server starts in server/scripts/server/server.js. Basically, the server loads a configuration file for the socket server, then creates a ConnectionManager and DataFeed (both of these are custom JS classes). The ConnectionManager class encapsulates all logic around socket connections. This includes managing the ServerSocket as well as all client socket instances and events. The DataFeed class handles data within the app. First, it generates random data, then sets up an interval to generate random data updates. For every data update, the ConnectionManager instance’s dispatch() method is invoked to send updates to all subscribed clients. Rather than trying to put lots of code snippets inline in this post (which would just be more confusing), check out the full source at: https://github.com/triceam/Websocket-Streaming-Example/tree/master/server

The client code all starts in client.html, with the application logic inside of client/scripts/client.js. Once the client interface loads, it connects to the web socket and adds the appropriate event handlers. Once subscribed to a data feed, realtime data will be returned via the web socket instance, transformed slightly to fit the data visualization structure, then rendered in an HTML canvas using the RGraph data visualization library. RGraph is free to get started with, however if you want to deploy a production app with it, you’ll need a license. You’ll notice that each feed updates independently, based upon the client subscriptions. Note: The data visualization is not temporally aligned… if you want the updates in time-sequence, there is a litte bit more work involved in the client-side data transformation.

Again, rather than trying to put lots of confusing code snippets inline in this post, check out the full client side source at: https://github.com/triceam/Websocket-Streaming-Example/tree/master/client

This example is intended to get your minds rolling with the concepts; it is not *yet* an all-encompassing enterprise solution. You can expect to see a few more data push scenarios here in the near future, based on different enterprise server technologies.

Enjoy!

Introducing the US Census Browser Application

I’d like to take this opportunity to introduce you to a new project I’ve been working on to showcase enterprise-class data visualization in HTML-based applications.   The US Census Browser is an open source application for browsing data from the 2010 US Census.

The app is completely written using HTML and JavaScript, even for the charting/data visualization components. You can check it out in several application ecosystems today:

Apple iTunes: http://itunes.apple.com/us/app/census-browser/id483201717
Google Android: https://market.android.com/details?id=com.tricedesigns.CensusBrowser
BlackBerry App World: http://appworld.blackberry.com/webstore/content/69236?lang=en
Amazon App Store: http://www.amazon.com/Andrew-Trice-US-Census-Browser/dp/B006JDATOY/ref=sr_1_1?ie=UTF8&s=mobile-apps&qid=1323874245&sr=1-1 (this includes support for Kindle Fire)

Support for additional platforms is planned for future development. Future targets include BlackBerry Playbook as well as Android 2.x devices, including the Amazon Kindle Fire and Barnes & Noble Nook Color – Android 2.x does not support SVG graphics in-browser, so I am working on some alternative features.

Update: Kindle Fire and Playbook have been approved, and are now supported. See links above.

You can also view the US Census Browser application in your desktop or mobile browser at: http://tricedesigns.com/census/

Please keep in mind that this application was designed for mobile devices.  Internet Explorer in particular does not work well with the Census Browser – use at your own risk.   The browser-based application has been tested and works properly in the latest versions of Chrome, Safari, Firefox, and Opera.   The US Census Browser application also does not work in Android 2.x and below, due to the fact that these versions of Android do not support SVG graphics in the mobile browser.

Full application source code for the HTML/JS interface and ColdFusion backend system are available at https://github.com/triceam/US-Census-Browser under the terms of the “Modified BSD License”. Be sure to review the README if you want to get this running on your own.

APPLICATION OVERVIEW
The application is essentially a single-page web site, which asynchronously loads data from the backend upon request, and displays that data to the user. The main application file is index.html, which loads the UI and appropriate client-side scripts. The main presentation logic is applied via CSS stylesheets, and the application control is handled by the ApplicationController class, inside of application.js. The ApplicationController class handles state changes within the application and updates the UI accordingly. The main data visualization and data formatting logic is all contained within the censusVisualizer class, which the ApplicationController class uses to render content. All DOM manipulation, event handling, and AJAX requests are performed using jQuery.

The data visualization is implemented 100% client-side, using the Highcharts JavaScript library. Highcharts renders vector graphics client-side, based upon the data that is passed into it. Check out the examples at: http://www.highcharts.com/demo/ for a sample of what it is capable of.

The fluid scrolling and swiping between views is implemented using the iScroll JavaScript library. Note: I’m using iScroll-lite.js. This is a great resource for any HTML-mobile, or mobile-web applications.

PHONEGAP USAGE
The client-side runtime does not have any dependencies for access to device-specific functionality. However, PhoneGap is being used as an application container so that the application can be distributed through various mobile “app stores”.

SERVER-SIDE
The back-end of this application is written using ColdFusion. Yep, that’s right. I used CF. In fact, the server side is ridiculously simple. It is only a single ColdFusion Component (CFC), with three remotely exposed methods for accessing data, and relies upon CF’s built in functionality to serialize JSON. CF is incredibly powerful, and made this project very simple and quick to develop.

Feel free to check it out on github: https://github.com/triceam/US-Census-Browser
You can also check out more technical details at: http://www.tricedesigns.com/2010-census/

Preview: New HTML5/PhoneGap Project

Here’s a quick preview of my new HTML5/PhoneGap data vizualization app. Once released, this will be available on multiple platforms, in multiple app stores, AND it will be completely open source.  Just waiting on app store approvals…

You can expect a full writeup on how this was built after it is released.

Enjoy!


UPDATE:

This application is now available in iOS and Android markets, and full source code is available. See details at http://www.tricedesigns.com/2011/12/05/introducing-the-us-census-browser-application/