Tag Archives: HTML5

Say What? Live video chat between iOS & WebRTC with Twilio & IBM Watson Cognitive Computing in Real Time

What I’m about to show you might seem like science fiction from the future, but I can assure you it is not. Actually, every piece of this is available for you to use as a service.  Today.

Yesterday Twilio, an IBM partner whose services are available via IBM Bluemix, announced several new SDKs, including live video chat as a service.  This makes live video very easy to integrate into your native mobile or web based applications, and gives you the power to do some very cool things. For example, what if you could add video chat capabilities between your mobile and web clients? Now, what if you could take things a step further, and add IBM Watson cognitive computing capabilities to add real-time transcription and analysis?

Check out this video from yesterday’s Twilio Signal conference keynote, where fellow IBM’ers Damion Heredia and Jeff Sloyer demonstrate exactly this scenario; the integration of the new Twilio video SDK between iOS native and WebRTC client with IBM Watson cognitive computing services providing realtime transcription and sentiment analysis.

If it doesn’t automatically jump to the IBM Bluemix Demo, skip ahead to 2 hours, 15 min, and 20 seconds.

Jeff and Damion did an awesome job showing of both the new video service and the power of IBM Watson. I can also say first-hand that the new Twilio video services are pretty easy to integrate into your own projects (I helped them integrate these services into the native iOS client (physician’s app) shown in the demo)!  You just pull in the SDK, add your app tokens, and instantiate a video chat.   Jeff is pulling the audio stream from the WebRTC client and pushing it up to Watson in real time for the transcription and sentiment analysis services.

Data Management for Apps that Work as Well Offline as They Do Online

Earlier this week I had the privilege of speaking at ApacheCon in Austin, TX on the topic of data management for apps that work as well offline as they do online.  This is an important topic for mobile apps, since, as we all painfully know already, there is never a case when you are always online on your mobile devices.  There always ends up being a time when you need your device/app, but you can’t get online to get the information you need.  Well, this doesn’t always have to be the case. There are strategies you can employ to build apps that work just as well offline as they do online, and the strategy I’d like to highlight today is based upon data management using the IBM Cloudant NoSQL database as a service, which is based upon Apache CouchDB.

Here’s a link to the presentation slides (built using reveal.js) – just use the space bar to advance the presentation slides:

The “couch” in CouchDB is actually an acronym for Cluster of Unreliable Commodity Hardware. At the core of this cluster is the concept of replication, which in the most basic of terms means that  data is shared between multiple sources.  Replication is used to share information between nodes of the cluster, which provides for cluster reliability and fault tolerance.

Replication between Nodes
Replication between Nodes (source)

If you’d like to learn more about replication in Cloudant and CouchDB, you can read more using the links below:

Cloudant is a clustered NoSQL database services that provides an extremely powerful and searchable data store.  It is designed to power the web and mobile apps, and all information is exposed via REST services. Since the IBM Cloudant service is based on CouchDB (and not so coincidentally, IBM is a major contributor to the CouchDB project), replication is also core the the Cloudant service.

With replication, you only have to write your data/changes to a single node in the cluster, and replication takes care of propagating these changes across the cluster.

If you are building apps for the web or mobile, there are options to extend the data replication locally either on the device or in the browser.   This means that you can have a local data store that automatically pushes and/or pulls data from the remote store using replication, and it can be done either via native languages, or using JavaScript.

If you want to have local replication in either a web or hybrid (Cordova/PhoneGap) app, you can use PouchDB.  PouchDB is a local JavaScript database modeled after CouchDB and implements that CouchDB replication API.  So, you can store your data in the browser’s local storage, and those changes will automatically be replicated to the remote Cloudant store.  This works in the browser, in a hybrid (web view) app, or even inside of a Node.js instance. Granted, if you’re in-browser you’ll need to leverage the HTML5 cache to have your app cached locally.

If you are building a native app, don’t worry, you can take advantage of the Cloudant Sync API to leverage the local data store with replication.  This is available for iOS and Android, and implements the CouchDB replication API.

The sample app that I showed in the presentation is a native iOS application based on the GeoPix MobileFirst sample app that I detailed in a previous post.  The difference is that in this case I showed it using the Cloudant Sync API, instead of the MobileFirst data wrapper classes, even though it was pointing at the exact same Cloudant database instance.  You can see a video of the app in action below.

All that you have to do is create a local data store instance, and then use replication to synchronize data between the local store and a remote store.

Replication be either one-way (push or pull), or two-way.  So, any changes between the local and remote stores are replicated across the cluster.  Essentially, the local data store just becomes a node in the cluster.  This provides complete access to the local data, even if there is no network available.  Just save your data to the local store, and replication takes care of the rest.

In the native Objective-C code, you just need to setup the CDTDatastore manager, and initialize your datastore instance.

self.manager = [[CDTDatastoreManager alloc] initWithDirectory:path error:nil];
self.datastore = [self.manager datastoreNamed:@"geopix" error:nil];

Once your datastore is created, you can read/write/modify any data in the local store.  In this case I am creating a generic data object (basically  like a JSON object), and creating a document containing this data.  A document is a record within the data store.

You can add attachments to the document or modify the document as your app needs.  In the code below, I add a JPG atttachment to the document.

//create a document revision
CDTMutableDocumentRevision *rev = [CDTMutableDocumentRevision revision];
rev.body = @{
			 @"sort": [NSNumber numberWithDouble:[now timeIntervalSince1970]],
			 @"clientDate": dateString,
			 @"latitude": [NSNumber numberWithFloat:location.coordinate.latitude],
			 @"longitude": [NSNumber numberWithFloat:location.coordinate.longitude],
			 @"altitude": [NSNumber numberWithFloat:location.altitude],
			 @"course": [NSNumber numberWithFloat:location.course],
			 @"type": @"com.geopix.entry"
			 };

//add the jpg attachment
NSData *imageData = UIImageJPEGRepresentation(image, 0.1);
[imageData writeToFile:imagePath atomically:YES];
 
CDTUnsavedFileAttachment *att1 = [[CDTUnsavedFileAttachment alloc]
								  initWithPath:imagePath
								  name:imageName
								  type:@"image/jpeg"];

rev.attachments = @{ imageName: att1 };

//create a new document from the revision
NSError *error = nil;
CDTDocumentRevision *doc = [self.datastore createDocumentFromRevision:rev error:&error];

if (doc == nil) {
	[logger logErrorWithMessages:@"Error creating document: %@", error.localizedDescription];
}

[logger logDebugWithMessages:@"Document created ID: %@", doc.docId];

Replication is a fire-and-forget process.  You simply need to initialize the replication process, and any changes to the local data store will be replicated to the remote store automatically when the device is online.

//initialize the replicator factory with the local data store manager
CDTReplicatorFactory *replicatorFactory = 
	[[CDTReplicatorFactory alloc] initWithDatastoreManager:self.manager];

NSURL *remoteDatabaseURL = [NSURL URLWithString:REMOTE_DATABASE_URL];

//setup push replication for local->remote changes
NSError *error = nil;
CDTPushReplication *pushReplication = 
	[CDTPushReplication replicationWithSource:self.datastore target:remoteDatabaseURL];

//create the replicator instance
self.replicator = [replicatorFactory oneWay:pushReplication error:&error];
if (!self.replicator) {
	[logger logErrorWithMessages:@"An error occurred: %@", error.localizedDescription];
}

//assign the replicator delegate
self.replicator.delegate = self;

//auto start replication
error = nil;
if (![self.replicator startWithError:&error]) {
	[logger logErrorWithMessages:@"An error occurred: %@", error.localizedDescription];
}

By assigning a replicator delegate class (as shown above), your app can monitor and respond to changes in replication state.  For example, you can update status if replication is in progress, complete, or if an error condition was encountered.

- (void)replicatorDidChangeState:(CDTReplicator *)replicator {
    [logger logDebugWithMessages:@"Replicator changed State: %@", [CDTReplicator stringForReplicatorState:replicator.state]];
}

- (void)replicatorDidChangeProgress:(CDTReplicator *)replicator {
    [logger logDebugWithMessages:@"Replicator progress: %d/%d", replicator.changesProcessed, replicator.changesTotal];
    
    NSDictionary *userInfo = @{ @"status":[NSString stringWithFormat:@"%d/%d", replicator.changesProcessed, replicator.changesTotal] };
    
    [[NSNotificationCenter defaultCenter]
     postNotificationName:@"ReplicationStatus"
     object:self
     userInfo:userInfo];
}

- (void)replicatorDidError:(CDTReplicator *)replicator info:(NSError *)info {
    [logger logErrorWithMessages:@"An error occurred: %@", info.localizedDescription];
    self.replicator = nil;
    
    [[NSNotificationCenter defaultCenter]
     postNotificationName:@"ReplicationError"
     object:self];
}

- (void)replicatorDidComplete:(CDTReplicator *)replicator {
    [logger logDebugWithMessages:@"Replication completed"];
    self.replicator = nil;
    
    [[NSNotificationCenter defaultCenter]
     postNotificationName:@"ReplicationComplete"
     object:self];
}

If you want to access data from the local store, it is always available within the app, regardless of whether or not the device has an active internet connection.  For example, this method will return all documents within the local data store.

-(NSArray*) getLocalData {
    
    NSArray *docs = [self.datastore getAllDocuments];
    return docs;
}

Be sure to review the documentation and/or Cloudant Synch API source code for complete details.

Helpful Links

IBM Watson QA + Speech Recognition + Speech Synthesis = A Conversation With Your Computer

Back in November I released a demo application here on my blog showing the IBM Watson QA Service for cognitive/natural language computing connected to the Web Speech API in Google Chrome to have real conversational interaction with a web application.  It’s a nice demo, but it always drove me nuts that it only worked in Chrome.  Last month the IBM Watson team released 5 new services, and guess what… Speech Recognition and Speech Synthesis are included!

These two services enable you to quickly add Text-To-Speech or Speech-To-Text capability to any application.  What’s a better way to show them off than by updating my existing app to leverage the new speech services?

So here it is: watsonhealthqa.mybluemix.net!

By leveraging the Watson services it can now run in any browser that supports getUserMedia (for speech recognition) and HTML5 <Audio> (for speech playback).

(Full source code available at the bottom of this post)

You can check out a video of it in action below:

If your browser doesn’t support the getUserMedia API or HTML5 <Audio>, then your mileage may vary.  You can check where these features are supported with these links: <Audio>getUserMedia

Warning: This is targeting desktop browsers – HTML5 Audio is a mess on mobile devices due to limited codec support and immature APIs.

So how does this all work?

Just like the QA service, the new Text To Speech and Speech To Text services are now available in IBM Bluemix, so you can create a new application that leverages any of these services, or you can add them to any existing application.

I simply added the Text To Speech and Speech To Text services to my existing Healthcare QA application that runs on Bluemix:

bluemix-dashboard
IBM Bluemix Dashboard

 

These services are available via a REST API. Once you’ve added them to your application, you can consume them easily within any of your applications.

I updated the code from my previous example in 2 ways: 1) take advantage of the Watson Node.js Wrapper that makes interacting with Watson a lot easier and 2) to take advantage of these new services services.

Watson Node.js Wrapper

Using the Watson Node.js Wrapper, you can now easily instantiate Watson services in a single line of code.  For example:

var watson = require('watson-developer-cloud');
var question_and_answer_healthcare = watson.question_and_answer(QA_CREDENTIALS);
var speechToText = watson.speech_to_text(STT_CREDENTIALS);

The credentials come from your environment configuration, then you just create instances of whichever services that you want to consume.

QA Service

The code for consuming a service is now much simpler than the previous version.  When we want to submit a question to the Watson QA service, you can now simply call the “ask” method on the QA service instance.

Below is my server-side code from app.js that accepts a POST submission from the browser, delegates the question to Watson, and takes the result and renders HTML using a Jade template. See the Getting Started Guide for the Watson QA Service to learn more about the wrappers for Node or Java.

// Handle the form POST containing the question
app.post('/ask', function(req, res){

    // delegate to Watson
    question_and_answer_healthcare.ask({ text: req.body.questionText}, function (err, response) {
        if (err)
            console.log('error:', err);
        else {
          var response = extend({ 'answers': response[0] },req.body);

          // render the template to HTML and send it to the browser
          return res.render('response', response);
        }
    });
});

Compare this to the previous version, and you’ll quickly see that it is much simpler.

Speech Synthesis

At this point, we already have a functional service that can take natural language text, submit it to Watson,  and return a search result as text.  The next logical step for me was to add speech synthesis using the Watson Text To Speech Service (TTS).  Again, the Watson Node Wrapper and Watson’s REST services make this task very simple.  On the client side you just need to set the src of an <audio> instance to the URL for the TTS service:

<audio controls="" autoplay="" src="/synthesize?text=The text that should generate the audio goes here"></audio>

On the server you just need to synthesize the audio from the data in the URL query string.  Here’s an example how to invoke the text to speech service directly from the Watson TTS sample app:

var textToSpeech = new watson.text_to_speech(credentials);

// handle get requests
app.get('/synthesize', function(req, res) {

  // make the request to Watson to synthesize the audio file from the query text
  var transcript = textToSpeech.synthesize(req.query);

  // set content-disposition header if downloading the
  // file instead of playing directly in the browser
  transcript.on('response', function(response) {
    console.log(response.headers);
    if (req.query.download) {
      response.headers['content-disposition'] = 'attachment; filename=transcript.ogg';
    }
  });

  // pipe results back to the browser as they come in from Watson
  transcript.pipe(res);
});

The Watson TTS service supports .ogg and .wav file formats.  I modified this sample is setup only with .ogg files.  On the client side, these are played using the HTML5 <audio> tag.

Speech Recognition

Now that we’re able to process natural language and generate speech, that last part of the solution is to recognize spoken input and turn it into text.  The Watson Speech To Text (STT) service handles this for us.  Just like the TTS service, the Speech To Text service also has a sample app, complete with source code to help you get started.

This service uses the browser’s getUserMedia (streaming) API with socket.io on Node to stream the data back to the server with minimal latency. The best part is that you don’t have to setup any of this on your own. Just leverage the code from the sample app. Note: the getUserMedia API isn’t supported everywhere, so be advised.

On the client side you just need to create an instance of the SpeechRecognizer class in JavaScript and handle the result:

var recognizer = new SpeechRecognizer({
  ws: '',
  model: 'WatsonModel'
});

recognizer.onresult = function(data) {

    //get the transcript from the service result data
    var result = data.results[data.results.length-1];
    var transcript = result.alternatives[0].transcript;

    // do something with the transcript
    search( transcript, result.final );
}

On the server, you need to create an instance of the Watson Speech To Text service, and setup handlers for the post request to receive the audio stream.

// create an instance of the speech to text service
var speechToText = watson.speech_to_text(STT_CREDENTIALS);

// Handle audio stream processing for speech recognition
app.post('/', function(req, res) {
    var audio;

    if(req.body.url && req.body.url.indexOf('audio/') === 0) {
        // sample audio stream
        audio = fs.createReadStream(__dirname + '/../public/' + req.body.url);
    } else {
        // malformed url
        return res.status(500).json({ error: 'Malformed URL' });
    }

    // use Watson to generate a text transcript from the audio stream
    speechToText.recognize({audio: audio, content_type: 'audio/l16; rate=44100'}, function(err, transcript) {
        if (err)
            return res.status(500).json({ error: err });
        else
            return res.json(transcript);
    });
});

Source Code

You can interact with a live instance of this application at watsonhealthqa.mybluemix.net, and complete client and server side code is available at github.com/triceam/IBMWatson-QA-Speech.

Just setup your Bluemix app, clone the sample code, run NPM install and deploy your app to Bluemix using the Cloud Foundry CLI.

Helpful Links

Updated: Parallax Effects in Hybrid/Web Apps

A while back I wrote about adding parallax effects to your HTML/JS experiences to make them feel a bit richer and closer to a native experience.  I’ve just added this subtle (key word *subtle*) effect to a new project and made a few changes I wanted to share here.

If you are wondering what I am talking about with “parallax effects” – Parallax movement is where objects in the background move at a different rate than objects in the foreground, thus causing the perception of depth.  Read more about it if you’re interested.

First, here’s a quick video of this latest app in action.  It’s a hybrid MobileFirst app, but this technique could be used in any Cordova/PhoneGap/MobileFirst/mobile web app experience.  The key is to keep it subtle and not too much “in your face”, and yes, it is very subtle in this video.  You have to watch closely.

The techniques that I wrote about in the previous post still apply – I’ve just added a bit more to cover more use cases.

First let’s look at the CSS.

body {
    background-image: url('../images/cloud_advisor_bkgd.png');
    background-attachment: fixed;
    background-position: center;
    background-size: auto 120%;
}

This sets the background image and default position.  The distinct change here is that I set the background size to “auto” width and 120% height.  In this case, you can have a huge image that shrinks down to just slightly larger than the window size, or a small image that scales up to a larger window size.  This way you don’t end up with seams in a repeated background or a background that is too big to highlight the parallax effect effectively.

Next let’s take a quick look at the JS involved.

var position = "center";
var lastPosition = "center";
var contentCSS = "";
var body = $("body");
var content = $(".content");
window.suspendAnimation = false;

var xMovement = 15;
var yMovement = 30;
var halfX = xMovement/2;
var halfY = yMovement/2;

window.ondeviceorientation = function(event) {
 var gamma = event.gamma/90;
 var beta = event.beta/180;

 var temp = 0;

 //fix for holding the device upside down
 if ( gamma >= 1 ) {
  gamma = 2- gamma;
 } else if ( gamma <= -1) {
  gamma = -2 - gamma;
 }

 // shift values/motion based upon device orientation
 switch (window.orientation) {
  case 90:
   temp = gamma;
   gamma = beta;
   beta = temp;
   break;
  case -90:
   temp = -gamma;
   gamma = beta;
   beta = temp;
   break;

 }

 // update positions to be used for CSS
 var yPosition = -yMovement - (beta * yMovement);
 var xPosition = -xMovement - (gamma * xMovement);
 var contentX = (-xMovement - xPosition)/2;
 var contentY = (-yMovement - yPosition)/2;

 // generate css styles
 position = xPosition.toFixed(1) + "px " + yPosition.toFixed(1) + "px";
 contentCSS = "translate3d( " + (contentX.toFixed(1)) + "px, " + (contentY.toFixed(1)) + "px, " + " 0px)";
}

function updateBackground() {

 if (!window.suspendAnimation) {
  if ( position.valueOf() != lastPosition.valueOf() ) {

   body.css( "background-position", position);
   content.css( "-webkit-transform", contentCSS);
   lastPosition = position;
  }
 } else {
  lastPosition = "translate3d(0px, 0px, 0px)";;
 }

 window.requestAnimationFrame(updateBackground);
}

window.requestAnimationFrame(updateBackground);

The html where this would be used would be something like this:

<body>
  <div class="content">put your foreground stuff here.</div>
</body>

There are some subtle but important changes here:

  1. In the requestAnimationFrame loop, it only applies changes *if* there are changes to apply.  This prevents needless calls to apply CSS even if the CSS styles hadn’t changed.  In this, I also truncate the numeric CSS string so that it isn’t reapplying CSS if the position should shift by 0.01 pixels. Side note: If you aren’t using requestAnimationFrame for HTML animations, you should learn about it.
  2. If you used my old code and were holding the device upside down, it wouldn’t work.  Not even a little bit.  This has that fixed (see comments inline above).
  3. This moves the background in CSS, which doesn’t cause browser reflow operations, and moves the foreground content (inside of a div) using translate3d, which also doesn’t cause browser reflow operations.  This helps keep animations smooth and the UX performing optimally.
  4. I also added a global variable to turn parallax on and off very quickly, if you need it.

The result is a faster experience that is more efficient and less of a strain on CPU and battery.  Feel free to test this technique out on your own.

If you use the code above, you can modify the xMovement and yMovement variables to exaggerate the parallax effect.

MBaaS – IBM Mobile Cloud Services, Bluemix & MobileFirst

MBaaS, or Mobile Backend as a Service, seems to be a particularly hot topic these days. MBaaS generally refers to backend services for mobile applications that provides data storage, user management, push notifications, and other pertinent mobile APIs.  mbaas

This is more than just “Cloud Services” which more generally refer to a scalable virtual cluster of computing or storage resources.  Bluemix is IBM’s suite of cloud service offerings, and covers lots of use cases:

Bluemix is an open-standards, cloud-based platform for building, managing, and running apps of all types, such as web, mobile, big data, and smart devices. Capabilities include Java, mobile back-end development, and application monitoring, as well as features from ecosystem partners and open source—all provided as-a-service in the cloud.

You can view the full catalog of Bluemix service offerings here.

Rather, MBaaS back-ends include services for data management, user management, notifications, and possibly more depending on the provider – all geared towards powering applications on mobile devices.

Why is it a hot topic? MBaaS enables growth of mobile applications with seamless (and virtually endless) scalability, all without having to manage individual systems for the application server, database, identify management, push notifications, or platform-specific services.

I’ve been writing a lot about IBM MobileFirst lately for a seamless API to deliver mobile apps to multiple platforms; though it has been in the context of an on-premise installation.  However, did you know that many of the exact same MobileFirst features are available as MBaaS services on IBM Bluemix?

IBM’s Mobile Cloud Services includes device management, user authentication, offline and back-end data storage, push notifications, operational analytics, and provides APIs for native iOS, native Android, hybrid apps, web apps, and even node.js clients for custom backend services.

MobileCloudServices

Here’s a bit more detail on what is currently exposed in IBM’s Mobile Cloud Services:

  • Mobile Data – The mobile data service includes a NOSQL database (powered by IBM Cloudant), file storage capabilities, and appropriate management and analytics features to measure the number of calls, storage usage, time/activity, and OS distribution.
  • Push Notifications – The push notification service allows you to easily push data to the right people at the right time on either Apple APNS or Google GCM platforms – all with a single API. Notifications can be sent by either an app or backend system, and can be sent to a single device, or a group of devices based on their tags/subscriptions.  Of course, with appropriate analytics for monitoring activity, distribution, and engagement.
  • Mobile Application Security – The mobile application security service enables you to provision or block any devices and/or users using your application, provides user authentication, and provides analytics for app/device usage, OS distribution, and time/activity.

Before starting development with IB’s Mobile Cloud Services, be sure to check out the following resources:

… and don’t forget the platform-specific developer guides:

Ready to get started?

Many of these are the exact same features that you can host in your own on-premise IBM MobileFirst Platform Foundation server – the difference is that you don’t have to maintain the infrastructure.  You can scale as needed through the Bluemix cloud offering.