Category Archives: Android

PhoneGap Day: PhoneGap & Hardware

I’d like to express a huge Thank You to everyone who attended PhoneGap Day in Portland last week, and to Colene and the team that put everything together! The day was loaded with fantastic presentations, and great community interaction… tons of information, tons of great questions, and tons of great people (oh, and beer – it wouldn’t be PhoneGap Day without beer).

My session was about the integration of PhoneGap with hardware. Basically, it was an overview and exploration how you can use native plugins to extend the capabilities of PhoneGap applications and interact with device peripherals. This enables new interaction paradigms, and helps evaluate and evolve what is attainable with web-related technologies.

In this session, I covered two use cases… The first use case is the use of a pressure-sensitive stylus for interacting with a PhoneGap application on iOS. The second use case is integration of a Moga gamepad with a PhoneGap application on Android. In both cases, the application experience is augmented by the peripheral device, which changes how it is possible for the user to interact and engage with the application content and context.

Pressure Sensitive Sketching in PhoneGap

In this example, the pressure-sensitive Pogo Connect Stylus uses a low energy Bluetooth 4 connection to relay touch/pressure information back to the PhoneGap application. This makes for a unique drawing and sketching experience powered with the HTML5 Canvas element. I’ve written about this example previously… Check out the video below to see it in action, and read the blog post for technical details and source code.

Moga Gamepad

The second example that I explored is a PhoneGap native plugin that is used to handle input from a Moga game controller inside of a PhoneGap application on Android.

This implementation is intended to be a proof of concept demonstrating how you could integrate the gamepad within your application. It currently only supports input from the joysticks (axisX and axisY) and the A and B buttons, and it does not handle all possible input from the controller.

This implementation is adapted directly from the com.bda.controller.example.demo.listen example from the Moga developers SDK samples available for download at:: http://www.mogaanywhere.com/developers/

Check out the video below to see it in action:

The game is based on the Universe prototype that was used as a sub-game inside of the MaxMe app for the recent Adobe MAX conference. I make no guarantees about the code for this game, it was in a huge rush!

Quick Links

If you want to see more images from PhoneGap Day, check out my Flickr photo set from the event.

Fresh Food Finder Featured in SELF Magazine

I am very excited to share that the Fresh Food Finder PhoneGap application which I built last summer was recently featured in the June 2013 issue of SELF Magazine. This app was built as a demonstration/sample application for PhoneGap, but has gained popularity for being an easy way to find farmers markets by leveraging the USDA farmers markets directory and has appeal to the general public.

In fact, I’ve seen so many downloads that I am working on an updated version of it based on user feedback, with more recent data (live services instead of embedded), and will be leveraging the awesome topcoat.io CSS framework for styling.

Fresh Food Finder in SELF Magazine
Fresh Food Finder in SELF Magazine

SELF Magazine has huge readership numbers (almost 1.5 million monthly subscribers according to Wikipedia). While it was just a very small callout in the magazine’s calendar, this has driven both app downloads and visitors to my blog (peaked at 348 downloads per day)!

App downloads May 1- July 8
App downloads May 1- July 8

This also brought the app higher up in the iOS market ranks (peak #57 in the “Food & Drink” category):

App store ranking May 1- July 8
App store ranking May 1- July 8

You can download the Fresh Food Finder app today in the following markets:

Grab the source code, read about how it was all put together, and see it in action in the video below:

Enjoy!

PhoneGap & Android Studio

Yesterday at GoogleIO, Google announced Android Studio, a new development environment for authoring Android applications. This is a great looking new IDE for Android, based off of IntelliJ IDEA, with some new Android-specific tools and features. You can read more about Android Studio on the Google Android Developers blog.

One of my first tasks upon downloading Android Studio was to get a PhoneGap app up and running in it. Here’s how to get started. Note: I used PhoneGap 2.7 to create a new project with the latest stable release, however you could use the same steps (minus the CLI create) to import an already-existing PhoneGap application. Be sure to backup your existing project before doing so, just in case you have issues (Android Studio is still in beta/preview).

First, follow the PhoneGap “Getting Started” instructions all the way up to (and including) the command line invocation of the “create” script.

01-cmd

Once you have the Java environment configured just run the create script to create a based PhoneGap project. In this case, I used the following command to create a new PhoneGap project:

./create ~/Documents/dev/android_studio_phonegap com.tricedesigns.AndroidStudioPhoneGap AndroidStudioPhoneGap

Next launch Android Studio. When the welcome screen appears, select the “Import Project” option.

02-welcome

Next, you’ll have to select the directory to import. Choose the directory for the PhoneGap project you just created via the command line tools.

03-select existing src

Once you click “OK”, you will proceed through several steps of the import wizard. On the next screen, make sure that “Create project from existing sources” is selected, and click the “Next” button.

04-create from existing src

You will next specify a project name and project location. Make sure that the project location is the same as the location you selected above (and used in the PhoneGap command line tools). I noticed that the default setting was to create a new directory, which you do not want. Once you’ve verified the name and location, click “Next”.

05-project location

On the next step, leave the default settings (everything checked), and click “Next”.

06-import project

Again, leave the default settings (everything checked), and click “Next”.

07-import project

Yet again, leave the default settings (everything checked), and click “Next”.

08-import project

For the last time, leave the default settings (everything checked), and click “Next”. This is the last one!

09-import project

Next, review the frameworks detected. If it looks correct to you, click the “Finish” button.

10-import project

Android Studio should now open the full IDE/editor. You can just double click on a file in the “Project” tree to open it.

11-android_studio

To run the project, you can either go to the “Run” menu and select “Run {project name}”, or click on the “Run” green triangle icon.

12-run

This will launch the application in your configured environment (either emulator or on a device). You can see the new PhoneGap application running in the Android emulator in the screenshot below. If you’d like to change your “Run” configuration profile, go to the “Run” menu and select “Edit Configurations”, and you can create multiple launch configurations, or modify existing launch configurations.

13-running

PhoneGap Legends – A Sample Game App

Next week I’ll be representing Adobe at GDC 2013, and demonstrating how Adobe Creative CloudPhoneGap, and PhoneGap Build can be great tools for building casual gaming experiences. In preparation, I’ve been working on a gaming sample application that shows off the potential of HTML games packaged with PhoneGap.

…and now I’d like to introduce you to PhoneGap Legends. PhoneGap Legends is a fantasy/RPG themed demo that leverages HTML DOM animation techniques and targets webkit browsers. I was able to get some really outstanding performance out of this example, so be sure to check out the video and read the details below. The name “PhoneGap Legends” doesn’t mean anything; I just thought it sounded videogame-ish and appropriately fitting.

PhoneGap Legends
PhoneGap Legends

This game demo is an infinitely-scrolling top-view RPG themed game that is implemented entirely in HTML, CSS, and JavaScript. There is a scrolling background, enemy characters, HUD overlays, and of course, our “protagonist” hero – all that’s missing is a story, and general game mechanics like interacting with sprites.

Again, I was able to get some *really outstanding performance* out of this sample, so I wanted to share, complete with source code, which you’ll find further in this post (and I encourage you to share it too). Take a look at the video below to see the game in action on a variety of devices. Every single bit of this is rendered completely with HTML, CSS, and JavaScript – there are no native portions of the application.

Update 3/24:  If you’d like to test this out on your own devices, you can now access it online at http://tricedesigns.com/portfolio/phonegap_legends/   However, it will still only work on webkit browsers (Chrome, Safari, Android, iOS, etc…), and is optimized for small-device screens.  If you attempt to use this on a very large screen, you’ll probably see some sprite clipping.

Disclaimer: This sample app is by no means a complete game or complete game engine. I’ve implemented some techniques for achieving great performance within a PhoneGap application with game-themed content, but it still needs additional game mechanics. I also wrote this code in about 2 days – it needs some additional cleanup/optimization before use in a real-world game.

Source

Full source code for this demo application is available on GitHub. This code is provided as-is: https://github.com/triceam/PhoneGap-Legends. I will be making a few updates over the next few days in preparation for GDC next week, but for the most part, it is stable.

Development Approach

The PhoneGap Legends application was built following the performance tips that I posted earlier this month. The game is built so that it uses a game loop based upon requestAnimationFrame to perform all UI updates. This gives it a scheduled interval that is in sequence with the browser’s redraw cycle.

In general, the DOM is as shallow as possible for achieving the desired experience. All “sprites”, or UI elements are basic DOM nodes with a fixed position and size.  All DOM elements have an absolute position at 0,0 and leverage translate3d for their x/y placement.  This is beneficial for 2 reasons: 1) It is hardware accelerated, and 2) there are very, very few reflow operations.  Since all the elements are statically positioned and of a fixed size, browser reflow operations are at an extreme minimum.

The background is made up a series of tiles that are repeated during the walk/movement sequence:

Sprite Sheet for Background Tiles

In the CSS styles, each tile is 256×256 square, with a background style that is defined for each “type” of tile:

.tile{
    width: 256px;
    height:256px;
    overflow: hidden;
    position: absolute;
    top:0px;
    left:0px;
    background-repeat: repeat;
    background-clip:border-box;
}
.tile_0 {      background: url('../images/background_tiles.png');     background-position: 0px 0px;}
.tile_1 {      background: url('../images/background_tiles.png');     background-position: 256px 0px;}
.tile_2 {      background: url('../images/background_tiles.png');     background-position: 512px 0px;}
.tile_3 {      background: url('../images/background_tiles.png');     background-position: 0px 256px;}
.tile_4 {      background: url('../images/background_tiles.png');     background-position: 256px 256px;}

The content displayed within each of the “sprite” DOM elements is applied using sprite sheets and regular CSS background styles. Each sprite sheet contains multiple images, the background for a node is set in CSS, and the position for each image is set using the “background-position” css style.  For example, the walking animation for the hero character is applied just by changing the CSS style that is applied to the “hero” <div> element.

Sprite Sheet for Hero

There is a sequence of CSS styles that are used to define each state within the walking sequence:

.hero_0_0{ background: url('../images/rpg_sprite_walk.png'); background-position:0px 0px;}
.hero_0_1{ background: url('../images/rpg_sprite_walk.png'); background-position:0px 96px;}
.hero_0_2{ background: url('../images/rpg_sprite_walk.png'); background-position:0px 192px;}
.hero_0_3{ background: url('../images/rpg_sprite_walk.png'); background-position:0px 288px;}
...

This game demo extensively uses translate3d for hardware accelerated composition.  However, note that the 3d transforms are all applied to relatively small elements, and are not nested. All of the “textures” are well below the max texture size across all platforms (1024×1024), and since it uses sprite sheets and reusable CSS styles, there are relatively few images to load into memory or upload to the GPU.

Attribution

The following Creative Commons assets were used in the creation of this sample app and the accompanying video:

Image – Grass Texture: http://opengameart.org/content/grass-with-moss
Image – Trees/Bushes: http://opengameart.org/content/lots-of-hyptosis-tiles-organized
Image – Main Character: http://opengameart.org/content/2d-rpg-character-walk-spritesheet
Image – Enemies: http://opengameart.org/content/rpg-enemies-11-dragons
Image – Compass Rose: http://en.wikipedia.org/wiki/File:Compass_rose_en_04p.svg

Font – Avalon Quest: http://www.ffonts.net/Avalon-Quest.font
Note: I used a free/open licensed font that could be embedded in the app for offline usage. This font was converted to a web-font package that can be embedded in the PhoneGap application using http://www.fontsquirrel.com/

Audio – Monster: http://www.freesound.org/people/Erdie/sounds/167890/
Audio – Music: http://www.freesound.org/people/rap2h/sounds/115261/

Source

Again, in case you missed it above, Full source code for this demo application is available on github. This code is provided as-is: https://github.com/triceam/PhoneGap-Legends.

Performance & UX Considerations For Successful PhoneGap Apps

I think some of the most common questions people ask when developing PhoneGap apps are about performance. How do I make my app faster? How do I make it feel native? Any tips of making an app feel at-home on a particular platform? How do you match native OS “look and feel”? In this post, I’ll attempt to provide some insight on techniques for creating great PhoneGap apps, and also try to debunk a few myths floating around the “interwebs”…

I’ve already posted about workflow and app store approval processes, and in this post the primary focus is runtime performance.

When we are talking about performance, we are really talking about perceived performance and responsiveness from an end-user perspective. Since PhoneGap applications are based upon HTML & web views, you are limited by the performance of a web view on a particular platform. With that said, that doesn’t mean all HTML-based apps are inherently slow. There are very many successful apps with an HTML-based UI. Some even get featured in the app stores. We can’t make the web views faster, but we can help you make your HTML experience faster inside of that web view.

However if the web view isn’t fast enough for your taste, don’t forget that you can use PhoneGap as a subview of a native app. In this case, you can leverage native user interface elements when you need them, and leverage PhoneGap’s HTML/JS user interface when you need it. Then use PhoneGap/Cordova’s native-to-JS bridge to handle bidirectional communication between the native code and the JavaScript inside of the web view component. More on this later…

HTML/Web View Performance

First, let’s talk specifics about mobile HTML & web performance. There are a lot of common “tips” floating around, some of these are extremely valid, although others are misleading. Please allow me to clear some things up…

Hardware Acceleration

All over the the web, you can see generic statements telling you to force hardware acceleration whenever possible, in every scenario, just by adding a css translate3d transform style:

transform: translate3d(0,0,0);

While it is true that you can force GPU rendering of HTML DOM content, you really need to understand how it can impact the performance of your application. In some cases, this can greatly improve the performance of your app, while in other cases, it can create performance issues that are extremely difficult to track down. In other words: Do not use this for every case, and on every HTML DOM element.

You should be very selective when you use transtale3d to force hardware acceleration for a number of reasons.

First, whenever you are using translate3d, the content being accelerated takes up memory on the GPU. Over-use of GPU rendering on HTML DOM elements can cause your application to use up all of the available memory on the GPU. When this happens, your app could either crash completely without any error messages or warnings, or the GPU could swap memory content from device RAM or permanent storage. In either of these cases, you will have worse performance than you did without forced GPU rendering.

Second, whenever you apply a translate3d transform, this causes the content of the HTML DOM element to be uploaded on the GPU for rendering. In many cases, the GPU upload time is imperceptible, and the results are positive. However, if you apply the translate3d style to a very large and very complex HTML DOM element, a noticeable delay can be encountered, even on the latest and greatest devices and operating systems. The more complex the HTML DOM, the larger the delay. When this happens, the web view (or browser) UI thread can be completely locked up for that period. While the worst I’ve seen of this was less than a second, it still can have a profoundly negative impact when your entire UI locks up for 500ms. In addition, any time you make changes to a DOM element (content or style-wise), that texture has to be re-uploaded to the GPU.

Third, avoid nesting DOM elements that use translate3d. Let’s consider a case where you have lots of DOM elements inside of a <div> elements, multiple levels deep, and all of those elements, including the outtermost <div> use translate3d for GPU rendering. For the outter-most <div> to be rendered, all of it’s child elements have to be rendered. In order for all of those child elements to be rendered, all of their child elements have to be rendered, etc… This means that all of the child elements have to be uploaded to the GPU before the parent element can be rendered. This adds up very quickly. Nested elements that use the css translate3d GPU hack will take longer to load, and can take up significantly more memory, causing very poor performance, slow GPU upload times, or complete app crashes (see paragraphs above).

Fourth, consider the maximum texture size for the GPU. If the DOM element is larger than the maximum texture size supported by the GPU (in either width or height), then you will encounter very poor performance and visual artifacts. Have you ever seen HTML DOM elements flicker during an animation or when scrolling, when using translate3d? If so, then there’s a good chance that the DOM element’s dimensions are larger than the texture size supported by the device’s GPU. The maximum texture size on many mobile devices is 1024×1024, but varies by platform. The maximum texture size on an iPad 2 is 2048×2048, where the maximum texture on the iPad 3/4 is 4096×4096. If the device maximum texture size is 1024, and you DOM element’s height is 1025 pixels, you will have those pesky flickering and performance issues.

Note: In some platforms, you can also have flickering due to the browser implementation, which can often be corrected by setting backface-visibility css. See details here.

Fifth, remember that you are on a mobile device. Mobile devices tend to have slower CPUs, slower bus, and less memory than desktop devices. While you may get great results leveraging translate3d on the desktop, it could kill your experience on mobile. In addition, different platforms have different levels of support for hardware acceleration, and performance can vary.

Translate3d can definitely yield positive results, but can also yield negative results too, depending upon how it is used.  Use it wisely, and always, always, always test on a device.

The Impact of Content Reflow

If you’ve never heard of “reflow” before, you should pay attention. “Reflow” is the browser process for calculating positions and geometries for HTML DOM elements. This includes measurement of width & height, the wrapping of text, the flow/relative position of DOM elements, and basically everything related.

The calculations used by the reflow processes can be computationally expensive, and you want to minimize this cost and the amount of content reflow to achieve optimal performance. If you’ve ever animated the width of a HTML DOM element, and seen the framerate drop to to 5 FPS or lower, then you’ve seen the impact of reflow processes in action.

Reflow processes are invoked any time DOM content changes, DOM elements are resized, CSS positioning/padding/margins are changed, etc… While you may not notice any impact of reflow processes on desktop browsers, reflow can have a significant performance impact on mobile devices. You can never get rid of reflow processes altogether (well, unless your content absolutely never changes), but you can mitigate the impact of those relflow processes.

If you’d like to learn more about reflow processes, I strongly suggest reading these:

If you’ve seen lists of tips for PhoneGap & Mobile web apps, you may have seen statements like “minimize DOM nodes”, “avoid deeply nested html”, use css animations, preload images, etc… The reason that these are suggested is that they are tips for minimizing the impact and occurrence of reflow operations.

  • Minimize DOM node instances – The fewer DOM nodes, the less amount of nodes that have to be measured and calculated in reflow operations.
  • Avoid deeply nested HTML DOM structures – The deeper the HTML structure, the more complex and more expensive reflow operations. In addition, changes at the lowest child level will cause reflow operations all the way up the tree, ending up more computationally expensive.
  • Use CSS transforms – In addition to forcing hardware acceleration as discussed above, you can alter HTML DOM elements *without* invoking reflow processes if you change them using CSS3 transform styles. This could be translation on the X, Y, or Z axis, scaling, or rotation. However, be careful not to overuse css translate3d for the reasons mentioned above.
  • Use CSS animations & transitions - CSS animations and transitions can definitely make things faster. However this is not a “catch-all” that covers every case. If you use CSS transitions with properties that introduce reflow operations (such as width or height changes), then your performance could stutter. You can achieve fluid animations by leveraging CSS transitions, in conjunction with css transforms (as mentioned above) b/c they alter the visual appearance, but don’t invoke reflow operations.
  • Use fixed widths and heights for DOM elements - If you don’t change the size of content, you don’t invoke reflow operations. This could be for <div> (or other) elements, or it could apply to the loading of images. If you don’t predefine an image size, and wait until the image is loaded, then there is an expensive reflow operation when the image loads. If you have multiple images, this adds up.
  • Preload images or assets used in CSS styles – This is a benefit for two reasons. First, your images will already be ready when you need them. This prevents a delay or flicker when waiting for content. Second, if you already have images or other assets preloaded into memory before they are needed, then you don’t have a secondary reflow operation that could occur after the image/assets have been loaded (1st reflow is when DOM is initially calculated, 2nd when its recalculated on load).
  • Be smart with your HTML DOM elements – Let’s say you want to create and populate a <table> element based on data you have in a JavaScript array. It is very expensive to first append a <table> element, then loop over the array and individually append rows to the existing DOM on every loop iteration. Rather, you could loop over the array, and create the HTML DOM element for the table only. Then, after the loop has been completed, append the <table> to the existing HTML DOM. Every time that you append DOM elements, you invoke a reflow operation. You want to minimize these operations.

In general, you want to minimize the amount of work required by the browser to calculate layout and positioning of DOM elements.

Keep Graphics Simple

Designers love to make great looking mockups, but sometimes the implementation of such designs don’t always yield great performance. Overuse of CSS shadows and CSS gradients can impair runtime performance across different platforms & browser implementations. I’m not saying don’t use them at all, just use them wisely. For example, CSS gradients and shadows induce slower performance on Android devices than a compariable iOS device, based on the OS/system web view implementation.

Touch Interactivity

You’ve probably heard before that “mouse events are slow and touch events are fast”. This is absolutely a true statement. Instead of using “mousedown”, “mousemove”, “mouseup”, or “click” events, you should use touch events: “touchstart”, “touchmove” and “touchend”.

On mobile devices, mouse events have latency that is introduced by the operating system layer. The OS tries to detect if a gesture has occurred. If no gesture has occurred, it passes the event on to the web view as a mouse event. When you use touch events instead of mouse events, you recieve the event immediately, without the operating system-introduced latency.

You may have noticed that there is no equivalent of “click” for touch events… That’s right, there isn’t one. However, you can manually manage the touch events yourself to infer a click action, or you can leverage an existing library to infer “taps” or gestures. Here are a few for consideration: Zepto, FastClick, Hammer.js, iScroll (yes, iScroll removes the click delay for you inside the scrollable area)

JavaScript Optimizations

In essence, write efficient code that doesn’t block the UI thread. I could go on for a while on this, but instead, just follow language optimizations and best practices. Here are a few articles on the subject:

There are lots of facets and permutations to consider when building your apps, so just try to write decent code that you wouldn’t be embarrassed to show to your coder friends.

Native Performance

So, let’s say that you want a native user experience, native navigation between containers, but you also want the ease of content creation with HTML? Nobody said you can’t use PhoneGap as a subview of a native application. In fact, there are some VERY widely used apps in existence that employ this technique… If only I could tell you (NDA).

This approach requires native coding expertise, but you can use CordovaView as a subview inside of a native application on both iOS and Android platforms. This allows you to take advantage of native components when you want to, and leverage a PhoneGap/HTML UI when you want to. PhoneGap provides you with the native-to-JavaScript bridge for full bidirectional communication between the native and JavaScript layers.

When using HTML, it is really easy to create custom UIs, whether they are complex graphics, tabular data, or anything really. With the CordovaView approach, you let each layer (native/HTML) do what it does best.

You can learn more about embedding the CordovaView inside of a native application from the PhoneGap Docs.

UI & UX Considerations

At risk of sounding overly-generalized: You want to pay attention to building a quality user experience. If you don’t focus on quality, your application will suffer. I’ve already covered user interface and user experience considerations in detail, but I’ll highlight one common question that I encounter: “How do I make an app that exactly matches native look and feel for all platforms?”

My response: Don’t.

Let me clarify this point: Im not saying don’t build something that looks “at home” on a particular platform. I’m suggesting that you don’t try to match the operating system in every single detail. The main reasons being that 1) this is very difficult to do, and 2) if anything changes in the OS, it will be very obvious in your application.

You may or may not have heard of the “uncanny valley” effect. When applied to PhoneGap apps, if you get really, really close to behaving exactly like a native app, but there are minor differences, then it can immediately signal that “something is different” or “something is wrong” with your app, and can cause an undesired negative experience for the user.

Instead, I recommend creating a unique identity for your app (that still meets App Store guidelines). This includes look and feel, button styles, navigation, content, etc… Rather than mimicking every aspect of the native UX, build a cohesive experience around your identity or brand, and carry that identity through all of your mobile apps. People will associate this identity with your app, rather than comparing it to the native OS.

However, if you do want to mimic native look and feel, it can be done completely with CSS styles.

Test On Devices

I can’t emphasize this point enough.  Always test on a device. Always. I like to target older devices, because if you can make it fast on an older device, it will be even faster on a newer one. Be sure to test on-device on all platforms that you are trying to target. For iOS, I test on an iPhone 4 (not 4s) and and iPad 2. On Android, I use a Motorola Atrix, Nexus 7 tablet, Kindle Fire (first generation), and a Samsung Galaxy 10.1 (first generation). I test other platforms whenever I borrow a device, and pretty much use whatever I can get my hands on.  Heck, I’ve even gone into retail stores just to install an app on one of their display devices to see how it looks.