Flex 2 BitmapData Tricks and Techniques

Most seasoned flash developers know that the Bitmap API has been around in the Flash player since the relase of FP8. Those that know all about the Bitmap API appreciate it for its many benefits... Those that don't know about it, trust me, you appreciate it too... you just don't know it. In Flex 2, you can take advantage of the Bitmap API to implement some very powerful and useful techniques that simply are not possible in non-Flash-based web applications.

Some of you may be asking: What is the Bitmap API?

The Bitmap API refers to the flash player's functionality that renders every visual object within a swf as a bitmap (which gets cached), rather than redrawing the entire object every frame interval. Only objects that are marked as "dirty" will be redrawn. The amount of resources used to render each frame is significantly reduced, thus improving the runtime performance of swf files on the client machine. You can find some more information on the Bitmap API at:

• http://www.adobe.com/devnet/flash/articles/bitmap_caching.html
• http://www.adobe.com/devnet/flash/articles/image_api_03.html

Now you may now be asking: What does this have to do with my Flex application? All Flex components that can be visually represented within your Flex application apply to this. They are drawn initially, and then cached as a Bitmap within the Flash player. This is where things get really interesting. Not only does this increase application performance, but it gives us access to the rendered bitmap of the instantiated object. Below you will find a very small, yet powerful, function that will allow you to capture the BitmapData of ANY visual flex component. I just used the UIComponent class as a parameter b/c it is the base class for virtually all flex components.

private function getUIComponentBitmapData( target : UIComponent ) : BitmapData
{ 
    var bd : BitmapData = new BitmapData( target.width, target.height );
    var m : Matrix = new Matrix();
    bd.draw( target, m );
    return bd;  
}

So now what? You can capture the BitmapData from any visual component. Why would you ever want to do that? I've got a few reasons (and this is just the tip of the iceberg).

1. Bitmap-Based Imaging Effects
2. UI Component Mirroring
3. Application Sharing
4. Screen Exports

Did you know that you can set the source of a Image object to a Bitmap object? Did you also know that you can create a Bitmap object from a BitmapData object. This means that you can easily take the BitmapData from an existing visual component, and push it into an Image object.

My first example shows how to put this code to use, although its rather boring compared to my other examples. It takes the BitmapData of the specified UI component and pushes it into an Image object. This code example can also be found on JAM.

<?xml version="1.0" encoding="utf-8"?>
<mx:Application 
 xmlns:mx="http://www.adobe.com/2006/mxml" 
 layout="absolute">
 
 <mx:Script>
  <![CDATA[
  
   import mx.core.UIComponent;
   
   private function captureFullScreen() : void
   {
    var bd : BitmapData = getBitmapData( UIComponent( mx.core.Application.application ) );
    targetImage.source = new Bitmap( bd );
   }
   
   private function captureHiddenDatagrid() : void
   {
    var bd : BitmapData = getBitmapData( UIComponent( hiddenDg ) );
    targetImage.source = new Bitmap( bd );
   }

   private function getBitmapData( target : UIComponent ) : BitmapData
   {
    var bd : BitmapData = new BitmapData( target.width, target.height );
    var m : Matrix = new Matrix();
    bd.draw( target, m );
    return bd;
   }
   
   
  ]]>
 </mx:Script>
 
 <mx:Button 
  id="captureButton" 
  label="Capture Full Screen" 
  click="captureFullScreen()" />
  
 <mx:Button 
  id="captureButton2" 
  label="Capture Hidden Datagrid" 
  click="captureHiddenDatagrid()"  
  x="153"/>
 
 <mx:Image 
  id="targetImage" 
  x="10" 
  y="30"/>
 
 <mx:DataGrid 
  x="99" 
  y="64" 
  id="hiddenDg" 
  visible="false">
  <mx:columns>
   <mx:DataGridColumn headerText="Column 1" dataField="col1"/>
   <mx:DataGridColumn headerText="Column 2" dataField="col2"/>
   <mx:DataGridColumn headerText="Column 3" dataField="col3"/>
  </mx:columns>
 </mx:DataGrid>
 
</mx:Application>

Launch Application
View Source
Download Source

Bitmap-Based Imaging Effects

Wouldn't it be cool if you moved one thing around on the screen, and it left motion trails? What if you wanted to capture part of a component, distort the image (colors, blur, size, crop, etc...) and then show it in a different part of your application? Well, the getUIComponentBitmapData function makes this difficult-sounding task much easier.

In this example, I have an image that is at 0,0 (top left corner) and is the exact same size as my application. The alpha on that image is 90%. On every "ENTER_FRAME" event, it takes a snapshot of the entire application and pushes it into the image. This causes the "trail" effect that you will see. Since the alpha is only 90%, it will eventually fade out, creating the fading image effect. The code is amazingly short, powerful and efficient.



Note: This process will become very processor intensive at high resolutions.

Launch Application
View Source
Download Source

User Interface Component Mirroring
Have you ever had the need to mirror the user interface? I know, this is not a common need but take it a step further (which I will do later in this post), and you have some really useful tools in your toolbox. This next example takes the BitmapData from a tree inside of a panel and pushes it into an Image inside of another panel. This occurs every 100 milliseconds. You'll see that there is a slight delay, but overall it mirrors the interaction very effectively.



Launch Application
View Source
Download Source

The next example acutally mirrors a flv file by pushing the BitmapData of the video playback's current frame into an Image object on every "ENTER_FRAME" event. You'll notice that a few frames are dropped per second, but you really have to look very closely to see the difference. One thing to keep in mind, more frames will be dropped on slower client machines.



Disclaimer: This video clip was downloaded from Google video. This is a scene from the surfing movie Billabong Odyssey, which is a great movie (that I do personally own). I highly reccommend it.

Launch Application
View Source
Download Source

Application Sharing

In my opinion, this next example is really amazing. It is also incredibly simple, yet powerful. I've extended my "User Interface Mirroring" example to share data between two different browsers using local shard objects. Keep in mind, this code only works on the same machine, but who is to say that you can't change that. You could easily chage the local shared object to a remote shared object, incorporating Flash Media Server, and this will work across different client machines. Also, what is stopping you from doing a RemoteObject call to push the image data to a server? This would enable you to pull down the image from the server to any client machine, without the use of Flash Media Server.



View Captivate Demo
Launch Broadcaster Application
Launch Receiver Application

View Source
Download Source

Side Note: If you looked at the source, you might be wondering why I didn't use getPixels and setPixels instead of my own functions to loop through the pixel arrays. I found a bug in the Flash player and get an "End Of File" error when using get pixels and set pixels on items rendered within the player memory. I'm putting together a test case to send to Adobe for this. I didn't use a BitmapData object on the local shared object because the player can't deserialize it back into a BitmapData object when reading the object back from the local shared object. Instead, I saved it in the LSO as a ByteArray.

Screen Exports
I think this next example is really cool too. Have you ever wanted to export your flex application's current view to the client machine? Using this approach, you can capture the UI as a jpg that can be downloaded by the user. This technique could have a number of uses:

• Capture the user's screen when an error occurs
• Save the current image within the application: possibly for a flex-based image editing solution... yes, I am working on one of those as a side project :-)
• Distributed application sharing... this could be a great asset for remote application assistance

In this example, I capture the UI BitmapData, convert it to a JPG ByteArray using a modified version of the JPGEncoder class that is a part of the source code to the ActionScript 3 corelib project available at Adobe Labs. I push it to the server using a RemoteObject call and save the JPG ByteArray in the active session. On the ResultEvent, I make a request to the server and pull down the jpg into a new window on the client machine's browser (completely outside of Flex). Its amazing how easy that was. This is another one of the incredibly rich interactions that is just not possible with an AJAX (DHTML)-based RIA. I used ColdFusion as a backend on this one. Sorry, Im not sharing the all of the source to this example, but you will be able to see a portion of it.



Note: this last example is hosted at tricedesigns.com, that is my personal ColdFusion server.
View Captivate Demo
Launch Application
View Partial Source
Download Partial Source

Questions or comments, you can contact me at andrew.trice( at )cynergysystems.com.

Singleton in AS3

I was explaining the singleton design pattern to a coworker today, and used Wikipedia as a source of information. There are examples of different languages on there, but no ActionScript! So, I decided to go ahead and add it. Here's the link to AS3 singleton example on wikipedia.

In case you didn't want to go to wikipedia...

package
{
public class Singleton
{
private static var singleton : Singleton

public static function getInstance() : Singleton
{
if ( singleton == null )
singleton = new Singleton( arguments.callee );
return singleton;
}

//NOTE: AS3 does not allow for private or protected constructors
public function Singleton( caller : Function = null )
{
if( caller != Singleton.getInstance )
throw new Error ("Singleton is a singleton class, use getInstance() instead");

if ( Singleton.singleton != null )
throw new Error( "Only one Singleton instance should be instantiated" );

//put instantiation code here
}
}
}

Enhanced Flex MDI Interface

Two weeks ago, I posted a prototype MDI (Multiple Document Interface) application. Well, I am very happy to say that this has been seriously improved. I'll say it again, this is one concept that Flex handles very well. I have yet to find anoter web development technology that allows as rich of an interface, with such easy of use, extensibility, and resusability. Hopefully this will help you to spur your own ideas.

Check out the demo (2 MB captivate movie):

...Or try it out yourself.

Basically, here's what I've done to make it better:

• Animation! Yes, that's correct. There are now animations for minimize, maximize, resize, etc... Inside my WindowComponent class I've taken advantage of Flex effect's easing functions on the animations. It will turn into some neat mac-genie-like effects (but I've toned it down b/c it can also make you nauseous).
• Did you say resize? Yes, I did. You can click and drag on the bottom right corner of the window to resize it (if resize is enabled).
• Ability to minimize windows to the bottom of screen (and they stay on the bottom when the screen is resized).
• Enable/Disable window options. The window options (minimize, maximize, resize, close) can be enabled/disabled per window instance through either mxml code or actionscript.
• When a window is maximized, it can't be dragged & dropped. It can only be dragged and dropped once it has been minimized or restored to default state.
• Did I just say that it could be implemented in mxml? Yes, I did. It has been modified so that you can create window components using either mxml or actionscript (your choice). This is the extensibility and resuability that I am talking about. It is extremely easy to build amazing applications using this approach. For example:
  <mdi:WindowComponent title="Example Window" >
  <mdi:content>
  <mx:Text width="100%" height="100%" text="This is a window." />
  </mdi:content>
  </mdi:WindowComponent>
  

That's all the changes that I can think of off the top of my head, but there could be more... Sorry, I'm not sharing the source of this. If you have questions or comments, feel free to let me know at andrew.trice(at)cynergysystems.com.

Try out the MDI interface for yourself.

More on Flex Framerate and Performance

Original link: http://www.cynergysystems.com/blogs/page/andrewtrice?entry=more_on_flex_framerate_performance

Recently my blog got added to MXNA, and it has gotten some serious hits. First of all, THANKS! I'm glad people are reading it. I'll do my best to keep it interesting and informative. Another interesting twist with more readership is that people pose questions to you, or suggest other ways of doing things. This is great feedback. Thanks to those who let me know.

Back in June, I wrote a blog posting on Flex Framerate and Performance. With all of the new readership coming from MXNA, this posting has become quite popular. I'd like to append to this topic and provide words of caution where they may be necessary, but I will reinforce that this has proven to be a powerful technique.

Originally I wrote "Increase the default framerate in your flex applications to squeeze as much performance as possible out of them". I'm not changing this opinion at all, but I'd like to add that the decision whether or not to use this technique should be determined case by case. I have used this technique to increase the quality and responsiveness of the user interface for enterprise-level Flex applications, and, in my situations, it has achieved great success. Although, this is a technique that should be used with caution. There could be possible negative side effects that should be watched out for. The decision whether or not to use this technique should be determined on a case by case basis, which ultimatley depends on how you are using flex and whether or not it helps to suit your needs. Based on my experience, it has proven very effective.

Basically, here is what I use it for:

• smoother scrolling
• smoother Drag/Drop
• smoother transitions and effects
• better overall UI performance of the applicaion

Now, lets think about the real world... It is very possible to create an application where this could cause problems. Not all developers use flex the same way, and that's where part of my advice runs into trouble.

Here's why you should use caution:

1) Distortion of embeded keyframe-based content.

Changing the framerate can cause some serious problems if you are using embded flash (swf) files that contain keyframe-based content. In the chart below, I have embeded a flash file (whose defaul framerate is 12 fps), and shown the effects of embedding that file into the flex application. You will notice that all 3 elements have different speeds, hence different framerates. The default framerate is 24 fps, and you will notice the difference between that and the swf's actual framerate of 12 fps. On my local machine, the target 99 fps actually runs at about 60 fps. I'll get more into how I calculated the fps a little later in this post (including code) and what the results mean to me. If you have a very high framerate and you do something *else* on your machine that hogs the cpu, the framerate within your flex application will drop significantly. The smooth animations you may be seeing below will slow down and/or operate at inconsistent speeds. If your keyframe based animation was designed to run at a specific fps, you probably don't want to go fooling around with it too much. Just set the application framerate to the same framerate of your keyframe-based content.

Standalone SWF FrameRate: 12 FPS	SWF Embeded Default FrameRate (24 FPS)	SWF Embeded Target 99 FPS

2) Complexity of Flex content.

In the real world, it is possible to create very "hoggy" flex applications. If you have a lot of effects, and a lot of transitions, or you have functionality that causes the entire viewport to be redrawn every frame (rather than using the cached bitmap of the objects), then the rendering of the flex application is going to require more processing power. In this scenario, setting the framerate very high could also be a bad idea. It could potentially cause a poor user experience by being a "resource hog", responding slow (or not at all), or potentially crashing the browser. Other developers have also informed me that it is possible to have process starvation, where the actionscript thread does not get executed, and result events don't fire. I have not experienced any of these scenarios, but I just want to cover all points of view here. I am aware that this was a more common problem in older versions of the flash player, but its been my experience that Flash player 9 handles this very well.

Others have pointed out that Firefox also restricts the cpu and framerate available to the flex application when it is in a non-showing tab or is not in focus. I'm aware of this, and it has been my experience that this doesn't cause any problems. Be advised, it could cause problems for you, especially if you rely on keyframe-based content. A typical flex application does not really change much when the user is not interacting with it, thus a typical flex application shouldn't be affected much... but it is still something to be aware of.

---

With that said, this is still a very powerful technique. If you want to use it, go for it, but use it wisely. Be sure to test your application on multiple systems (old & new, windows & mac, single core & dual core, etc...) and make sure it is a good experience for everyone. I have used this many times and have had great results.

Some may think a target of 99 fps is too aggressive, and I can see your point, given that the actual human eye actually percieves fluid motion at around 20-30 fps, with "true" fluid motion at around 72 fps, but this can be achieved at lower speeds, given the proper lighting conditions, brightness, contrast, and smoothness of the images (including motion blur). If you don't want to run the risk hogging too many resources, but still want better-than-average usability, set the frame rate to 50 (or any number higher than the default 24 fps). The interaction will be more fluid and will be a better experience for the user. Lets face it though, not all applications need this. If you are not using any interfaces where the increase in frame rate will be noticed, then you do not need to increase the frame rate.

The latest version of the Flash 9 player does a great job rendering swf files... With the implementation of bitmap caching and the optimizations of the Flash player 9 engine, the methods used for drawing the UI require much less overhead than in previous versions of the Flash player. It has been my experience that that framerate is an actual target. If the client machine cannot keep up, the frame rate is throttled to one that is manageable by the client system. Let's take this mini flex application for example:



The target framerate is 99 fps. The actual framerate on my machine hovers around 60 fps. Looking at the task manager, my cpu usage is zero.



Below, you'll find the code that I used to calculate the frame rate. Basically, I update a counter on every EnterFrame event. I have a timer that will reset the count every second. Pretty simple, and it does a good job.

<?xml version="1.0" encoding="utf-8"?>
<mx:Application
xmlns:mx="http://www.adobe.com/2006/mxml"
layout="absolute"
creationComplete="onCreationComplete()" frameRate="99" viewSourceURL="srcview/index.html">

<mx:Script>
<![CDATA[

private var fps : uint;
private var timer : Timer;

private function onCreationComplete() : void
{
timer = new Timer(1000);
timer.addEventListener( TimerEvent.TIMER, onTimerEvent );
timer.start();
fps = 0;
this.addEventListener(Event.ENTER_FRAME, onFrameEnter);

}

private function onFrameEnter( event : Event ) : void
{
fps++;
trace("enterframe");
}

private function onTimerEvent( event : Event ) : void
{
fpsText.text = fps.toString();
fps=0;
}

]]>
</mx:Script>
<mx:Label x="10" y="10" text="fps:"/>
<mx:TextInput x="45" y="8" width="123" id="fpsText" editable="false"/>

</mx:Application>

Now, lets take a look at the impact that 99 fps really has on my machine. I opened up 8 different flex applications in firefox, each with 99 fps. According to my fps meeter, the framerates all dropped to around 30-40fps, but the applications still ran very smoothly, and firefox was only at 24 percent cpu usage. This is a very reasonable number, given how much is actually going on within the browser. (Pentium 4, 3.2 GHz, 1.5 Gigs of RAM)





Download Source
View Source