I’ve spent the last day or two reacquainting myself with what’s been done to achieve reliable gesture recognition in AS3. I have reviews of two different packages I tried, one of which is relatively new and one of which has been around for some time.

One note about all of the gesture recognition classes I’ve looked at is that they are direction-dependent. That is to say that if you’re attempting to capture the gesture corresponding to a vertical line, a stroke from top to bottom is not the same thing as a stroke from bottom to top. Furthermore, neither of those will necessarily be captured in a stroke from top to bottom to top. This all has to do with the algorithms being employed to detect gestures and attempts to make them run as efficiently as possible.

The first package I looked at was offered up by a Mr. Felix at bertiesbraum.de. Felix’s gesture recognizer consists of AS3 implementations of the “$1 Gesture Recognizer” and “ShortStraw Corner Finder” algorithms, which he has the decency to cite and link to (unlike me).

His post does a much better job of explaining basic usage of the classes than I care to at the moment, but what I will say is that I was unable to get the classes to function without false positives. This is likely because, as Felix notes, “The gestures are also rotation, position and scale invariant,” which is to say that a single stroke probably ends up being a single stroke, regardless of which direction it points. With more complex figures I was able, in the sample application Felix provides, to demonstrate direction dependence. In my case I’m trying to detect single strokes in distinct directions, and so it may be that these algorithms are over-engineered for my purposes.

The second package I looked into was posted in a short article at ByteArray.org by Didier Brun. I caught this post when it first came out (almost three years ago) and was very impressed with the concept but didn’t have any real use for it at the time. I toyed with it today and have been pleased with the results.

Initially Mr. Brun’s classes posed the same problem that most gesture recognition algorithms do: it was unable to recognize identical patterns if they proceeded in different directions. Additionally, all of the packages I’ve seen presume that the user will begin their gesture by pressing the mouse button down, and end the gesture by releasing it. I’m trying to detect whether or  not a user is shaking an object they’ve picked up in an up-and-down fashion, so it’s important to me to recognize sub-gestures within a larger “gesture.” If I depend solely on button presses and releases, the gesture that ends up being analyzed consists of tens of motions at the least.

I came up with a solution using Mr. Brun’s classes that allowed me to achieve my ends. Brun’s gesture recognition class offers a custom event that fires when capture begins and ends, when a match is made or not made, and while capture is taking place. The last of those was most important to me. I added a public property to his existing GestureEvent class. The modified class looks like this:

package com.foxaweb.ui.gesture {

	import flash.events.Event;

	public class GestureEvent extends Event {

		// ------------------------------------------------
		//
		// ---o static
		//
		// ------------------------------------------------

		public static const START_CAPTURE:String="startCapture";
		public static const STOP_CAPTURE:String="stopCapture";
		public static const CAPTURING:String="capturing";
		public static const GESTURE_MATCH:String="gestureMatch";
		public static const NO_MATCH:String="noMatch";

		// ------------------------------------------------
		//
		// ---o properties
		//
		// ------------------------------------------------

		public var datas:*;
		public var fiability:uint;
		public var moves:String;

		// ------------------------------------------------
		//
		// ---o constructor
		//
		// ------------------------------------------------

		public function GestureEvent(type:String, bubbles:Boolean=false, cancelable:Boolean=false){
			super (type,bubbles,cancelable);
		}

		// ------------------------------------------------
		//
		// ---o methods
		//
		// ------------------------------------------------

		public override function clone():Event{
			return new GestureEvent(type, bubbles, cancelable) as Event;
		}

	}
}

I also made a change to the MouseGesture class to make sure that “moves” were being attached to GestureEvents dispatched during the “CAPTURING” phase. The modified function looks like this:

protected function captureHandler(e:TimerEvent):void{

	// calcul dif
	var msx:int=mouseZone.mouseX;
	var msy:int=mouseZone.mouseY;

	var difx:int=msx-lastPoint.x;
	var dify:int=msy-lastPoint.y;
	var sqDist:Number=difx*difx+dify*dify;
	var sqPrec:Number=DEFAULT_PRECISION*DEFAULT_PRECISION;

	if (sqDist>sqPrec){
		points.push(new Point(msx,msy));
		addMove(difx,dify);
		lastPoint.x=msx;
		lastPoint.y=msy;

		if (msx<rect.minx)rect.minx=msx;
		if (msx>rect.maxx)rect.maxx=msx;
		if (msy<rect.miny)rect.miny=msy;
		if (msy>rect.maxy)rect.maxy=msy;
	}
	// event
	var evt:GestureEvent = new GestureEvent(GestureEvent.CAPTURING);
	evt.moves = moves.join("");
	dispatchEvent(evt);

	//dispatchEvent (new GestureEvent(GestureEvent.CAPTURING));

}

The final step was to deal with the entire set of move data in my application. I configured my instance of the MouseGesture class to handle GestureEvent.CAPTURING events and created the following function to deal with them.

private function _captureHandler(e:GestureEvent):void {
	var reShake:RegExp = /(62|26)/g;
	var shakes:int = e.moves.match(reShake).length;
	field_shakes.text = "Number of Shakes: " + shakes;
	if (shakes >= 12) {
		stage.dispatchEvent(new MouseEvent(MouseEvent.MOUSE_UP));
		field_shakes.text = "Congratulations!";
	}
	trace(e.moves);
}

The important thing to note here is the regular expression being used. The strings “62″ and “26″ correspond to the output that would be expected from a vertical shake gesture in the MouseGesture class (respectively “up->down” and “down->up”). What the function does is scan the entire list of movements and look for instances in which the patterns I’m trying to match have occurred in sequence. For each occurrence of either pairing, we know that the user has successfully “shaken” the can. A single call fired to the “_captureHandler” function might have a “moves” list that looks like this…

0666662255222666666222566222266222266672226666122561

…in which each digit corresponds to one of 8 possible directions the user moved his mouse. If you count carefully, you’ll find 8 instances of either the sequence “62″ or the sequence “26.”

Ultimately Mr. Brun’s class ended up being more useful for the project that I’m working on, but I hope to see other implementations in the future that will allow for more complex gestures to be recognized.