“Minority Report” interface using Flash and FLAR toolkit

As part of the RoomWare workshop end of August 2009 I will be introducing a “Minority report” interface using Flash and FLAR toolkit.

It is part of a 2 type interfacing strategy in Flash, including:

1. Gestural interfacing using your hands ad fingers in front of a screen
2. Table interfacing using a transparant table-top, objects and a fixed camera

This article is about 1: the Gestural interfacing implementation.

The “Virtual reality glove”

Code

Download the ZIP file. Beware: you compile it with a Flash CS4 file (in the “fla” folder) and this is still my dirty prototyping code. It uses the Squidder demo as a basis and adds the “Minority Report” dynamics on top.

Feel free to experiment with it and build your own things. Share with me what you created by posting your links in the comment section below. I am very happy to learn to what levels you will take it to.

Demo video

Do it yourself here.

Preface: limited, low cost and easy to implement

This solution is not intended to be an industry-strength type solution. It is very limited and dependent of light and you still need markers to make the gestures (where recent solutions using more powerful environments than flash recognizes hands, fingers and gestures). It is also very low cost and very easy to implement.

In contrary to the more fancy solutions you only need this to make your own “Minority Report” interface:

1. A computer
2. A web cam
3. Paper
4. A printer

The AS3 class

The class is written in such a way that you can integrate it fast and simple in your project and so that you can build your own strategies for gestures and gesture based interfacing.

Basics

Distance and pressure

The Distance of the FLAR toolkit fiducial to the camera is translated into “pressure”. This pressure area has a range which can be set per object. Using the “range” you can determine when the object will start to apply pressure to the “camera”.

Rotation and tilt

Each FLAR toolkit fiducial also returns “rotation” and “tilt”. Tilt is on the “X” axis or the “Y” axis. You can translate the titl to “pressure” as well.

Putting it together: Rotation, tilt and pressure

Research

In the past week I have tested several scenario’s and combinations, filtering out the most usable set of movements using your hands or fingers in free space (the “Minority report” interface)

Visual feedback

It helps to give your user visual feedback, so that the result of a movement can be predicted and the sense of “I am in control” is optimal. The below images shows you the visual language I used for testing.

Pressure (towards the camera)

Pressure is visualized by a circle within the pointer. The farther away, the less the pressure is. The closer to the camera, the bigger the pressure. I started using pressure to “grab” and “activate” things, but soon discovered that this excluded other strategies where I use “pressure” to tilt an object in a 3D space.

In code you use pressure (called Zstrength) like this:

   var nrm:FlarTransformNormalizer=normalizedObject; // The normalized information from a FLAR object
   var stepSize:Number=5;
   _yStep=nrm.Zstrength * stepSize * nrm.deltaY; // Uses the pressure towards the camera to define the strength of a movement

Rotation

I translated rotation to a “dial”. When you point your finger like a pistol to the camera, the dial is on “neutral”. When you rotate left or right, the dial is rotated likewise. Rotating turned out to be the most effective way to apply “pressure” or variable “impulse” to an object while maintaining the freedom to move your fingers into a 3D space. The green area is the threshold which will be discussed below.

In code, the rotation is gained like this:

   var nrm:FlarTransformNormalizer=normalizedObject; // The normalized information from a FLAR object
   var myRotation==nrm.rotation // returns a value between 0 and 360

There is no “mouse button”

To have no “mouse button” introduces a practical problem. How do you tell the system that it should perform an action or not? I found the easiest way to deal with the “no mouse button” to use a “neutral zone” which spans an angle of 10 degrees left and right. When the angle of the pointer is in this area, nothing happens.

Solution: rotation = impulse

Sometimes you want to do precision work, the other time you want to make big movements with minimal effort. The more you turn the dial, the bigger the impulse will be of your movement. See the video on YouTube for the implementation. In code it looks like this:

   var nrm:FlarTransformNormalizer=normalizedObject; // The normalized information from a FLAR object
   var stepSize:Number=5;
   _yStep=nrm.dialStrength * stepSize * nrm.deltaY; // Uses the dialstrength to define the strength of a movement

dialStrength already includes a treshold using nrm.dialStrengthTreshold. The default value is 10.

Movement + rotation left/right = direction

Again: the system registers only movement. When you want to drag an object on screen, you need to cancel out the directions the user does not want to move to. Otherwise it will “stick” to your pointer. I tried using pressure as “you make a circular movement towards the camera when dragging something” and found out it was not that simple. See the YouTube video to see how this works in practice.

Turning the dial lef/right is. The red semi-circle shows your range of movement. Dialing left means you can drag objects to the left hand side and up/down. Dialing to the right will move the object to the right. Movements towards the “white” area are neglected.

Tilt

Tilt is an interesting one. It turned out to be hard to use in practice in combination with other movements, unless you use two hands. With tilt, you apply “pressure” either to the right-hand, left hand, upper or bottom side. Like a joystick. To maintain these positions while doing other movements however, require a longer learning curve. For that reason I discarded tilt for the “Minority Report” interface.

Combined actions

Until now I have only discussed single handed actions. By “pairing” FLAR objects you can create combined actions. In the video on YouTube you can see how I rotate and scale the image based on two-handed actions. (The image above is rotated 90 degrees)

Again, the rotation of each FLAR object is used to define the “strength” of an action, represented by the thickness of the connecting line. One finger/hand represents one action (rotate object) the other and the other action (scale object).