Mouse WebCam

Anticipating for a pair of weeks Minority Report, I asked two students to do a Gesture-Driven mouse, using a quickcam and visual basic. The project started in January and the different modules have been first integrated by July 2003, which casually is coincidental with the release of PlayStation II Eye-Toy. So we are in the wave... I have now asked my students to clean the code to put it in the free public domain. If you want to get some clues on visual basic programming and be able to develop EyeToy-like software in your PC, keep watching this page.

Instead of the nice black leather glove with three blue diodes, we use a greenish kitchen glove. This is actually more difficile than Tom Cruise's gadget, as detecting lights is easier than detecting colors. But in this way we avoid extra soldering.

We found that the capture stops when other window event happens, so while mouse can move smoothly (sort of), we need internally relaunch the capture after every event. Also, dragging does not work because the OS takes control. If such feature is needed, you will work at deeper levels of Windows (HID drivers etc.). Fortunately menus and buttons live in upper layers.

To simulate mouse buttons with a gesture, but it seems that a second momenta calculation to fit the ellipse will be enough. On a pentium 4 the analysis takes about 90 milliseconds and we can do it even better, so it seems it will work! Note the small tags next to the images... we use them to disable drawing, which takes a lot of time -half second per image!- in our current, debug, implementation. Ellipse orientation, eccentricity and size are enough parameters to define four states: NOHAND, LEFTBUTTON, RIGHTBUTTON, NOBUTTON.

If you are intrigued, you can download

early 0.75 version, July 2003
final (sort of) 1.00 version, March 2004
Student Report for work on windows internals, Word doc (Spanish, 5MBytes).
Student Report for analysis section, zipped Word doc
PDF slide for the analysis presentation (to come)

Let me be serious here: The code is buggy and will not be functional until version 1.0. Or beyond. Do not download it if you are not a programmer or interested on the internals of programming. It involves pointers, callbacks and a lot of obscure tricks, and my students are very young so their code is still very naive and convolute (In any case, you can ask us for help). If you are only interested in the final version, just bookmark this page; or send me an email [arivero at posta.unizar.es] asking to be notifyed

In any case, the .zip can be compiled in any VisualBasic 6.0, and it contains:

Some common libraries
A capture module that feeds captured bitmaps into an analysis module.
An analysis form with edge detection routines for the whole image and also center of mass and position for a hand wearing a glove.
We are moving definitively to first and second moments analysis (center of mass and ellipse fit), to determine both the position of the hand and if it is open or closed. A closed hand is a "mousedown click", so one can grasp a object and drag it across the screen. We are keeping edge detection in the analysis code window just as an example for other alternatives, but we are not using it.
A routine to select the camera driver
A main form to start capturing and moving the mouse according to the results of analisis. You could use this to interact with other programs. Currently one needs to wear a greenish kitchen glove. Still, it is better/cheaper than the three-diode glove used by Tom Cruise in the film.
A secondary form to load still images for analysis.
Some still images, sometimes.
A lot of bugs.

Note that all the provided code is to be considered under Gnu Public License, the owners being Enrique Garcia -the analysis code- and Laura Morentin -all the rest-.

If you are aware of similar projects, please tell me.

Some pointers:

The main source for recipes on getting video from Visual Basic is Ray Mercer. Also check E. J. Bantz.

The strange relationship between bitmaps and picture boxes can be studied in some articles from the Microsoft Base. Check numbers 187568 and 186260 Also a simple fading example can be of some help. Or a reading of our own code.

To understand use of pointers in visual basic, you need to look for some article on Safe Arrays, which are the movable arrays of VB. There was some other good articles in PDF format, but I do not remember where.

Have luck!

Similar Works

The touchstone should be real interaction with a Windows system. This is, you should be able to raise the hand from the keyboard, grasp a window, move the window to other place and then remove the hand down to the keyboard to keep working. You should be authorized to use gesture tricks to keep dragging across high distances in the screen. For instance you can put the other hand over the camera, so "Nomouse", then move the hand with the glove, then release the camera. Other example, just move the hand far back until "nomouse", then move to the initial position, then keep scrolling... your hand does naturally a circle and you are scrolling!

A recent (2004) gesture recognition bare hand
A veteran (1996) Hand-gesture project is ZYKLOP
iGesture pad, from Fingerworks.
Manor's project at MIT, two handed, no gestures?
Virtual Devices' keyboard
MouseVision' VisualMouse uses any USB camera to track the head.
EagleEyes'CameraMouse also tracks head or fingers. In this kind of devices there are no gestures suport, so they usually include a toolbar to emulate the clicks. Insterestingly, Eagleayes Project also develops an electrode system for muscle-driven input, which surely could also be used to read fingering gestures.
The Camera Mouse project was started by Betke J. Gips, http://www.cs.bu.edu/fac/betke/, and their students.
between the head approaches, I like very much the one with the nose, Nouse.
I finally checked the Sony PS2 device. At the end, they are using the fastest alternative: change detection. It simply detects the differences between two images, such diferences conform the "active areas". I'd preferred to use edge detection for it, but it is true that "edging" is slower than moving. In fact we need to disable it when our cam is running in real time mouse emulation mode. If you want to program change detection, you will see it is really easy, simply copy the previous bitmap to some global variable in the analysis module. It is just that it is not useful to emulate a mouse, as one risks to detect a lot of simultaneous changes. One could emulate a "whole body mouse", perhaps. But no very useful thing.
Sylvie Noel keeps a blog page on interface issues.
The diode trick has been essayed by the guys at MultiBlah site.

We are going to try a C version over a PDA; so the mouse signals will be sent over the USB mimicking a standard USB mouse, thus completely software and platform intependent.