Mouse WebCam

Anticipating for a pair of weeks Minority Report, I asked two students to do a Gesture-Driven mouse, using a quickcam and visual basic. The project started in January and the different modules have been first integrated by July 2003, which casually is coincidental with the release of PlayStation II Eye-Toy. So we are in the wave... I have now asked my students to clean the code to put it in the free public domain. If you want to get some clues on visual basic programming and be able to develop EyeToy-like software in your PC, keep watching this page.

Instead of the nice black leather glove with three blue diodes, we use a greenish kitchen glove. This is actually more difficile than Tom Cruise's gadget, as detecting lights is easier than detecting colors. But in this way we avoid extra soldering.

We found that the capture stops when other window event happens, so while mouse can move smoothly (sort of), we need internally relaunch the capture after every event. Also, dragging does not work because the OS takes control. If such feature is needed, you will work at deeper levels of Windows (HID drivers etc.). Fortunately menus and buttons live in upper layers.

To simulate mouse buttons with a gesture, but it seems that a second momenta calculation to fit the ellipse will be enough. On a pentium 4 the analysis takes about 90 milliseconds and we can do it even better, so it seems it will work! Note the small tags next to the images... we use them to disable drawing, which takes a lot of time -half second per image!- in our current, debug, implementation. Ellipse orientation, eccentricity and size are enough parameters to define four states: NOHAND, LEFTBUTTON, RIGHTBUTTON, NOBUTTON.

If you are intrigued, you can download

Let me be serious here: The code is buggy and will not be functional until version 1.0. Or beyond. Do not download it if you are not a programmer or interested on the internals of programming. It involves pointers, callbacks and a lot of obscure tricks, and my students are very young so their code is still very naive and convolute (In any case, you can ask us for help). If you are only interested in the final version, just bookmark this page; or send me an email [arivero at] asking to be notifyed

In any case, the .zip can be compiled in any VisualBasic 6.0, and it contains:

Note that all the provided code is to be considered under Gnu Public License, the owners being Enrique Garcia -the analysis code- and Laura Morentin -all the rest-.

If you are aware of similar projects, please tell me.

Some pointers:

The main source for recipes on getting video from Visual Basic is Ray Mercer. Also check E. J. Bantz.

The strange relationship between bitmaps and picture boxes can be studied in some articles from the Microsoft Base. Check numbers 187568 and 186260 Also a simple fading example can be of some help. Or a reading of our own code.

To understand use of pointers in visual basic, you need to look for some article on Safe Arrays, which are the movable arrays of VB. There was some other good articles in PDF format, but I do not remember where.

Have luck!

Similar Works

The touchstone should be real interaction with a Windows system. This is, you should be able to raise the hand from the keyboard, grasp a window, move the window to other place and then remove the hand down to the keyboard to keep working. You should be authorized to use gesture tricks to keep dragging across high distances in the screen. For instance you can put the other hand over the camera, so "Nomouse", then move the hand with the glove, then release the camera. Other example, just move the hand far back until "nomouse", then move to the initial position, then keep scrolling... your hand does naturally a circle and you are scrolling! We are going to try a C version over a PDA; so the mouse signals will be sent over the USB mimicking a standard USB mouse, thus completely software and platform intependent.