Sixth Sense Technology

SixthSense is a gesture-based wearable computer system developed at MIT Media Lab by Steve Mann in 1994 and 1997
The SixthSense technology contains a pocket projector, and a camera contained in a head-mounted, handheld or pendant-like, wearable device. Both the projector and the camera are connected to a mobile computing device in the user’s pocket. The projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces; while the camera recognizes and tracks users' hand gestures and physical objects using computer-vision based techniques. The software program processes the video stream data captured by the camera and tracks the locations of the colored markers (visual tracking fiducials) at the tips of the user’s fingers. The movements and arrangements of these fiducials are interpreted into gestures that act as interaction instructions for the projected application interfaces. SixthSense supports multi-touch and multi-user interaction.

Example applications

Four colored cursors are controlled by four fingers wearing different colored markers in real time. The projector displays video feedback to the user on a vertical wall.

The projector displaying a map on the wall, and the user controlling it using zoom and pan gestures.

The user can make a frame gesture to instruct the camera take a picture. It is hinted that the photo will be automatically cropped to remove the user's hands.

The system could project multiple photos on a wall, and the user could sort, re-size and organize them with gestures. This application was called Reality Window Manager (RWM) in Mann's headworn implementation of Sixth Sense.^[13]

A number pad is projected onto the user's palm, and the user can dial a phone number by touching his palm with a finger. It was hinted that the system is able to pin point the location of the palm, and that the camera and projector are able to adjust themselves for surfaces that are not horizontal.

The user can pick up a product in supermarket (e.g. a package of paper towels), and the system could display related information (e.g. the amount of bleach used) back on the product itself.

The system can recognize any book picked up by the user and display Amazon rating on the book cover.

As the user opens a book, the system can display additional information such as reader's comments.

The system is able to recognize individual pages of a book and display annotation by the user's friend. This demo also suggested the system would be able to handle tilted surface.

The system is able to recognize newspaper articles and project the most recent video on the news event on a blank region of the newspaper.

The system is able to recognize people by their appearances and project a word cloud of related information retrieved from the internet on the person's body.

The system is able to recognize a boarding pass and display related information such as flight delay and gate change.

The user can draw a circle on his or her wrist, and the system will project a clock on it.