Software

Here is the software to run on the mobile computing device-

v0.1

As always, I would love to hear any comments, questions, complaints, or just anything you have to say about the device.

How To Assemble

If you have your components for this device already (listed in the last post, go on now, read it if you haven't), you'd know what pieces to have to build the sixth sense device. Check out the picture below for how to connect the components, and if you still do not understand it, I will have a description underneath the picture.


-The WEBCAM connects to the MOBILE COMPUTING DEVICE
-The PROJECTOR connects to the MOBILE COMPUTING DEVICE
-The MICROPHONE is already connected to the WEBCAM if you've bought the suggested components
-The MIRROR is just positioned and angled under the PROJECTOR
-The MOBILE COMPUTING DEVICE is just in your pocket or backpack and should be running the software which I will give out soon

If you have any questions, comments, or concerns, I would love to hear any feedback you have

Where To Buy

If you want to know where to get the components to build this device, here are some links to them. I suggest buying these brands for the hardware of the sixth sense machine. There will also be a description of what each part does.

Camera

The camera is the key input device of the SixthSense system. The camera acts as a digital eye of the system. It basically captures the scene the user is looking at. The video stream captured by the camera is passed to mobile computing device which does the appropriate computer vision computation. The major functions of the camera can be listed as:

• Captures user’s hand movements and gestures (used in reorganization of user gestures)
• Captures the scene in front and objects the user is interacting with (used in object reorganization and tracking)
• Takes a photo of the scene in front when the user performs a ‘framing’ gesture
• Captures the scene of projected interface (used to correct the alignment, placement and look and feel of the projected interface components)

Suggested Product: Logitech QuickCam Pro for notebooks

Projector

The projector is the key output device of the SixthSense system. The projector visually augments surfaces, walls and physical objects the user is interacting with by projecting digital information and graphical user interfaces. The mobile computing device provides the projector with the content to be projected. The projector unit used in prototype runs on a rechargeable battery. The major functions of the projector can be listed as:

• Projects graphical user interface of the selected application onto surfaces or walls in front
• Augments the physical objects the user interacting with by projecting just-in-time and related information from the Internet

Suggested Products: You can buy either laser (AAXA, Microvision) or L.E.D (3M MPro110) projectors.

Mirror

The mirror reflects the projection coming out from the projector and thus helps in projecting onto the desired locations on walls or surfaces. The user manually can change the tilt of the mirror to change the location of the projection. For example in application where the user wants the projection to go on the ground instead of the surface in front, he can change the tilt of the mirror to change the projection. Thus, the mirror in the SixthSense helps in overcoming the limitation of the limited projection space of the projector.

Suggested Product: Any 1”X1” first surface mirror

Microphone

The microphone is an optional component of the SixthSense. It is required when using a paper as a computing interface. When the user wants to use a sheet of paper as an interactive surface, he or she clips the microphone to the paper. The microphone attached this way captures the sound signals of user’s touching the paper. This data is passed to computing device for processing. Later, combined with the tracking information about user’s finger, the system is able to identify precise touch events on the paper. Here, the sound signal captured by the microphone provides time information whereas the camera performs tracking. The applications enabled by this technique are explained earlier.

Suggested Product: Microphone of Logitech QuickCam pro for notebooks

Mobile computing device

The SixthSense system uses a mobile computing device in user’s pocket as the processing device. The software program enabling all the features of the system runs on this computing device. This device can be a mobile phone or a small laptop computer. The camera, the projector and the microphone are connected to this device using wired or wireless connection. The detail of the software program that runs on this device is provided in next section. The mobile computing device is also connected to the Internet via 3G network or wireless connection.

Suggested Product: Any Windows computer 

Sixth Sense: What Is It?

Anyone ever heard of a sixth sense device? Believe it or not, a device has been created that recognizes and tracks users hand gestures and physical objects using computer-vision based techniques. Cool, huh? And the price? Nothing close to what you'd think. Couple thousand? Nope. Try couple hundred. Only around $350 to make this wicked device. All you need is a webcam (preferably with a built in microphone); a pocket projector; a mirror; some velcro to connect everything; red,blue, green and yellow tape; and a mobile computing device.


That guy above is Pranav Mistry. That thing around his neck? The sixth sense device.He invented it by asking himself a question- "How can we leverage our knowledge about everyday objects and how we use them to interact with the digital world?", which basically means how can we use what we know about everyday objects and the digital world and combine the two? Take a look at this interview and see how he describes it- http://www.ted.com/talks/pranav_mistry_the_thrilling_potential_of_sixthsense_technology.html