ABSTRACT 'SixthSense' is a wearable gestural interface that augments the physical world around us with digital information and

lets us use natural hand gestures to interact with that information. All of us are aware of the five basic senses seeing, feeling, smelling, tasting and hearing. But there is also another sense called the sixth sense. It is basically a connection to something greater than what their physical senses are able to perceive. To a layman, it would be something supernatural. Some might just consider it to be a superstition or something psychological. But the invention of sixth sense technology has completely shocked the world. Although it is not widely known as of now but the time is not far when this technology will change our perception of the world. Pranav Mistry, 28 year old, of Indian origin is the mastermind behind the sixth sense technology. The device sees what we see but it lets out information that we want to know while viewing the object. It can project information on any surface, be it a wall, table or any other object and uses hand / arm movements to help us interact with the projected information. The device brings us closer to reality and assists us in making right decisions by providing the relevant information, thereby, making the entire world a computer. The SixthSense prototype implements several applications that demonstrate the usefulness, viability and flexibility of the system. The map application lets the user navigate a map displayed on a nearby surface using hand gestures, similar to gestures supported by Multi-Touch based systems, letting the user zoom in, zoom out or pan using intuitive hand movements.The user can stop by any surface or wall and flick through the photos he/she has taken. SixthSense also lets the user draw icons or symbols in the air using the movement of the index finger and recognizes those symbols as interaction instructions. For example, drawing a magnifying glass symbol takes the user to the map application or drawing an @ symbol lets the user check his mail. The SixthSense system also augments physical objects the user is interacting with by projecting more information about these objects projected on them. The current prototype system costs approximate $350 to build

INTRODUCTION We've evolved over millions of years to sense the world around us. When we encounter something, someone or some place, we use our five natural senses to perceive information about it; that information helps us make decisions and chose the right actions to take. But arguably the most useful information that can help us make the right decision is not naturally perceivable with our five senses, namely the data, information and knowledge that mankind has accumulated about everything and which is increasingly all available online. Although the miniaturization of computing devices allows us to carry computers in our pockets, keeping us continually connected to the digital world, there is no link between our digital devices and our interactions with the physical world. Information is confined traditionally on paper or digitally on a screen. SixthSense bridges this gap, bringing intangible, digital information out into the tangible world, and allowing us to interact with this information via natural hand gestures. SixthSense frees information from its confines by seamlessly integrating it with reality, and thus making the entire world your computer. Sixth Sense technology is the science of tomorrow with the aim of connecting the digital world with the physical world seamlessly, eliminating hardware devices. Mistrys flirtations with the digital world began in the early 2000s when he pieced together four mouse rollers with pulleys and springs to give shape to a motion sensing device.

It is similar to Telepointer, a neck worn projector/camera system developed by Media Lab student Steve Mann (which Mann originally referred to as "Synthetic Synesthesia of the Sixth Sense"). What the researchers have done is combine a number of standard gadgets including a webcam, projector, and mobile phone, to form a brand new interaction experience. In its current form the battery-powered projector is attached to a hat, the webcam is hung around the neck (or also positioned on a hat), and the mobile phone provides the connection to the Internet. Earlier SixthSense Prototype

Recent Prototype

The wearer uses hand gestures combined with the gadgets to perform actions. So, for example, they could make a circle over their wrist with the fingers of one hand. WUW would recognize this action and project a clock face on to their wrist. Make a picture frame with both your hands and WUW will take a picture as if it was a camera. Stand near a wall and you can get a projected desktop allowing you to open applications, view the pictures youve taken, or surf the Net. With the invention of intuitive computer interfaces, the digital and physical worlds came together closer than ever. One just has to place an object (anything from a flight boarding pass to a key) on a Tangible Public Map (TaPuMa) on its horizontal smart screen to get inside-out information about it. Keep a coffee cup on a particular spot on the map and a whole range of cafs in the area is brought to you on a platter. All you have to do to get seamlessly connected with the digital world is wear a simple pendant-like equipment consisting of a camera and a portable battery-powered projection system with a mirror (a more modish version is just round the corner). The device (now known as the Sixth Sense Device) when connected to a cell phone acts as a computation and communication tool. The camera tracks hand gestures and helps gather meta information (information from the surroundings) and articulates it with the digital domain. Wearing marker caps on ones fingers (a more
4

stylish option is to paint the said fingernails in a different colour each) and making gestures with them, one can use any interface (Yes! Any interface! No longer is the human race tied to the bulky world of computer screens) to access and modify data. Clicking a picture is as easy as conjuring up a rectangle in the air aimed at the object of visual desire with the thumb and index fingers. A few finger motions help edit and resize pictures and another set of gestures later, the pictures find themselves E-mailed to recipients. The three basic components of an augmented reality system are the head-mounted display tracking system mobile computer for the hardware

. The main goal of this new technology is to merge these three components into a highly portable unit much like a combination of a high tech Walkman and an ordinary pair or eyeglasses. The head-mounted display used in augmented reality systems will enable the user to view superimposed graphics and text created by the system. Another component of an augmented reality system is its tracking and orientation system. This system pinpoints the user's location in reference to his surroundings and additionally tracks the user's eye and head movements. Augmented reality systems will need highly mobile computers. As of now many computers arent there to satisfy to provide this option

Gesture Recognition:
It is a technology which is aimed at interpreting human gestures with the help of mathematical algorithms. Gesture recognition technique basically focuses on the emotion recognition from the face and hand gesture recognition. Gesture recognition technique enables humans to interact with computers in a more direct way without using any external interfacing devices. It can provide a much better alternative to text user interfaces and graphical user interface which requires the need of a keyboard or mouse to interact with the computer. Interfaces which solely depends on the gestures requires precise hand pose tracking. In the early versions of gesture recognition process special type
5

of hand gloves which provide information about hand position orientation and flux of the fingers. In the SixthSense devices coloured bands are used for this purpose. Once hand pose has been captured the gestures can be recognised using different techniques. Neural network approaches or statistical templates are the commonly used techniques used for the recognition purposes. This technique have an high accuracy usually showing accuracy of more than 95%. Time dependent neural network will also be used for real time recognition of the gestures

Computer Vision:
Computer vision is the technology in which machines are able to interpret/extract necessary information from an image. Computer vision technology includes various fields like image processing, image analysis and machine vision. It includes certain aspect of artificial intelligence techniques like pattern recognition. The machines which implement computer vision techniques require image sensors which detect electromagnetic radiation which are usually in the form of ultraviolet rays or light rays. The computer vision find itself applicable in varies field of interest. One such field is bio medical image processing. Its also used in autonomous vehicle like SUVs. The computer vision technique basically includes four processes

Recognition:
One of the main task of computer vision technique is to determine whether the particular object contain the useful data or not.Motion Analysis: Motion analysis includes several tasks related to estimation of motion where an image sequence is processed continuously to detect the velocity at each point of the image or in the 3D scene.Scene Reconstruction: Computer vision technique employs several methods to recreate a 3D image from the available images of a scene.

Image Restoration:
The main of aim of this step is to remove noise from an given image. The simplest method involves using simple filters like low pass or median filters. In order to get better quality images more complex methods like In painting are used.

Radio Frequency Identification:

6

Radio frequency identification systems transmit the identity of an object wirelessly, using radio magnetic waves. The main purpose of a radio frequency identification system is to enable the transfer of a data via a portable device. The portable device is commonly known as tag. The data send by the tag is received and processed by a reader according to the needs of the application. The data send by the tag contains various informations identification or location of the information, or specifics about the product that has been tagged, for example price, colour, date of purchase, etc. This technology gained importance because of its ability to track moving object. A typical radio frequency tag consists of a microchip attached to a radio antenna which is mounted on a substrate. To retrieve the data from the tag a reader is needed. A typical radio frequency reader consists of two antennas that emit radio waves and at the same are capable of accepting the signals from the tag. The reader then passes the information that it has received to a computer device in digital form. The computer then interprets this digital data and processes it. Radio frequency identification techniques are widely used in the fields like asset tracking, supply chain management, manufacturing, payment systems etc. One of the major advantages of devices using radio frequency technology over other similar devices is that RFID devices need not be positioned precisely relative to the scanner. But till then there are certain areas of concern for this technology. Some problem related to this technology is tag collision and reader collision. Usually the reader collision occurs when the signals from two or more readers overlap, while tag collision occurs when many tags are present in a small area

THE MASTER MIND Pranav Mistry, 28 year old, of Indian origin is the mastermind behind the sixth sense technology. Born in Palanpur, Gujarat he received his Bachelors of technology in computer science and engineering from India and later received masters in design from MIT Boston. He invented Sixth Sense / WUW ( Wear UR World) which is a wearable gestural , user friendly interface which links the physical world around us with digital information and uses hand gestures to interact with them. Pranavs eagerness to pop pixels out of the digital world into the real world led to the birth of Sixth Sense technology. Currently he is a PhD student at MIT and he won the Invention of the Year 2009 - by Popular Science. The device sees what we see but it lets out information that we want to know while viewing the object. It can project information on any surface, be it a wall, table or any other object
7

and uses hand / arm movements to help us interact with the projected information. The device brings us closer to reality and assists us in making right decisions by providing the relevant information, thereby, making the entire world a computer.

SIXTH SENSE:THE DEVICE

Sixth Sense is a mini-projector coupled with a camera and a cellphonewhich acts as the computer and your connection to the Cloud, all the information stored on the web. The hardware components are coupled in a pendant like mobile wearable device. Both the projector and the camera are connected to the mobile computing device in the users pocket.

Software: Software program processes the video stream data captured by the camera Pranav Mistry says the SixthSense software will be open source. As far as this seems to be a little set of items, there will not be user interfaces or much advanced programs for the users. There will be much harder and secured coding inside the device to make sure the security of the software. It will be interesting to know the new language for coding for a SixthSense device. The software works on the basis of computer vision. Theres a small camera acting as your eye, your third eye, your digital eye, connecting you to the world of digital information. Processing is happening in your mobile phone, and basically works on computer vision algorithms that we developed, taking advantage of some open-source code but mainly writing code at the lab.. We wrote 50,000 lines of code. The software recognizes 3 kinds of gestures:

multi-touch gestures, like the ones you see in Microsoft Surface or the i-Phone where you touch the screen and make the map move by pinching and dragging.

call freehand gestures, like when you take a picture Or you might have noticed , do a namaste gesture to start the projection on the wall.
9

iconic gestures, drawing an icon in the air. Like, Whenever we draw a star, show me the weather. When I draw a magnifying glass, show me the map.

You might want to use other gestures that you use in everyday life. This system is very customizable. Because its the choice, but the user need not change their habits. The Sixth Sense to changes for them. Hardware: In order to control the SixthSense, it requires some advance hardware as it appears to be. In some of the public presentations, the presenter was wearing some controlling devices including color markers, Camera and projector. They has to be compact and easily controllable. However the hardware integration of SixthSense technology is quite innovative since they have manage to develop camera and pen like day to day objects. Projector: . The projector is the key output device of the SixthSense system. The projector visually augments surfaces, walls and physical objects the user is interacting with by projecting digital information and graphical user interfaces. The mobile computing device provides the projector with the content to be projected. The projector unit used in prototype runs on a rechargeable battery.

The major functions of the projector can be listed as: Projects graphical user interface of the selected application onto surfaces or walls in front

10

Augments the physical objects the user interacting with by projecting just-in-time and related information from the Internet

Camera. The camera is the key input device of the SixthSense system. The camera acts as a digital eye of the system. It basically captures the scene the user is looking at. The video stream captured by the camera is passed to mobile computing device which does the appropriate computer vision computation. The major functions of the camera can be listed as: Captures users hand movements and gestures (used in reorganization of user gestures) Captures the scene in front and objects the user is interacting with (used in object reorganization and tracking) Takes a photo of the scene in front when the user performs a framing gesture Captures the scene of projected interface (used to correct the alignment, placement and look and feel of the projected interface components)

Colored markers: (visual tracking fiducials) placed at the tip of the users fingers using simple computer-vision techniques helps the webcam to track the movement of fingers. The movements and arrangements of
11

these fiducials are interpreted into gestures that act as interaction instructions for the projected application interfaces

Mirror: The mirror reflects the projection coming out from the projector and thus helps in projecting onto the desired locations on walls or surfaces. The user manually can change the tilt of the mirror to change the location of the projection. For example in application where the user wants the projection to go on the ground instead of the surface in front, he can change the tilt of the mirror to change the projection. Thus, the mirror in the SixthSense helps in overcoming the limitation of the limited projection space of the projector.

Mobile device It may be a laptop, PDA, smart phones etc. These are connected to other hardware devices and sends information to projector for projection. The important thing is that the device is a mobile device. Means, it is so light that we can take it with us where ever we want to. It is as small as a cell phone and is so simple to use.

12

WORKING Sixth sense device analyses what user sees and visually augments the surfaces and physical objects user is interacting with. What the researchers have done is combine a number of standard gadgets including a webcam, projector, and mobile phone, to form a brand new interaction experience. The key here is that Sixth Sense recognizes the objects around you, displaying information automatically and letting you access it in any way you want, in the simplest way possible

13

The technology in itself is nothing more than the combination of some stunning technologies, but the idea of combining those technologies is really great. The technology is mainly based on hand gesture recognition, image capturing, processing, and manipulation, etc. The camera is used to recognize and track users hand gestures and physical objects using computer-vision based techniques, while the projector is used to project visual information on walls or on any physical thing around us. Other hardware includes mirror and colored caps to be used for fingers. The software of the technology uses the video stream, which is captured by the camera, and also tracks the location of the tips of the fingers to recognize the gestures. This process is done using some techniques of computer vision. Basically it is a device which is a mini projector and which can be projected on any surface, it carries the information stored in it and also collects information from the web. It is the one which obey hand gestures of yours and gives you what you want to see and know.

It is the combined technology of computer along with cell phone. It works when a person hang it on his neck and start projecting through the micro-projector attached to it. Your fingers works like the keyboard as well as the mouse.

14

POTENTIAL APPLICATIONS In Pranav Mistrys demonstration, he showed several applications that can be made using the SixthSense technology. It would be a great achievement for the Information Technology as well. Since it is the beginning of the SixthSense applications, there will be lot of tough problems ahead the SixthSense application developers. Motion Capture The innovative camera is something that got applauds of the crowd in the demonstration. Four rubber rings in the four fingers two in each can be used do draw a rectangular. It will be the area covered by the photograph. Then the photos can be taken to any interface for editing, managing the galleries and sharing. This is much controversial. There is an inbuilt storage device where the data can be stored. When using a surface, the whole data is taken into that surface and stored in the device after the editing. And also the capturing device being so small like 4 rubber rings can be used to capture the photos of any location. Example using 2 fingers to send 5 seconds, 3 fingers to send 10 seconds likewise it can be done. There are so many new applications to be developed in the near future. A remote controller for media devices can be implemented to be worked with fingers using this technology. Except having the remote controller to send fast forward, rewind or next or previous options, a SixthSense device can be used. The current process of sending fast-forward or rewind with advanced options. Many games can be made using this technology. A car race using an imaginary steering wheel will be an attractive application. The player will have to drive the vehicle as a real one using his arms. The SixthSense tool kit will be a good replacement for the XBOX game kit and it will not be much expensive as it is said. Other potential applications include: Capture photos with fingers: Why to take camera on your holiday and no tension for the photo space as this Sixth Sense computer will work like your camera. It captures the photo, when you make a square with your fingers, highlighting which one you want to frame.
15

Drawing application: The drawing application lets the user draw on any surface by tracking the fingertip movements of the users index finger

16

Phone Call: You can call to your friend by typing the numbers on your hand. It displays the keypad of the phone over your palm and the keys appear on the four finger. Use your other hand's finger to press the keys.

Zooming features: We can zoom in an zoom out a picture

17

Check the brand of the Product: It helps you to choose the best brand product from the super market.

18

Read Books easily: Check out the ratings of the Book you are going to buy, it checks the ratings from the internet. And another amazing thing is that it reads the book for you.

Newspapers: Did you saw the moving pictures of the Newspaper in the movie Harry Patter, it is quite similar to it. It Searches the most appropriate video from the web by seeing the headlines or the caption of the News report.

19

Check your Flight Status: You can check the status of the flight while you are on Taxi. Just place the ticket in front of the projector and it checks its status from the internet.

See time without wearing wrist watch: Watches are ancient history now with the prevalence of cell phones. With sixth sense technology, it is just drawing a circle on the wrist away from catching the current time.

20

RELATED TECHNOLOGIES
Augmented reality is a term for a live direct or indirect view of a physical real world environment whose elements are augmented by virtual computer generated imagery.Gesture recognition is a topic in computer science and language technology with the goal of interpreting human gestures via mathematical algorithms.Computer vision is the science and technology of machines that see. It is concerned with the theory behind artificial systems that extract information from images.Radio Frequency Identification is basically an electronic tagging technology that allows the detection, tracking of tags and consequently the objects that they are affixed to.

ADVANTAGES OF SIXTH SENSE DEVICES Portable:

One of the main advantages of the sixth sense devices is its small size and portability. It can be easily carried around without any difficulty. The prototype of the sixth sense is deigned in a such a way that it gives more importance to the portability factor. All the devices are light in weight and the smart phone can easily fit into the users pocket

Support multi touch and multi user interaction:

Multi touch and multi user interaction is another added feature of the sixth sense devices. Multi sensing technique allows the user to interact with system with more than one than one finger at a time.. Sixth sense devices also in-cooperates multi user functionality. This is typically useful for large interaction scenarios such as interactive table tops and walls.

Cost effective:
The cost incurred for the construction of the sixth sense proto type is quiet low. It was made from parts collected together from common devices. And a typical sixth sense device cost up to $300. The sixth sense devices have not been made in large scale for commercial purpose. Once that happens its almost certain that the device will cost much lower than the current price

Connectedness between real world and digital world:

Forming a connection between the real world and the digital world was the main aim of the sixth sense technology

Data access directly from the machines in real time:

21

With help of a sixth sense device the user can easily access data from any machine at real time speed. The user doesnt require any machine-human interface to access the data. The data access through recognition of hand gestures is much easier and user friendlier compared to the text user interface or graphical user interface which requires keyboard or mouse

Mind map the idea anywhere:

With the advent of the sixth sense device, requirement of a platform or a screen to analyse and interpret the data has become obsolete. We can project the information into any any surface and can work and manage the data as per our convenience . Open source software: The software that is used to interpret and analysis the data collected by the device is made open source. This enables other developers to contribute to the development of the device.

Upcoming Projects Making Use Of the Sixth Sense Technology Mouse less:
Mouse less is a concept which is developed in order to avoid the usage of mouse in computers. It helps to provide a familiarity of the physical mouse without actually requiring a physical mouse Even though the computer hardware technology has been developed to a greater extent, still there has been no proper substitute for a physical mouse. This new invention aims to remove the requirement of having a physical mouse altogether but still it provides the feeling of a physical mouse that users are familiar with. It basically consists of an Infrared laser beam and an Infrared camera. Both laser and camera are embedded together with the computer. The laser beam module is modified with a line cap and placed such that it creates a plane of IR laser just above the surface the computer sits on. The user cups their hand, as if a physical mouse was present underneath, and the laser beam lights up the hand which is in contact with the surface. The IR camera detects those bright IR blobs using computer vision. The change in the position and arrangements of these blobs are interpreted as mouse cursor movement and mouse clicks

Third Eye
Third Eye is a new technique that enables multiple viewers to see different things on a same display screen at the same time With thirdEye, - We can have a public sign board where a Japanese tourist sees all the instructions in Japanese and an American in English. - We don't need to have the split screen in games now. Each player can see his/her personal view of the game on the TV screen. - Two people watching TV can watch their favorite channel on a single TV screen. - A public display can show
22

secret messages or patterns. - In the same movie theater, people can see different end of a suspense movie

Inktuitive:
Even though there has been a tremendous development of computer aided designing tools, still paper and pencil is the preferred by most of the designers ,especially in the earlier stages of designing .The project inktuitive is aimed to combine the process of creation that is inherent in paper and pencil with the power of computing that the digital design tools provide. It also extends the natural work-practice of using physical paper by giving the pen the ability to control the design in physical three-dimensional space, freeing it from its tie to the paper. There always remains an intuition of pen and paper ,but the lines that have been drawn are be captured and translated into shapes in the digital world. The physical paper is augmented with overlaid digital strokes. The platform provides a novel interaction mechanism for drawing and designing using above the surface pen movements. In other words inktuitive can be considered as an intuitive physical design workspace that aims to bridge the gap and bring together the conventional design tools such as paper and pencil with the power and convenience of the digital tools for design.<br

TaPuma:
TaPuma is the short form of Tangible Public Map. It is another attempt to bring the digital world nearer to the real world by exploring the techniques of sixth sense technology. Key word based search mechanism has vastly influenced and modified the ways in which we search informations. The keyword based search allows the user to filter out the required data from the vast data that are available on search engines such as yahoo and google. TaPuMa allows people to use their own belongings, the objects they usually carry with them to access relevant and just-in-time information and locations of places from a public map. The project also explores and analyzes the advantages and challenges of this novel interaction mechanism, where real life objects serve as the interface for information acquirement. The broad concept behind the project TaPuMa is Object Amelioration, where the functions of everyday objects can be expanded by using their affordances or functionalities in a variety of different contexts

23

CONCLUSION Using this technology Finally one gets a taste of the world of Harry Potter.

Sixth Sense technology is the science of tomorrow with the aim of connecting the digital world with the physical world seamlessly, eliminating hardware devices. Mistrys flirtations with the digital world began in the early 2000s when he pieced together four mouse rollers with pulleys and springs to give shape to a motion sensing device. This device interprets gestures made in the physical world and replicates them in the digital world. Next he gave the omni-present sticky note a digital makeover. Digitized sticky notes (or Quickies) set as reminders can jog your memory regarding tasks through an SMS or E-mail. Jotting down a question on a quickie prints you the answer. All it takes to message someone on their cell phone is write a note on a quickie. Next in tow for Mistry was a pen that can draw in 3D (a technology popular by the name Inktuitive). With the invention of intuitive computer interfaces, the digital and physical worlds came together closer than ever. One just has to place an object (anything from a flight boarding pass to a key) on a Tangible Public Map (TaPuMa) on its horizontal smart screen to get inside-out information about it. Keep a coffee cup on a particular spot on the map and a whole range of cafs in the area is brought to you on a platter. Pranavs eagerness to pop pixels out of the digital world into the real world led to the birth of Sixth Sense technology. All you have to do to get seamlessly connected with the digital world is wear simple pendant-like equipment consisting of a camera and a portable battery-powered projection system with a mirror (a more modish version is just round the corner). The device (now known as the Sixth Sense Device) when connected to a cell phone acts as a computation and communication tool. The camera tracks hand gestures and helps gather meta information (information from the surroundings) and articulates it with the digital domain. Wearing marker caps on ones fingers (a more stylish option is to paint the said fingernails in a different colour each) and making gestures with them,
24

one can use any interface (Yes! Any interface! No longer is the human race tied to the bulky world of computer screens) to access and modify data. Clicking a picture is as easy as conjuring up a rectangle in the air aimed at the object of visual desire with the thumb and index fingers. A few finger motions help edit and resize pictures and another set of gestures later, the pictures find themselves E-mailed to recipients

25

REFRENCES http://www.pranavmistry.com/projects/sixth-sense-technology/ http://gizmodo.com/5167790/sixth-sense-technology-may-change-how-we-look-at-the-world-forever http://www.ixibo.com/2009/03/sixth-sense-technology/ http://www.youthkiawaaz.com/2010/06/pranav-mistry-%E2%80%93-innovator-of-sixth-sensetechnology-%E2%80%93-technology-that-makes-sense/ http://www.geek.com/articles/gadgets/mit-develop-wear-ur-world-6th-sense-hand-gesture-gadget2009026/ http://notesofgenius.com/multimedia-applications-sixthsense-technology/

26