IMPLEMENTATION OF HAND TRACKING ON MOUSE

POINTER CONTROLLER USING PYRAMIDAL LUCASKANADE ALGORITHM

Hendri Karisma* and Irfan Maliki
Informatic Engineering Major
Indonesian Computer University (UNIKOM)
Bandung, 40132, Indonesia
*Corresponding author. Tel : 62-8565 9378 552 (Hendri Karisma), 62-8132 2055 979 (Irfan
Maliki), Fax : E-mail address : situkangsayur@gmail.com (Hendri Karisma), irfanmaliki007@gmail.com (Irfan
Maliki)

Keywords : Hand Tracking, Pyramidal Lucas-Kanade, Mouse Pointer.

ABSTRACT
The variety of techniques that allow for interactive dialogue between man and computer
communications in today is quite varied, one of which is the interaction techniques that
are natural. Interaction techniques that are natural is the ability to sense user interaction
without direct physical engagement with the computer system. Input to interact,
performed by using a web-camera. Visual input can provide sensing capabilities to the
computer as a human to perform sensing with the eyes. One of the natural motion can
be used as a tool to communicate the movement of the hand or hand gesture. The
method used in this study is pyramidal Lucas-Kanade. This method is used to build the
system controller mouse pointer on a computer using hand gestures from the user.
Mouse movement will be adjusted by the movement of the hand that entered on the
web-cam in real-time. The results of the research has been done, the system controller
can be implemented and the mouse pointer pyramidal Lucas-Kanade algorithm works
as expected and was able to detect movement of a pixel point quickly.

INTRODUCTION
1

Computer vision-based interaction technique is a technique that is natural

interaction which does not require direct contact between the user input device.
Visual input can provide sensing capabilities to the computer as a human to
perform sensing using the eye.
With the implementation of visual technology in interacting with computers most
likely in the next 10 years, there will be developments in the area known as the
Human Computer Interaction (HCI). In the next 10 years, mouse and keyboard is
no longer the main device to interact with computers. HCI technologies such as
speech recognition and gesture recognition will begin to be found everywhere.
Hand tracking is one of implementation and motion tracking technique, where
the tracking done by human hands on the object, in this case the video source is
a video produced by the web-cam in real-time visual display objects of the hands
of an individual or a user, then performed the detection of motion or movement
of the user's hands, so the computer can detect movement and direction of
movement of the user's hand. In the implementation of the tracking hand there is
the use of certain methods which form the basis or the core of the hand tracking,
one of which is pyramidal Lucas-Kanade algorithm.
Pyramidal Lucas-Kanade algorithm using the approach of the pyramid principle,
the work of the detail image of the lowest to the highest image detail. The aim is
that a large movement can be calculated. While the assumptions used in the
pyramidal Lucas-Kanade algorithm is a small and coherent motion, so it can
capture a large movement. Solutions used to capture the movement of the
2

Lucas-Kanade algorithm is to use a large window. However, the use of a large

window often made a motion that was captured is not a coherent movement.
Pyramidal Lucas-Kanade algorithm solve the problem without removing the
assumption of the coherent motion.
Based on the above exposure in this study will be the implementation of hand
tracking using pyramidal Lucas-Kanade algorithm to build the system controller
mouse pointer on a computer using hand gestures from the user. Mouse
movement will be adjusted by the movement of the hand that entered on a webcamera in real-time.
The expected goal of this research is to assist the user in interacting with a
computer to control the mouse pointer by using the natural movement of the
hand.
The expected goal of this research is to assist the user in interacting with a
computer to control the mouse pointer by using the natural movement of the
hand.
RESEARCH METHODOLOGY
In computer vision we need a process in which the computer must be able to
follow the movement of the object specified by the source of a video image from
disk or from the camera inputs in real time. Identification carried out to find the
object of interest of a subsequent frame is from a video. Tracking is a term used
to track or find an object in an image, in the process there is something important
that happened on that object when the image is of a subsequent frame of a
3

video that motion. Motion is the movement or direction of movement made

visible object or objects that occur due to the change of frame.
After getting the shape of human hand can be determined human hand midpoint
of the form. The midpoint of the reference point to search pixel movement. In
Figure 2, indicated the completion of pyramidal Lucas-Kanade algorithm
approach to the principle of the pyramid.

Figure 1 Pyramidal Lucas-Kanade

Pyramidal Lucas-Kanade algorithm is the first work on the highest layer of the
pyramid. Then the result is used as a starting point for work on the layer below it.
This continues until it reaches the lowest level. Here is an explanation of
pyramidal Lucas-Kanade algorithm in:
Let I and J are two 2D gray scale image. Two values I (x) = I (x, y) and J (x) = J
(x, y) is the grayscale value of the two images at location x = [x, y] T, with x and
y are the coordinates of two pixels of a generic image point x. The image I will be
referenced as the first image and the image of J as the second image.
Think of a point u = [ux uy] T in the first image I. The purpose of tracking fiture is
to find the location v = u + d = [ux uy + dx + dy] T in the second image J with I (u)
4

and J (v) is similar. Vector d = [dx dy] T is the image velocity at x, also known as
optical flow at x. It is important to define the notion of neighborhood similarity in
2D. Suppose wxdan wyy two integer that defines the window half the size of the
integration over x. D is defined as a vector image velocity that minimizes the
function defined by equation (3).

...3
In the detection phase of the hand there are certain processes in the
implementation of such methods, including requires a bit and the image of the
contour of the skin color segmentation. Threshold the image-making is done by
segmenting the maximum and minimum values of skin color in YCrCb color
format. Figure 3 describes the use of color YCrCb format.

Figure 3 Examples of the use of pyramidal

Figure 2 The use of color YCrCb format

Lucas-Kanade

Figure 4 Workflow processes in a system built

Before implementing the pyramidal Lucas-Kanade algorithm, corner finding

needs to be done. Corner finding is a technique used to perform a search on
each side of an image and find a good angle to be used as a reference in the
conduct of corner tracking on every frame. Needed in the pyramidal LucasKanade algorithm is two grayscale images (eg image A and image B). A hand
containing an object image on the starting position, while B is the last image that
occurs in the image. Search conducted corner to begin the process of object
tracking hands, then do the computation by the method of Lucas-Kanade
pyramidal algorithm and to obtain the coordinates of the last position of the
hands as well as sync with the mouse pointer position. In general, the stages are
done in implementing the pyramidal Lucas-Kanade algorithm can be seen in
Figure 3.

RESULTS AND DISCUSSION

Input from a system built which formed the image of a human hand or human
body. The detection is done is by using skin color, so if there is a cross between
the position of body parts to one another, the detection of possible errors will
occur because of the same color in large sizes. Figure 10 is a GUI that is built
6

using the QT framework for the system. At figure 5 that is the result of
segmentation and contour finding from the human hand using YCrCb color
format, and 1 bit image.

Figure 5 Color Segmentation Result

Figure 6 Result of Contour finding

At Figure 7 to 10 describes the image of a system that has been built with some
of the conditions of the captured frame. The number of fingers detected is used
to perform the right click and left click instead of mouse.

Figure 7 Conditions when the amount to 5

Figure 8 Conditions when the finger

fingers

number 3

Figure 9 Condition when fingers numbered 1

Figure 10 Display histogram form

Visualization system is equipped with a histogram of the image being shown the
results of the computation.

CONCLUSIONS
Conclusions derived from research conducted as follows:
1. Mouse pointer control system that is used to interact with computers using
natural movement of the hand has been successfully built.
2. Pyramidal Lucas-Kanade algorithm is used to detect the movement of objects
by hand utilizing the difference in skin color YCrCb format, has worked quite
well in the process. Pyramidal Lucas-Kanade algorithm works as expected
and can detect movement of a pixel point quickly. But the weakness occurs in
the utilization range of skin color in YCrCb format, because the instability is
likely to occur when the tracking process. This is caused by the contours that
were created only on the basis of skin color differences. So that when the
detection of two objects with colors similar to skin color, then the two objects
are considered to be one object in hand with the size of the field is quite
broad.
8

3. Light into one of the factors that affect the quality of the results of the
processes that occur within the system. The better the lighting, the process of
computing detection produce the contours of a good hand.
REFERENCES
1)

Bouguet, Y.J., 2000, Pyramidal Implementation of The Lucas-Kanade

Feature Tracker, Intel Corporation, Microprocessor Research Labs, ursl
http://www.intel.com/research/opencv, accessed in 30 Juni 2011.

2)

Fadlisyah, 2007, Computer Vision dan Pengolahan Citra, Andi Publisher.

3)

Gary B, and Adrian K., 2008, Learning OpenCV Computer Vision with
the OpenCV Library, OReilly Media inc.

4)

Intel, 2000, Open Source Computer Vision Library, Intel Corporation.

5)

Mahtarami, A., Hariadi, M., 2010, Hand Motion Tracking Based

Pyramidal Lucas-Kanade, Thesis, Elektro Engineering Major, F industrial
technology faculty, (Sepuluh November Institute of Technology ) ITS.

6)

Mark S., Jasmin B., 2006. C GUI Programming with Qt 4, Second

Edition (c), Prentice Hall.