Input devices and interaction

Ruth Aylett

Contents

Tracking
What is available

Devices
Gloves, 6 DOF mouse,
WiiMote

Why is it important?

Interaction is basic to VEs

We defined them as interactive in real-time

No interaction => NOT a VE

Ideal interaction:

Very low latency - i.e fast

Multi-modal
Unencumbered
Intuitive

Technology falls well short of this of course

Tracking the human body

Large displays require position and orientation of viewers

body to be tracked
tracking information fed to runtime system as input signal.

Most commonly tracked is head but

sometimes also hands, arms,
legs, eyes etc.
Head tracking used to update virtual
viewpoint orientations.

Body tracking needed for lifelike

interaction with objects and creatures.
say user wishes to wave at another person
in the VE: their real-world motions can be
tracked and replicated in the VE.

Interaction types

Navigation
Staying on the ground?
Walking v flying
Depends on size of model wrt display system
Degree of immersion

Interaction with other users

Gesture

Interaction with objects

Depends on the object and interaction
Select, lift, rotate, throw, steer, hit

Virtual Tennis

Movie
Virtual Tennis

Tracking the human head

An essential basic requirement in immersive VR

systems.
Imagine axes mounted on top of your head
pans, tilts and yaws of head
measured around those axes.

HMDs often have rotation sensors

to measure these three angles.
Angles passed to run-time VR
software which updates viewing
angles.

HMD

Tracking devices

Many tracking devices and systems developed over the

years
some aimed specifically at VR systems
others borrowed from other areas.

Some systems are portable and cheap - some require

permanent installations in large rooms and are very
expensive indeed.
Trackers can be magnetic, electro-magnetic, acoustic, inertial,
optical, or mechanical.

Electro-magnetic trackers
transmitter generates electromagnetic signals
received by a receiver (or sensor).
Signal strength used to determine absolute position and
orientation of receiver relative to transmitter.

Example: Polhemus FASTRAK

FASTRAK electro-magnetic sensor from Polhemus

accurately computes the position and orientation of tiny receiver
as it moves through space.

Dynamic, real time six degree-of-freedom measurement

of position (X, Y, and Z) and orientation (yaw, pitch, and
roll)
RS-232 signal updated at 120 records/sec.

Transmitter constantly puts out a weak magnetic field.

passive receiver generates an electric signal as it is moved
through the field.
Polhemus' processing electronics then amplify and analyse this
signal to determine the real-world position and orientation of the
receiver relative to the transmitter.

Polhemus FASTRAK system

Polhemous trackers well proven and widely used since the

very early 1990s.
The FASTRAK system shown
here has one receiver
and one transmitter.
System expanded
by adding up to three more
receivers
can attach receivers to
different parts of body
log data for gait and limb
analysis or computer
animation.

Electromagnetic Tracking
Polhemus

Electromagnetic Tracking
Ascension
Ascension market a number of systems based on DC
rather than AC fields including Flock of Birds and a full
gait analysis system called MotionStar.

Electromagnetic Tracking
Advantages
Small receivers
Reasonably cheap
Line-of-sight (LOS) not required
Disadvantages
Accuracy diminishes with distance
Not very large working volume
High latency due to filtering
Transmitter/receiver required

Electro-magnetic interference

Major problem of electro-magnetic trackers

magnetic fields easily affected by the surrounding environment.

Large metal objects produce eddy currents in the

presence of the magnetic fields
These can interfere and distort the original signal causing
inaccurate measurements.
same effect appears near electric currents, such as in cabling
also ferromagnetic materials
Also electromagnetic sources such as computer monitors.

Ferromagnetic
distortion

and/or

metal

surfaces

cause

field

Ultrasonic trackers

Two main components

transmitter generating an ultrasound signal
receiver detecting the signal.

Distance is calculated by
measuring time-of-flight
of ultrasonic pulse.
Three transmitters and receivers
needed to calculate full 3D
position and orientation.

Ultrasonic tracking used by Logitech Head Tracker

(shown) and 3D mouse.

Ultrasonic trackers

The Power Glove made by toy

company Mattel (who make Barbie)
introduced in 1989 for use with
the Nintendo Entertainment
System (NES).

Ultrasonic device for use in place of

standard Nintendo controllers
Detected finger motion
Plus full set of buttons on the wrist.

In fact not much use for Nintendo

gamers
But amazingly advanced piece of VR
kit for its time.

Acoustic Tracking
Advantages
Well known transducers (mics), lightweight
Low cost device
Disadvantages
Line-of-sigh (LOS) required
Echoes
Low accuracy (speed of sound in air varies)
Transmitter/receiver required

Inertial tracking systems

Very popular (because cheap)

based on inertial gyro technology
Detects acceleration and thus can calculate velocity (since mass
in known) giving 3DoF
Newish example is the Intersense IS-300.

Can be coupled with add-on ultrasonic system to give 6

DoF sensing
example of a hybrid technology tracker.

IS-300 can operate in metallic

environments,
6 DoF tracker operates only
in LoS of transmitter.

Other examples:
Intersense Intertrax2 and the Ascension 3D-Bird.

Inertial Tracking
Advantages
Cheap
Small size
No transmitter/receiver required
LOS not required
Disadvantages
Only 3 DOF on their own
Drift
Not accurate for slow movements

Optical tracking methods

Many different forms

Often use image processing and pattern recognition and matching
Much work outside of VR: numerous ideas suitable for tracking
object position and pose

For example fiducial mark detection

light sources or reflective
colour markers attached to
object at important locations such
as joints or extremities.

Easier for image processing

algorithm to track in cluttered
conditions.

How it is done

Optical tracking methods

Outside-in tracker
tracking apparatus is fixed
object to be tracked (e.g. the user) is viewed from the "outside".

Inside-out systems
take tracking measurements from the object to be tracked
for instance a camera can be mounted on the HMD
images analysed to produce pose and distance estimations based on
the position of fixed patterns within the environment.

Visible images or infra-red used.

Many optical systems (but not all!) are one-offs, expensive
and require careful calibration procedures.

Infra-red cameras

Optical Tracking
Advantages
Can work over a large area.
Inherently wireless
Disadvantages
LOS needed
Transmitter/receiver required
Expensive
Requires computer vision technology

Eye trackers

Eye tracking systems are examples of optical tracking

devices.
viewpoint in the virtual world follows
the gaze of users eye.

Originally developed as a mouse

replacement
simply look at object
interact through eye movement
(such as a slow blink).

Support physically impaired users.

Combined eye and head tracking systems
also exist - use in practice is complicated.

Mechanical trackers

Mechanical linkage system

arm-like structure of several joint, one end fixed, the
other free to move with the user.

Measure position and angular orientation of free

end
by measuring angles at each joint and
factoring in length of each segment.

Fake Space BOOM (right)

Mechanical Tracking
Advantages
Simple sensors, no need for transmitter/receiver
low-cost device
very low latency
High positional accuracy
Disadvantages
The user is tethered
Lots of inertia
Typically small working volume
Mechanical parts wear out

Unencumbered tracking

Depends on identifying hand/hand on

video
One approach is using blobs

Cybergloves and similar

Inherent in the folklore and hype of VR is the cyberglove

- a wearable device that monitors the the position and
orientation of hand and fingers.
The name CYBERGLOVE is
registered by Virtual Technologies Inc
(VTi).
uses 18 or 22 patented
angular sensors for tracking the
position of fingers and hand.

Gloves
Virtual Technologies
CyberGlove

- 18-sensor model

- 22-sensor model
Variants are:

- CyberTouch

- CyberGrasp

Gloves
Fifth Dimensions Technologies - Data Glove

Data Glove
finger flexure
hand orientation -roll & pitch

Gloves
Fakespace - Pinch Glove
Pinch Glove
gesture recognition
reliable
low cost
electrical sensors in
each fingertip
contact among any 2 or more
digits

Mouse as input device in VR

Normal 2D mouse can be used (as in Cortona

for example).
Need user selectable modes to switch between
DoFs.

More sophisticated mice provide 3 or more

DoF: these include the Spaceball (shown
here) and Spacemouse.
Standard games joysticks or
gamepads also used
to give 2 or more DoFs.

6 DOF Mice

3 translation DOF
3 rotation DOF

6 DOF Mice
Spaceball
by
Labtec

Spacemouse by DLR (Logitech - USA)

6 DOF Mice
Cyberpuck
SpaceOrb

The WiiMote

3 accelerometers
Enough for 6 DOF
But will drift
Bluetooth connection
to 10m

Optical (IR) sensor

To 5m from sensor
bar
Triangulation from
ends of bar
Allows accurate
pointing

Speaker

WiiMote interaction

Head-tracking
WiiMote stationary, head-mounted IR source

Finger-tracking - touch-free interaction

IR tape on finger + fixed IR source

Gesture recognition
Using accelerometers
Feature classification
Fast movements work better; beware variable arm
orientation

Software

Free libraries
WiiGLE
http://mm-werkstatt.informatik.uniaugsburg.de/documents/WiiGLE/doku.php

Provides a set of classifiers

WiiGee
http://www.wiigee.org/index.html
Java-based, one classifier

Issues with Bluetooth stacks

Flakey implementations, especially Vista
BlueSoleil seems a good driver
http://www.bluesoleil.com/