Gestural Interaction

Enhancing Human Judgment

when exploring large
information spaces
Ray Garcia, MIT Media Lab, March 23rd, 2009
rgarcia at media dot mit dot edu
 Scene – Augmenting
Intelligence
LARGE SCALE RICH INFORMATION
SPACE Layered
MEDIATIONAL MEANS Expression

SELECTION
NAVIGATION
GESTURAL Grok
SHAPING INTERFAC
E
MATERIAL OF MANIPULATION High
Fidelity INTUITION

Visual UNCOVERING PATTERNS FOR DECISION

Configuration MAKING

FINDING THE STORY IN THE GAPS

WHOLE
BEING
Geek
INTELLIGEN
ANALYSIS CE
 Human Communications
• Layered channels of expression
– Voice with tonal variation
– Facial qualifiers
– Body positions
– Spatial distance to objects
– Hand movements
– Writing, drawing
 Hidden honest signals
• Thoughts inscribed in media
– Media: stone carving to pixels
– Language expression
• Emotions
• Symbols
• Visual representations
• Musical notations
• Artistic rendering
 Technology as Mediational
Means
• Remote versus Local communication
• Asynchronous versus synchronous
• Human to Computer to Human
• Technology provides the object,
symbols, material to manipulate
• Need to learn the affordance of the
interface
 Improving Fidelity of HCI
• Signals
– Add gestural interface to complement
symbol and voice commands.
– Initially only hand movement
– Future include facial expression
• Response
– Visual screen that fill peripheral vision
– Spatial Audio
– Expressive 3D object in motion
 Material of Manipulation
• Light and sound produce meaningful
representations
• Appeal to human cognition and
perceptive input
• Build comprehension and
significance of information space
• Meta-cognition may be enhanced by
mix mode of interaction
 Whole being Intelligence
• Intelligence is in the nervous system
as well as the brain
• Human judgment under uncertainty
uses whole being intelligence
• Much information is ambiguous and
vague
• Human comprehension is based on
impressions of patterns
• Interpretation of representations
needs advances
 Augmenting Human
Intelligence
• HCI as a moderating social
engagement
• Two negotiating parties
• Fusing interpretations of
representations
• Synergistic symbiotic interaction
through Gestures
 Scenario
• Saul manipulates representation
using gestural interface
• Ava manipulated the system
configuration
• Saul uses intuition to find patterns
while Ava provides analysis to extract
the material and representations
• The material is large scale rich
information space needing discovery
and analysis to uncover patterns for
decision making
 Finding the Story
• The scenario fosters story telling
• Pattern recognition has limitation
• People fill the gaps by creating a
hidden narrative using interpretive
explanation
• In business the story is left to
intuition
• The intuition dominants decision
making in real-world situations
 Dance of Gestural
Interaction
• Discovery of the aesthetic of
representation and expression that
fosters better augmented human
judgment
• Archeology of knowledge as told by
the story of a social experience
manipulating the material of rich
information spaces
 Research is needed
• Whole being intelligence
• Cognitive interfaces
• Improve Human to Computer to
Human social communications
• Adopt Gestural Interfaces to enhance
expression
• Goal should be to improve human
judgment under uncertainty when
information space is large