social interaction analysis lecture 1: introduction

daniel gatica-perez

april 2012

many of us have been to meetings like this

or come across people like this dominance and leadership fundamental constructs in social interaction related to power and social verticality profound effects on relationships & organizations

many of us have watched YouTube videos like this

40% of the most popular content are vlogs

(Burgess and Green, YouTube, 2009)

video blogging highly creative form of participation life documentary & communication tool social behavior & metadata

the goal: inferring patterns from social interaction

interacting people (face-to-face or online)

nonverbal communication
(Knapp and Hall)

nonverbal behavior and social inference

(Knapp and Hall)

nonverbal cues
alternative channel to spoken words indicators of states, traits, and relationships honest: unconscious, hard to fake rapid: accurate judgments can be done from thin-slices

NVB & F2F interaction

attraction (Pentland 05) roles (Zancanaro 06) dominance (Hung 08) personality (Pianesi 08) leadership (Jayagopi 09)

the nature of social interaction

- verbal cues: speech - nonverbal cues from audio and video - attention, body motion, postures complementary multimodal cues multimodal nature

computer vision audio processing audio-visual fusion

interactive nature - interaction can be measured - interaction regulates individuals complementary multiparty cues

cognitive science

the wealth of social interaction

# people
large (~10) small (~5) dyadic

temporal scale
visual focus who looks at whom addressing who speaks to whom turn-taking floor control, monologues, discussions interest degree of collective engagement internal states dominance degree of influence or status traits and relationships

conversational dynamics

computational modeling of social interaction

Social sensing
Nonverbal behavior (NVB) + Proximity + Body activity + Visual attention + Speaking turns + Prosody * Ubiquitous sensing * Principled methods for - Robust NVB extraction - Perceptual integration

Social models
Internal states + Interest Personality traits + Dominance Relationships + Status + Roles * Behavior integration - NVB is always grounded (socially, temporally) - From thin-slices to long-term * Interpretable social models

general methodology

interaction data

audio data

video data

audio-visual behavior extraction probabilistic modeling

visual focus who looks at whom? addressing who speaks to whom? turn-taking what is the interaction phase? interest degree of engagement dominance who is the boss?

some challenges

automatic nonverbal cue extraction some audio/vision methods are robust but some cues are subtle social inference from nonverbal cues inconsistencies in the relation between NVB and social constructs NVB depends on context personal: individual differences social: type of relationship temporal: relationships evolve cultural: regional differences

social interaction vs. human-computer interaction

HCI: human-computer interaction assumes that people are willing to interact with machines beyond mouse and keyboard behavior: what people can do in front of a machine

social interaction computers in background people is what matters put the people first and everything else will follow

applications: face-to-face support

DiMicco, MIT, 2004

Kaplan, EPFL, 2006

Sturm, Eindhoven, 2007

Kim, MIT, 2008

applications: remote communication support

applications: online video enhancement

applications: social media search

applications: traditional media search

state of affairs (circa 2009)

state of affairs (2)

state of affairs (3)

10

outline for the course

introduction measuring behavioural cues from conversational data social inference in small groups social inference in social media

questions?

11