Automatic Generation of Animated Presentations

Gur Wizenfeld

Advanced Software Tools Seminar June 2004

Agenda

Introduction to animated presentations generation

Stating the problem Applications

Solution Considerations Approaches

Hierarchical task decomposition (Karp & Feiner)

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Methodology

Purpose
Introduce the problem and its complexity Introduce and evaluate various solution approaches
Steven Drucker, Intelligent Camera Control for Graphical Environments [3]. Peter Karp and Steven Feiner, Issues in the Automated Generation of Animated Presentations [5]. Peter Karp and Steven Feiner, Automated Presentation Planning Using Task Decomposition with Heuristic Reasoning [6]. D.A. Friedman and Y. A. Feldman, Knowledge-Based Cinematography and its Applications, Proc. 16th European Conf. Artificial Intelligence (ECAI 2004), August 2004 [4]. David Christianson et al., Declarative Camera Control for Automatic Cinematography [2]. Maneesh Agrawala, Designing Effective Step-By-Step Assembly Instructions, SIGGRAPH 2003 [1].

Main Resources

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Introduction to Animated Presentations Generation

Stating the Problem

Given a script, describing a dynamic virtual 3D environment, generate an animated presentation of the scripts relevant information.

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Example: A Soccer Game

Versatile content Versatile style

Slow motion Multiple camera angles Textual annotations

Dynamic Parallel occurrences Real-time (unpredictable)

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Stating the Problem (Detailed)

Environment
Spatial organization of 3D objects 3D representation of objects (e.g., polygonal mesh) Dynamic Objects may be capable of modifying the environment.

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Stating the Problem (Detailed)

Script
Written in a formal language Describes all modifications to the environment over time Possibly annotated Possibly unpredictable (e.g., in interactive applications)

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Stating the Problem (Detailed)

Presentation
A description of every frame of the animated presentation
A combination of viewports The contents of each viewport is generated by rendering the environment from a virtual cameras point of view.

Specifying camera parameters for each viewport in each frame is sufficient. Possible extensions:
Audio Textual annotations Lighting
Gur Wizenfeld - Advanced Software Tools Seminar 2004
14:46

Applications: 3D Visualization

Virtual environment walkthroughs

Architecture Virtual malls

Scientific Visualization
Large multidimensional datasets exploration
Medical imaging Weather simulation
Gur Wizenfeld - Advanced Software Tools Seminar 2004

Applications: Entertainment

Computer animated movies Games Highly interactive Unpredictable script Rich 3D environment Fast Paced MUDs
Parallel & distributed Raises the need for automated summaries
Gur Wizenfeld - Advanced Software Tools Seminar 2004

Applications: Design & Planning

CAD Manufacturing process planning Animation generation

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Solution Considerations

Solution Considerations: User Task

Exploration
Orientation Navigation Search/Wander

Planning Control & Teleoperation

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Solution Considerations: Degree of Interaction

Frequency of interaction
Continuous vs. autonomous

Style of interaction
Direct control of camera vs. symbolic specification

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Solution Considerations: Supporting Cognitive Capabilities

Presentation viewing and understanding is based on complex underlying perceptive and cognitive mechanisms.
Motion from still images 3D spatial relations from 2D projections Recognition Attention Memory (Story understanding)

A comprehensive cognitive model would help, but is difficult to construct.

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Solution Considerations: Collective Cinematic Experience

Collective human experience of cinematic expression has formed a cinematic language.

Grammar Rules Idioms
An Idiom is a stereotypical way to capture some specific action as a series of shots.

Evolving genres
Gur Wizenfeld - Advanced Software Tools Seminar 2004
Trevor Goring www.famousframes.com

Solution Considerations: Continuity

Avoid jump cuts (i.e., cuts between which the camera rotates about the object by less than 30.) Do not interrupt a steady, flowing action with a cut (or use time overlapping shots.) Maintain the relative screen position of an object across cuts.

Avoid line crossing in a cut (or use an intermediate shot.)

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Example: Avoid Line Crossing

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Andrew Burward-Hoy www.famousframes.com

Presentation Hierarchy
Film

Sequence

Scene

Shot

>Transition<

Shot

>Transition<

Shot

Frame

...

Frame

Frame

...

Frame

Frame

...

Frame

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Requirements Summary

Select Content
Relevance Summarization

Map Script-time to presentation-time

Parallelism Pacing

Determine Presentation Hierarchy

Structuring shots to scenes, etc. Transitions Viewports layout

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Requirements Summary

Camera placement
Visibility
Occlusion avoidance Projection size

Recognition
Preferred viewing directions (gaze vectors)

Maximization of geometric information

Spatial relations Distortion avoidance Selection of geometric features

Obey cinematic conventions

Use idioms correctly Maintain continuity Adhere to genre rules

Take into account cognitive capabilities and limitations.

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Solution Map
Script

Script Understanding & Analysis Presentation Generation Presentation Rendering

Presentation

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Hierarchical Task Decomposition

Automated Presentation Planning of Animation Using Task Decomposition with Heuristic Reasoning
P. Karp and S. Feiner

Solution Highlights

Creates a complete presentation structure (from film level to single shot level.) Uses communicative goals to guide presentation generation. Formalizes and uses cinematic knowledge in a straightforward manner. Reduces redundant computational effort.

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Hierarchical Task Breakdown Example

Plan a vacation OR Organized Tour Independent Travel

OR

AND

Call Travel Agent

Order On-Line

Research

Make Reservations

AND

OR

AND

Compare Sites

Choose Best Bid

Internet

Travel Guide

Other Travelers

Flights

Hotels

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Presentation Generation Task Decomposition

Film structure hierarchy is used to guide decomposition of the presentation task.

Communicative Goals define the requirements at each level.

Gur Wizenfeld - Advanced Software Tools Seminar 2004

ESPLANADE Solution Testbed

Expert System for PLANning Animation, Design and Editing Environment: crane activation in a warehouse
Gur Wizenfeld - Advanced Software Tools Seminar 2004

Presentation Processing in Levels

Film
Group contiguous actions that occur in the same loaction.

Sequence

Scene Structuring Techniques (idioms):

Master shot discipline How-To structure Causal Structure

Group actions occuring in during same time period and in the same locale.

Scene
Divide to shots by scene structuring techniques. Choose Transitions.

Shot

Choose shot contents.

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Presentation Example

Script:
Move roll-2 with the crane. Move roll-7 with the crane. Causal information between button pressing and crane operations.

Communicative Goals:
Show moving of roll-2. Show moving of roll-7. Show causal relation between pressing the up button and raising the cranes hoist.
Gur Wizenfeld - Advanced Software Tools Seminar 2004

Film
Goals:
Show roll-2 m ov e m e nt

Decisions:
U s e s ingle se que nce

Show roll-7 movement

Show button-cra ne ca us a lity

Place: Warehouse

Sequence

Goa ls:

D e c isions:

Show roll-2 movement Show roll-7 movement Show button-crane causality

Divide to scenes by crane movement

Scene 1

Goals: Show action: Move crane to roll-7

Decisions: Master-shot discipline scene structuring

Scene 2

G oa ls:

D e c is ions :

Show action:
A tta c h c ra ne to roll-7

Master-shot discipline
s c e ne s tr uc tur ing

Scene 3

Goals:
Show a c tion: Lift C ra ne

Decisions:
c a us a l

Show button-crane causality

scene structuring

...

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Scene 1
Goals: Show action: Move crane to roll-7 Decisions: Master-shot discipline scene structuring

Shot 1 Warehouse Longshot

Shot 2 Crane Close-up

Shot 3 Buttons Medium-shot

Shot 4 Buttons Closeup

Shot 5 Roll-7 Close-up

...

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Evaluation

Pros
Provides a solution for a complete presentation hierarchy. Can be generalized to support arbitrarydepth hierarchies.

Cons
Resulting presentations are rigid and predictable.

Separates communicative goals from their implementation as idioms.

Formalizes goals and idioms in a natural manner. Goals and idioms can be added. Task decomposition approach reduces computational effort. Every combination of goals requires a specific idiom.

Does not support conflicts between goals and idioms.

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Summary

Conclusions

Use of cinematic rules is crucial.

Rules evolve flexibility and scalability Rules often conflict contradiction handling

Search space can be reduced by an appropriate abstraction. Symbolic representation is critical for usability. Script understanding and automatic communicative goal extraction are still a long way ahead.
Gur Wizenfeld - Advanced Software Tools Seminar 2004

References
1.

2.

Background 1. [1] M. Agrawala, D. Phan, J. Heiser, J. Haymaker, J. Kilngner, P. Hanrahan and B. Tversky, Designing Effective Step-By-Step Assembly Instructions, In SIGGRAPH 2003. 2. [2] D.B. Christianson, S.E. Anderson, L. He, D.H. Salesin, D.S. Weld and M.F. Cohen, Declarative Camera Control for Automatic Cinematography, Proceedings of the Thirteenth National Conference on Artificial Intelligence, vol. 1, pages 148155, 1996. 3. [3] S.M. Drucker, Intelligent Camera Control for Graphical Environments, PhD Thesis, MIT, 1994. 4. [4] D.A. Friedman and Y.A. Feldman, Knowledge-Based Cinematography and its Applications, Proc.16th European Conf. Artificial Intelligence (ECAI 2004), August 5. 2004 Hierarchical Task Decomposition 1. [5] P. Karp and S. Feiner, Issues in the automated generation of animated presentations, Proceedings of Graphics Interface '90, pages 39--48, May 1990. 2. [6] P. Karp and S. Feiner, Automated presentation planning of animation using task decomposition with heuristic reasoning, Proc. of Graphics Interface 93, pages 118-127, 1993.

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Thank You

Detours

Detour: Advances in computer games

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Detour: Idioms

An Idiom is a stereotypical way to capture some specific action as a series of shots.

(AcFilmIdiom name approach-idiom :parameter (AcParamApproach :actor1 :actor2 :start :stop) (AcFilmShot name shot1 (AcFragGoBy name frag1 :time ?start :primary-moment beginning :entry-pos center :exit-pos out-left ...) ...) ...) Gur Wizenfeld - Advanced Software Tools Seminar 2004

Detour: Idioms
Approach Idiom Structure AcFilmIdiom Approach-Idiom (Sub-Scene) Shot1 Fragment1.1: Go-By
:time ?start :entry-pos *center* :camera-placement *Apex*

Shot2 Fragment2.1: Go-By

Shot3 Fragment3.1: Go-By

:primary-moment *beginning* :exit-pos *out-left*

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Detour: Rendering
Crane Tilt Zoom

Roll

Pan Truck Dolly

{X, Y, Z, Rx, Ry, Rz, fov}7

Gur Wizenfeld - Advanced Software Tools Seminar 2004

Detour: Rendering

Transforms objects from a perspective view in the world space into the screens display space.

Gur Wizenfeld - Advanced Software Tools Seminar 2004