Graphics and Imaging

Architectures
Kayvon Fatahalian
http://www.cs.cmu.edu/afs/cs/academic/class/15869-f11/www/

About Kayvon
New faculty, just arrived from Stanford
Dissertation: Evolving real-time graphics pipeline for
micropolygon rendering

(micropolygon meshes = high-resolution meshes used in off-line rendering for film)

In addition to rendering: involved in many research projects

related to computing on GPUs

Brook (precursor to CUDA/OpenCL)
Sequoia (locality-aware parallel programming language)
GRAMPS (stream computing for heterogeneous HW)

Kayvon Fatahalian

CMU 15-869, Fall 2011

COMPUTING IS BECOMING HIGHLY VISUAL!

Visually rich user interfaces
(that require 3D graphics!)
Innovative new input modalities
(touch, gyro, cameras)
Ubiquitous, high-resolution cameras
(emergence of computational photography)
Games, entertainment
(continuous push towards higher visual realism)

Kayvon Fatahalian

CMU 15-869, Fall 2011

COMPUTING EFFICIENTLY IS INCREASINGLY CRITICAL!

Ubiquitous parallelism

Core i7 (Nahalem)

Heterogeneous parallelism
Power constraints
Mobile, mobile, mobile

Real-time 3D graphics has always found a way to consume all available compute:
GPUs are efficient parallel, heterogeneous systems (that are relatively easy to
program)
Kayvon Fatahalian

CMU 15-869, Fall 2011

My Macbook Pro 2011 (two GPUs)

AMD Radeon HD GPU

Quad-core Intel Core i7 CPU

(Contains integrated Intel GPU)

From ifixit.com teardown

Kayvon Fatahalian

CMU 15-869, Fall 2011

iPad 2 main board

Apple A5 SoC

Dual-core ARM CPU

Image processing DSP

GPU

Video
Encode/
Decode

Flash memory

From ifixit.com teardown

Kayvon Fatahalian

CMU 15-869, Fall 2011

Touchscreen controller (integrated DSP) ~ $1

From US Patent Application 2006/0097991

Kayvon Fatahalian

CMU 15-869, Fall 2011

My Nikon D7000 camera

16.2 MP sensor
14 bits per pixel
6 fps burst
Kayvon Fatahalian

CMU 15-869, Fall 2011

What this course is

This is a course about how real-time graphics systems work

GRAPHICS
ALGORITHMS

mapping/scheduling

MACHINE
ORGANIZATION

(the workload)

geometry processing
rasterization
texture mapping
anti-aliasing

Parallelism
Locality
Communication
The design of throughput processing cores
The role of fixed-function HW
Kayvon Fatahalian

CMU 15-869, Fall 2011

What this course is

This is a course about how real-time graphics systems work

GRAPHICS
ALGORITHMS

mapping/scheduling

MACHINE
ORGANIZATION

(the workload)

ABSTRACTIONS
(the real-time graphics pipeline)
choice of primitives
level of abstraction

Kayvon Fatahalian

CMU 15-869, Fall 2011

What this course is not

This is not an [OpenGL, CUDA, OpenCL] programming course

But we will be discussing the design of these

abstractions (and their future) in great detail!

Kayvon Fatahalian

CMU 15-869, Fall 2011

Logistics 1: schedule
Expect it to be fluid

its my first time doing this

students in this class have wide variation in background
http://www.cs.cmu.edu/afs/cs/academic/class/15869-f11/www/

First half of class: focused on real-time 3D graphics

3D graphics abstractions and the implementation of the real-time 3D

graphics pipeline (e.g., OpenGL/Direct3D) on modern GPUs

Second half of class: broader sampling of applications and systems

future 3D graphics pipelines, GPU programming model issues (OpenCL/

CUDA), hybrid CPU/GPUs, image processing architectures, cameras

Several external speakers (AMD, Intel, NVIDIA)

Kayvon Fatahalian

CMU 15-869, Fall 2011

Logistics 2: expectations
This is a reading/project course

(40%) Paper readings

Weekly: you will be asked to submit brief reviews of selected

papers (or answer a specific questions)
Participate in class discussions

(10%) Two quick (2-3 day) C++ programming assignments

Meant to concretize concepts, not stress you out

(50%) Semester-long, self-selected research project (teams of 1-2)

Aim for publishable results

Kayvon Fatahalian

CMU 15-869, Fall 2011

3D rendering

Image credit: Henrik Wann Jensen

Model of a scene:
3D surface geometry (e.g., triangle mesh)
surface materials
lights
camera

Image

How does each triangle contribute to each pixel in the image?

Kayvon Fatahalian

CMU 15-869, Fall 2011

What is an architecture?
(white board)

Kayvon Fatahalian

CMU 15-869, Fall 2011

Real-time graphics pipeline (entities)

3
1

4
2
Vertices

Primitives
(triangles, points, lines)

Fragments

Pixels
Kayvon Fatahalian

CMU 15-869, Fall 2011

Real-time graphics pipeline (operations)

Vertex Generation

Vertices

Vertex stream

3
4

Vertices in 3D space

Vertex Processing
Vertex stream

Vertices in positioned on screen

Primitive Generation

Primitives

Primitive stream

Primitive Processing
Primitive stream

Triangles positioned on screen

Fragment Generation
(Rasterization)

Fragments

Fragment stream

Fragments (one per each covered pixel)

Fragment Processing
Fragment stream

Pixels

Shaded fragments

Pixel Operations

Output image (pixels)

Kayvon Fatahalian

CMU 15-869, Fall 2011

Real-time graphics pipeline (state)

Memory Buffers (system state)
Vertex Generation

Vertices

Vertex stream

Vertex Processing
Vertex stream

3
4

Vertex Data buffers

Texture buffers
Vertex transform matrices

Primitive Generation

Primitives

Primitive stream

Primitive Processing

Texture buffers

Primitive stream

Fragment Generation
(Rasterization)

Fragments

Fragment stream

Fragment Processing

Texture buffers

Fragment stream

Pixels

Pixel Operations

Output image buffer

Kayvon Fatahalian

CMU 15-869, Fall 2011

History:
Evolution of interactive 3D graphics
Following slides borrowed from Kurt Akeley and Pat Hanrahan
http://graphics.stanford.edu/wikis/cs448-07-spring

Kayvon Fatahalian

CMU 15-869, Fall 2011

Slide credit Kurt Akeley and Pat Hanrahan

Kayvon Fatahalian

CMU 15-869, Fall 2011

Early years
Evans and Sutherland built machines for flight simulators

throughout the 70s

Early 80s: Custom ASIC for geometry processing

The Geometry Engine: a VLSI Geometry System for Graphics

(Jim Clark, SIGGRAPH 82)
Work started at Stanford (led to Silicon Graphics, 1982)

Kayvon Fatahalian

CMU 15-869, Fall 2011

Slide credit Kurt Akeley and Pat Hanrahan

Kayvon Fatahalian

CMU 15-869, Fall 2011

Slide credit Kurt Akeley and Pat Hanrahan

Kayvon Fatahalian

CMU 15-869, Fall 2011

Slide credit Kurt Akeley and Pat Hanrahan

Kayvon Fatahalian

CMU 15-869, Fall 2011

Slide credit Kurt Akeley and Pat Hanrahan

Kayvon Fatahalian

CMU 15-869, Fall 2011

Slide credit Kurt Akeley and Pat Hanrahan

Kayvon Fatahalian

CMU 15-869, Fall 2011

Off-line
rendering for
film
Off-line graphics advancing as well

Image credit: Pixar, Toy Story 1 , 1995

Pixar created in 1986

Initially thought they were going to build hardware (Pixar image computer)

Feature rich: smooth/complex surfaces, programmable shading, anti-aliasing,

motion blur
Implemented as a software application on CPUs
Hours/frame

Reyes graphics system at Pixar (Renderman)

Kayvon Fatahalian

CMU 15-869, Fall 2011

Early PC graphics

Wolfenstein 3D, 1992

Doom, 1993

Software applications running on CPU

Extreme amount of software optimization,
algorithm development (and hacks)
Kayvon Fatahalian

CMU 15-869, Fall 2011

PC 3D graphics
3D graphics acceleration for PC motivated by games
add-in boards made by 3DFX, NVIDIA, ATI, PowerVR, and more

3DFX founders were from SGI

3DFX Voodoo (1997)

standardized around accelerating OpenGL/Direct3D

offloaded rasterization, texture mapping, frame-buffer processing

initially only subset of GL implemented by Quake

Quake 1

Kayvon Fatahalian

CMU 15-869, Fall 2011

Slide credit Kurt Akeley and Pat Hanrahan

Kayvon Fatahalian

CMU 15-869, Fall 2011

Fourth generation
Programmable shading

Kayvon Fatahalian

CMU 15-869, Fall 2011

Current generation
Tessellation (smooth/complex surfaces)

Image credit: NVIDIA/Unigine

Kayvon Fatahalian

CMU 15-869, Fall 2011

Current generation
Some GPU computations now driven by alternative
programming interface (not 3D graphics pipeline)

to augment 3D rendering (game physics, global lighting)

for non-graphics applications (scientific computing)

Image credit: NVIDIA

Image credit: Folding @ Home

Kayvon Fatahalian

CMU 15-869, Fall 2011

Today: high versatility, high peak compute

Intel Core i7 (quad-core CPU)

AMD Radeon HD 5870 GPU

~100 GFLOPS
730 million transistors

~2.7 TFLOPS
2.2 billion transistors
More compute power than Pixars
entire render farm for Toy Story 1!
Kayvon Fatahalian

CMU 15-869, Fall 2011

Readings
D. Blythe, The Rise of the Graphics Processor
Proceedings of the IEEE, 2008

Real-Time Rendering, Ch 2. The Graphics Rendering Pipeline

(handout)

Kayvon Fatahalian

CMU 15-869, Fall 2011