Shared-Memory Model and Threads: Intel Software College

Introduction to Parallel Programming Part 2
Shared-Memory Model and Threads

Intel Software College
Objectives
At the end of this module you should be able to:
Describe the shared-memory model of parallel
programming
Describe the differences between the fork/join
model and the general threads model
Demonstrate how to implement domain and
functional decompositions using threads
Decide whether a variable in a multithreaded
program should be shared or private
Copyright 2006, Intel Corporation. All rights reserved.

Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Cooperating Parallel Processes

Parallel computing multiple processes working
together to speed the solution of a task
Working together process cooperation

Kinds of cooperation
Sharing information (communication)
Keeping out of each others way
(synchronization)

The Shared-Memory Model

CPU
CPU
CPU
Private
Memory
Private
Memory
Private
Memory
Shared Memory

Evaluating Parallel Models

How do processes share information?
How do processes synchronize?
In shared-memory model, both accomplished
through shared variables
Communication: buffer
Synchronization: semaphore

Methodology
Study problem, sequential program, or code segment
Look for opportunities for parallelism
Use threads to express parallelism

What Is a Process?
A program in some state of execution
Code
Data
Logical address space
Information about a process includes

Process state
Program counter
Values of CPU registers
Memory management information
7

What Is a Thread?
A unit of control within a process Carver and Tai
Main thread executes programs main function
Main thread may create other threads to execute
other functions
Threads have own program counter, copy of CPU
registers, and stack of activation records
Threads share processs data, code, address space,
and other resources
Threads have lower overhead than processes
8

Another Way of Looking at It

Process
File Descriptors
Address
Space
Other Data
Thread 0
(Program Counter, Registers, etc.)
So why does this view matter?
9

Why This View Matters

Process
File Descriptors
Address
Space
Other Data
Thread 0
Thread 1
10

What Are Threads Good For?

Making programs easier to understand
Overlapping computation and I/O
Improving responsiveness of GUIs
Improving performance through parallel execution
11

Fork/Join Programming Model

When program begins execution, only master thread
active
Master thread executes sequential portions of

program
For parallel portions of program, master thread forks
(creates or awakens) additional threads
At join (end of parallel section of code), extra
threads are suspended or die
12

Join
Fork
Join
Other Threads
Fork
Master
Thread
13

Relating Fork/Join to Code

Sequential code
for {
Parallel code
}
Sequential code
for {
Parallel code
}
Sequential code
14

More General Threads Model

When program begins execution, only one user
thread, called the main thread, is active
The main thread can create other threads, which

execute other functions
Created threads can also create additional threads
How this is done varies according to programming
language or API
15

Fork/Join versus General Model

You can implement fork/join in the general model
Hence fork/join a special case of general model
More structured
More easily optimized
General model
More flexible
Better support for functional decompositions
16

Incremental Parallelization
Sequential program a special case of threaded
program
Programmers can add parallelism incrementally

Profile program execution
Repeat
Choose best opportunity for parallelization
Transform sequential code into parallel code
Until further improvements not worth the effort

17

Utility of Threads
Threads are flexible enough to implement
Domain decomposition
Functional decomposition
Pipelining
18

Domain Decomposition Using Threads

Thread 0
Thread 2
f()
f()
Shared
Memory
Thread 1
f()
19

Functional Decomposition Using Threads

Thread 0
e()
Thread 1
Shared
Memory
h()
20
f()
g()

Pipelining Using Threads

Thread 0
Data e ( )
set 4
Thread 1
Thread 2
f()
g()
Data
set 3
Data
set 2
Input
Data sets
5, 6, ...
21
Data
set 1
Output
Shared Memory

Shared versus Private Variables

Thread
Private
Variables
Shared
Variables
Private
Variables
Thread
22

Domain Decomposition
Sequential Code:
int a[1000], i;
for (i = 0; i < 1000; i++) a[i] = foo(i);
23

Sequential Code:
int a[1000], i;
for (i = 0; i < 1000; i++) a[i] = foo(i);
Thread 0:
for (i = 0; i < 500; i++) a[i] = foo(i);

Thread 1:
for (i = 500; i < 1000; i++) a[i] = foo(i);
24

Sequential Code:
int a[1000], i;
for (i = 0; i < 1000; i++) a[i] = foo(i);
Thread 0:
for (i = 0; i < 500; i++) a[i] = foo(i);

Thread 1:
for (i = 500; i < 1000; i++) a[i] = foo(i);
Private
25
Shared

Functional Decomposition
int e;
main () {
int x[10], j, k, m;
}
j = f(x, k); m = g(x, k); ...
int f(int *x, int k)

{
int a;
a = e * x[k] * x[k];
}
int g(int *x, int k)
{
int a;
k = k-1;
}
26
return a;
a = e / x[k];
return a;

int e;
main () {
int x[10], j, k, m;
}
j = f(x, k);

{
int a;
a = e * x[k] * x[k];
}
{
int a;
k = k-1;
}
27
m = g(x, k);
Thread 0
return a;
Thread 1
a = e / x[k];
return a;

volatile int e; Static variable: Shared
main () {
int x[10], j, k, m;
}
j = f(x, k);

{
int a;
a = e * x[k] * x[k];
}
{
int a;
k = k-1;
}
28
m = g(x, k);
Thread 0
return a;
Thread 1
a = e / x[k];
return a;

volatile int e;
Heap variable: Shared
main () {
int x[10], j, k, m;
}
j = f(x, k);

{
int a;
a = e * x[k] * x[k];
}
{
int a;
k = k-1;
}
29
m = g(x, k);
Thread 0
return a;
Thread 1
a = e / x[k];
return a;

volatile int e;
main () {
int x[10], j, k, m;
j = f(x, k); m = g(x. k);
}
Functions local variables: Private
{
int a;
a = e * x[k] * x[k];
}
{
int a;
k = k-1;
}
30
Thread 0
return a;
Thread 1
a = e / x[k];
return a;

Shared and Private Variables

Shared variables
Static variables
Heap variables
Contents of run-time stack at time of call
Private variables
Loop index variables
Run-time stack of functions invoked by thread
31

The Shared-Memory Model (Reprise)

CPU
CPU
CPU
Private
Memory
Private
Memory
Private
Memory
Shared Memory
32

The Threads Model

Thread
Thread
Private
Variables
Private
Variables
Thread
Private
Variables
Shared Variables
33

References
Jim Beveridge and Robert Wiener, Multithreading Applications in
Win32, Addison-Wesley (1997).
David R. Butenhof, Programming with POSIX Threads,

Addison-Wesley, (1997).
Richard H. Carver and Kuo-Chung Tai, Modern Multithreading:
Implementing, Testing, and Debugging Java and C++/Pthreads/
Win32 Programs, Wiley-Interscience (2006).
Doug Lea, A Java Fork/Join Framework, Proceedings of the
ACM 2000 Conference on Java Grande, pp. 36-43.
Michael J. Quinn, Parallel Programming in C with MPI and
OpenMP, McGraw-Hill (2004).
34

35


Shared-Memory Model and Threads: Intel Software College

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Shared-Memory Model and Threads: Intel Software College

Uploaded by

Copyright:

Available Formats

Introduction to Parallel Programming Part 2

Shared-Memory Model and Threads

Intel Software College

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Cooperating Parallel Processes

Working together process cooperation

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

The Shared-Memory Model

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Evaluating Parallel Models

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Information about a process includes

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Another Way of Looking at It

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Why This View Matters

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

What Are Threads Good For?

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Fork/Join Programming Model

Master thread executes sequential portions of

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Relating Fork/Join to Code

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

More General Threads Model

The main thread can create other threads, which

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College

Fork/Join versus General Model

Shared-Memory Model and Threads

Copyright 2006, Intel Corporation. All rights reserved.

Intel Software College