COURSE TITLE COURSE CODE COURSE CATEGORY PERIODS/WEEK PERIODS/SEMESTER CREDITS MODULE I II III IV

: PARALLEL PROCESSING (Elective) : 5062 :E :4 : 72 :4 TIME SCHEDULE 18 18 18 18 72 PERIODS

TOPICS Introduction to Parallel Computing Principles of Parallel Algorithm Design Analytical Modeling of Parallel Programs Programming Shared Address Space Platforms TOTAL

OBJECTIVES MODULE - I 1.1 Introduction to Parallel Computing- Motivating Parallelism 1.1.1 The Computational Power Argument from Transistors to FLOPS 1.1.2 The Memory/Disk Speed Argument 1.1.3 The Data Communication Argument 1.2 Scope of Parallel Computing 1.2.1 Applications in Engineering and Design 1.2.2 Scientific Applications 1.2.3 Commercial Applications 1.2.4 Applications in Computer Systems 1.3 Parallel Programming Platforms 1.3.1 Implicit Parallelism: Trends in Microprocessor Architectures 1.3.2 Pipelining and Superscalar Execution 1.3.3 Very Long Instruction Word Processors 1.4 Limitations of Memory System Performance 1.4.1 Improving Effective Memory Latency Using Caches 1.4.2 Impact of Memory Bandwidth 1.4.3 Alternate Approaches for Hiding Memory Latency 1.4.4 Tradeoffs of Multithreading and Pre fetching 1.5 Physical Organization of Parallel Platforms 1.5.1 Architecture of an Ideal Parallel Computer 1.5.2 Interconnection Networks for Parallel Computers 1.6 Network Topologies 1.6.1 Evaluating Static Interconnection Networks

MODULE -2 Principles of Parallel Algorithm Design 2.1 Preliminaries 2.1.1 Decomposition, Tasks, and Dependency Graphs 2.1.2 Granularity, Concurrency, and Task-Interaction 2.1.3 Processes and Mapping 2.1.4 Processes versus Processors 2.2 Decomposition Techniques 2.2.1 Recursive Decomposition 2.2.2 Data Decomposition 2.2.3 Exploratory Decomposition 2.2.4 Speculative Decomposition 2.2.5 Hybrid Decompositions 2.3 Characteristics of Tasks and Interactions 2.3.1 Characteristics of Tasks 2.3.2 Characteristics of Inter-Task Interactions 2.4 Methods for Containing Interaction Overheads 2.4.1 Maximizing Data Locality 2.4.2 Minimizing Contention and Hot Spots 2.4.3 Overlapping Computations with Interactions 2.4.4 Replicating Data or Computations 136 2.4.5 Using Optimized Collective Interaction Operations 2.4.6 Overlapping Interactions with Other Interactions 2.5 Parallel Algorithm Models 2.5.1 The Data-Parallel Model 2.5.2 The Task Graph Model 2.5.3 The Work Pool Model 2.5.4 The Master-Slave Model 2.5.5 The Pipeline or Producer-Consumer Model 2.5.6 Hybrid Models MODULE-3 3.1 Analytical Modeling of Parallel Programs 3.1.1 Sources of Overhead in Parallel Programs 3.2 Performance Metrics for Parallel Systems 3.2.1 Execution Time 3.2.2 Total Parallel Overhead 3.2.3 Speedup 3.2.4 Efficiency 3.2.5 Cost 3.3 The Effect of Granularity on Performance 3.4 Scalability of Parallel Systems 3.4.1 Scaling Characteristics of Parallel Programs 3.4.2 The Iso efficiency Metric of Scalability 3.4.3 Cost-Optimality and the Iso efficiency Function

1.6.2 Evaluating Dynamic Interconnection Networks 1.6.3 Cache Coherence in Multiprocessor Systems

3.4.4 A Lower Bound on the Iso efficiency Function 3.4.5 The Degree of Concurrency and the Iso efficiency Function 3.4.6 Minimum Execution Time and Minimum Cost-Optimal Execution Time 3.4.7 Asymptotic Analysis of Parallel Programs 3.4.8 Other Scalability Metrics 3.5 Programming Using the Message-Passing Paradigm 3.5.1 Principles of Message-Passing Programming 3.5.2 The Building Blocks: Send and Receive Operations 3.5.3 Blocking Message Passing Operations 3.5.4 Non-Blocking Message Passing Operations 3.6 MPI: the Message Passing Interface 3.6.1 Starting and Terminating the MPI Library 3.6.2 Communicators 3.6.3 Getting Information 3.6.4 Sending and Receiving Messages 244 3.6.5 Example: Odd-Even Sort 248 MODULE - 4 4.1 Programming Shared Address Space Platforms 4.1.1 Thread Basics 4.1.2 Why Threads? 4.2 The POSIX Thread API 4.3 Thread Basics: Creation and Termination 4.4 Synchronization Primitives in Pthreads 4.4.1 Mutual Exclusion for Shared Variables 4.4..2 Condition Variables for Synchronization 4.5 Controlling Thread and Synchronization Attributes 4.5.1 Attributes Objects for Threads 4.5.2 Attributes Objects for Mutexes 4.6 Thread Cancellation 4.7 Composite Synchronization Constructs 4.7.1 Read-Write Locks 4.7.2 Barriers 4.8 Tips for Designing Asynchronous Programs 4.9 OpenMP: a Standard for Directive Based Parallel Programming 4.9.1 The OpenMP Programming Model 4.9.2 Specifying Concurrent Tasks in OpenMP 4.9.3 Synchronization Constructs in OpenMP 4.9.4 Data Handling in OpenMP 4.9.5 OpenMP Library Functions 4.9.6 Environment Variables in OpenMP 4.9.7 Explicit Threads versus OpenMP Based Programming

MODULE - I Introduction to Parallel Computing Motivating Parallelism - Scope of Parallel Computing - Parallel Programming Platforms - Implicit Parallelism: Trends in Microprocessor Architectures Limitations of Memory System Performance - Physical Organization of Parallel Platforms network topologies MODULE II Principles of Parallel Algorithm Design Preliminaries - Decomposition Techniques- Recursive Decomposition, Data Decomposition ,Exploratory Decomposition, Speculative Decomposition , Hybrid Decompositions Characteristics of Tasks and Interactions - Methods for Containing Interaction Overheads -Parallel Algorithm Models - Data-Parallel Model, Task Graph Model, Work Pool Model , Master-Slave Model, Pipeline or Producer-Consumer Model, Hybrid Models MODULE III Analytical Modeling of Parallel Programs Sources of Overhead in Parallel Programs - Performance Metrics for Parallel Systems - The Effect of Granularity on Performance -Scalability of Parallel Systems - Minimum Execution Time and Minimum Cost-Optimal Execution Time - Asymptotic Analysis of Parallel Programs - Other Scalability Metrics Principles of Message-Passing Programming - The Building Blocks: Send and Receive Operations - MPI: the Message Passing Interface MODULE - IV Programming Shared Address Space Platforms Thread Basics - The POSIX Thread API - Thread Basics: Creation and Termination - Synchronization Primitives in Pthreads -Controlling Thread and Synchronization Attributes - Thread Cancellation - Composite Synchronization Constructs - Tips for Designing Asynchronous Programs - OpenMP: a Standard for Directive Based Parallel Programming BOOKS : Introduction to Parallel Computing (2nd Edition) An Introduction to Parallel Programming
Peter Pacheco (Author)

CONTENTS

Ananth Grama (Author), George Karypis (Author), Vipin Kumar (Author), Anshul Gupta (Author)