COMP 206: Computer Architecture and Implementation

Montek Singh
Mon, Oct 31, 2005 Topic: Memory Hierarchy Design (HP3 Ch. 5)

(Caches, Main Memory and Virtual Memory)

1

Outline
Motivation for Caches Principle of locality
Levels of Memory Hierarchy

Cache Organization
Cache Read/Write Policies Block replacement policies Write-back vs. write-through caches Write buffers

Reading: HP3 Sections 5.1-5.2

The Big Picture: Where are We Now?

The Five Classic Components of a Computer

Processor
Input Control Memory Datapath

Output

This lecture (and next few): Memory System

The Motivation for Caches

Motivation Large (cheap) memories (DRAM) are slow Small (costly) memories (SRAM) are fast Make the average access time small service most accesses from a small, fast memory reduce the bandwidth required of the large memory
Memory System

Processor

Cache

DRAM

The Principle of Locality

Frequency of reference

Address Space

2n

The Principle of Locality Program accesses a relatively small portion of the address space at any instant of time Example: 90% of time in 10% of the code Two different types of locality Temporal Locality (locality in time):
if an item is referenced, it will tend to be referenced again soon
Spatial Locality (locality in space):

if an item is referenced, items close by tend to be referenced soon

Levels of the Memory Hierarchy

Capacity Access Time Cost/bit CPU Registers 500 Bytes 0.25 ns ~$.01 Cache 16K-1M Bytes 1 ns ~$.0001 Main Memory 64M-2G Bytes 100ns ~$.0000001

Staging Transfer Unit

Upper Level Faster

Registers

Words
L1, L2, Cache Blocks

programmer/compiler 1-8 bytes

cache controller 8-128 bytes

Memory
Pages Disk Files
user/operator Mbytes OS 4-64K bytes

Disk 100 G Bytes 5 ms 10-5- 10-7 cents

Tape/Network infinite secs. 10-8 cents

Larger Lower Level

6

Tape/Network

Memory Hierarchy: Principles of Operation

At any given time, data is copied between only 2

adjacent levels

Upper Level (Cache): the one closer to the processor

Smaller, faster, and uses more expensive technology

Lower Level (Memory): the one further away from the

processor

Bigger, slower, and uses less expensive technology

Block The smallest unit of information that can either be present or not present in the two-level hierarchy
To Processor Upper Level (Cache)
Blk X

Lower Level (Memory)

From Processor

Blk Y 7

Memory Hierarchy: Terminology

Hit: data appears in some block in the upper level

(e.g.: Block X in previous slide)

Hit Rate = fraction of memory access found in upper level Hit Time = time to access the upper level

memory access time + Time to determine hit/miss

Miss: data needs to be retrieved from a block in the

lower level (e.g.: Block Y in previous slide)

Miss Rate = 1 - (Hit Rate)

Miss Penalty: includes time to fetch a new block from lower

level

Time to replace a block in the upper level from lower level + Time to deliver the block the processor

Hit Time: significantly less than Miss Penalty

8

Cache Addressing
Set 0

Set j-1 Block 0 Sector 0 Byte 0 Block k-1 Replacement info Sector m-1 Byte n-1 Valid Tag Dirty Shared

Block/line is unit of allocation Sector/sub-block is unit of transfer and coherence Cache parameters j, k, m, n are integers, and generally

powers of 2

Cache Shapes

Direct-mapped (A = 1, S = 16)

2-way set-associative (A = 2, S = 8)

4-way set-associative (A = 4, S = 4)

8-way set-associative (A = 8, S = 2)

Fully associative (A = 16, S = 1)

10

Examples of Cache Configurations

# Sets 1 j j j 64 # Blocks k 1 k 4 8 # Sectors m m 1 m 2 # Bytes n n n n 32 Name Fully associative Direct mapped A cache that is not sectored 4-way set-associative cache PowerPC 601

11

Storage Overhead of Cache

Total number of bits j repl k tag m 3 n 8 Number of data bits j k m n 8 repl k tag 3 k m 1 k m n 8
System IBM 360/85 IBM 3033 Motorola 68030 Intel i486 DEC Alpha AXP 21064 IBM PowerPC 601 # Address bits 24 32 32 32 34 32 (j,k,m,n) (1,16,16,64) (64,16,1,64) (24,4,2,2) (128,4,1,16) (256,1,1,32) (64,8,2,32) Cache size 16 KB 64 KB 256 B 8 KB 8 KB 32 KB Storage overhead 0.85% 5.95% 28.10% 19.90% 9.37% 5.76%

12

Cache Organization
Direct Mapped Cache Each memory location can only mapped to 1 cache location No need to make any decision :-)
Current item replaces previous item in that cache location

N-way Set Associative Cache Each memory location have a choice of N cache locations Fully Associative Cache Each memory location can be placed in ANY cache location Cache miss in a N-way Set Associative or Fully

Associative Cache

Bring in new block from memory Throw out a cache block to make room for the new block Need to decide which block to throw out!
13

Write Allocate versus Not Allocate

Assume that a write to a memory location causes a

cache miss

Do we read in the block?

Yes: Write Allocate No: Write No-Allocate

14

Basics of Cache Operation: Overview

READ HIT CPU reads from cache MISS Allocate and load block from MM, then CPU reads from it Write through into MM with or without write allocate

WRITE Write into cache plus write through into MM

WRITE Write into cache only Write allocate with write and set dirty bit (so that back on replacement, block is written back to MM only if modified)

15

Details of Simple Blocking Cache

HIT READ
CPU reads cache

MISS
CPU detects miss, stalls Cache selects replacement block New block loaded from MM Requested word sent to CPU CPU resumes operation CPU detects miss CPU writes MM (cache also if write allocate) stalls until write completes

Write Through
WRITE
CPU writes cache CPU writes MM and stalls until write completes

HIT READ
CPU reads cache

MISS
CPU detects miss, stalls Cache selects replacement block New block loaded from MM Word sent to CPU CPU resumes operation CPU detects miss, stalls Cache selects replacement block Old block evicted from cache New block loaded from MM (write allocate) CPU resumes operation
16

WRITE

CPU writes cache

Write Back

A-way Set-Associative Cache

A-way set associative: A entries for each cache index A direct-mapped caches operating in parallel
Example: Two-way set associative cache Cache Index selects a set from the cache The two tags in the set are compared in parallel Data is selected based on the tag result
Valid Cache Tag Cache Index Cache Data Cache Data Cache Block 0 Cache Block 0 Cache Tag Valid

:
Addr. Tag

:
Addr. Tag

Compare

SEL11

Mux

0 SEL0

Compare

OR Hit Cache Block

17

Fully Associative Cache

Push the set-associative idea to its limit! Forget about the Cache Index Compare the Cache Tags of all cache tag entries in parallel Example: Block Size = 32B, we need N 27-bit comparators
31 Cache Tag (27 bits long) 4 Byte Select Ex: 0x01 Cache Tag X X X X X Valid Bit Cache Data Byte 31 Byte 63 0

: :

Byte 1

Byte 0

Byte 33 Byte 32

:
18

Cache Block Replacement Policies

Random Replacement Hardware randomly selects a cache item and throw it out
Least Recently Used Hardware keeps track of the access history Replace the entry that has not been used for the longest time For 2-way set-associative cache, need one bit for LRU repl. Example of a Simple Pseudo LRU Implementation Assume 64 Fully Associative entries Hardware replacement pointer points to one cache entry Whenever access is made to the entry the pointer points to:
Move the pointer to the next entry
Otherwise: do not move the pointer
Replacement Pointer Entry 0 Entry 1

:
Entry 63
19

Cache Write Policy

Cache read is much easier to handle than cache write Instruction cache is much easier to design than data cache
Cache write How do we keep data in the cache and memory consistent? Two options (decision time again :-) Write Back: write to cache only. Write the cache block to memory when that cache block is being replaced on a cache miss
Need a dirty bit for each cache block Greatly reduce the memory bandwidth requirement Control can be complex
Write Through: write to cache and memory at the same time

What!!! How can this be? Isnt memory too slow for this?
20

Write Buffer for Write Through

Processor Cache DRAM

Write Buffer

Write Buffer: needed between cache and main mem Processor: writes data into the cache and the write buffer Memory controller: write contents of the buffer to memory Write buffer is just a FIFO Typical number of entries: 4 Works fine if store freq. (w.r.t. time) << 1 / DRAM write cycle Memory system designers nightmare Store frequency (w.r.t. time) > 1 / DRAM write cycle Write buffer saturation

21

Write Buffer Saturation

Processor Cache DRAM Write Buffer

Store frequency (w.r.t. time) > 1 / DRAM write cycle

If this condition exist for a long period of time (CPU cycle time too

quick and/or too many store instructions in a row)

Store buffer will overflow no matter how big you make it CPU Cycle Time << DRAM Write Cycle Time

Solutions for write buffer saturation

Use a write back cache

Install a second level (L2) cache

Processor

Cache

L2 Cache

DRAM
22

Write Buffer