CHAPTER 12: FILE SYSTEM IMPLEMENTATION

File System Structure File System Implementation Directory Implementation Allocation Methods Free-Space Management Efficiency and Performance Recovery Log-Structured File Systems NFS

FILE-SYSTEM STRUCTURE

Disks provide the bulk of secondary storage on which a file system is maintained can be written in place Sequential access and direct access I/O in blocks not in bytes File system allows the data on disks to be stored, located, and retrieved easily To define how the FS should look to the user To create algorithms and data structures to map the logical FS onto the physical secondary-storage devices.

File-System Structure: Layered file system

File-System Structure: Layered file system

I/O control consists of device drivers and interrupt handlers to transfer information between the main memory and the disk system. Input: retrieve drive 1, cylinder 73, track 2, sector 10 Output: low-level, hardware-specific instructions that are used by the hardware controller Basic file system issues generic commands to the appropriate device driver to read and write physical blocks on the disk Input: retrieve block 123 Output: retrieve drive 1, cylinder 73, track 2, sector 10

File-System Structure: Layered file system

The file-organization module knows about the files and their logical blocks, as well as physical blocks. To translate a files logical block addresses to its physical block addresses. Each files logical block addresses are numbered from 0 (or 1) through N. Each files physical block addresses are different, are unique within a partition. Free-space manager: Tracks unallocated blocks And provides these blocks when requested.

File-System Structure: Layered file system

The logical file system manages metadata information (metadata v.s. actual data) To manage the directory structure To manage the file structures via FCB (file control blocks) Contains information such as ownership, permissions, location of the file contents Responsible for protection and security.

File-System Structure: Example FSes

Windows: FAT (File Allocation Table) (12, 16, 32) NTFS (Windows NT File System) UNIX: UFS (Unix File system) ext2 ext3 more /proc/filesystems cd /usr/src/linux-2.4/fs # to explore for more.

FILE-SYSTEM IMPLEMENTATION

Overview: Data structures for implementing a FS On-disk structures In-memory structures

Partitions and mounting VFS

File-System Implementation: Overview

On-Disk structures Boot control block (boot block, partition boot sector) Used to boot an OS from that partition. Partition control block (super block, Master File Table) Block numbers, block size, free-block count, freeblock pointers, free FCB count and FCB pointers Directory structure used to organized the files Linear list / hash tables FCB (inode, vnode, Master File Table record) File permissions, ownership, size, location of the data blocks

File-System Implementation: Overview

In-memory structures used for both file-system management and performance improvement via caching In-memory partition table containing information about each mounted partition. In-memory directory structures containing the directory information of recently accessed directories. System-wide open-file table containing a copy of the FCB of each open file as well as other information. Per-process open file-tables containing a pointer to the appropriate entry in the system-wide open-file table, as well as other information.

File-System Implementation: Overview

To create a new file An application program calls the logical file system The logical file system To allocate a new FCB (see the next slide for FCB) To read in the appropriate directory UNIX treats a directory exactly as a file. Windows NT treats a directory as a record inside the MFT. To add a new entry To fill in the new entry with the filename and the new FCB To write it back to the disk

File-System Implementation: Overview: FCB

File-System Implementation: Overview

To open a file To pass a file name to the logical file system To search the directory for the given file name To read in the files FCB To put the files FCB to the system-wide open-file table To add a new entry in the per-process open-file table, with a pointer to the entry in the systemwide open-table and some other fields To return a pointer to the appropriate entry in the per-process file system table. (file descriptor or file handle)

File-System Implementation: Overview

To close a file To remove the entry in the per-process open-file table To decrement the system-wide open-file entrys open count. If open count is 0, copy the updated file information to the disk-based directory structure and delete this entry.

File-System Implementation: Overview

File-System Implementation: Overview

To use the file system as interface to other system aspects, such as networking. To use caches to speed up the file operations BSD UNIX To use the file system for other purpose such as networking interface.

File-System Implementation: partitions and mounting

Partitions vs disks A disk can be sliced into multiple partitions A partition can span multiple disks Partitions can either be raw or cooked Raw partition: No file system swap space, database Cooked partition: Has file system Boot block can be used for selective booting. Super block

File-System Implementation: partitions and mounting

To mount a partition before using its FS Manual mounting vs automatic mounting How to mount a partition To read in the super block via its device driver To verify its consistency To repair it if necessary (fsck) To add an entry in the in-memory mount table structure. How to mount a partition for Windows To mount at boot To mount manually

File-System Implementation: partitions and mounting

How to mount a partition for UNIX To mount a partition at a directory To add a entry at the mount-table To let one field of the mount-table entry point to the super block of the FS on that device To set a flag in the in-memory copy of the inode for that directory, indicating this directory is a mount point To set a field in the in-memory copy of the inode for that directory point to an entry in the mount table, indicating which device is mounted there.

File-System Implementation: VFS

How to support multiple FSes How to integrate many Fses into a directory structure? How to seamlessly move among various FSes? To write directory and file routines for each types. To use VFS (VFS uses oo techniques to simplify, organize, and modularize the implementation) Contributed by SUN Microsystems.

File-System Implementation: VFS

Top layer: file-system interface Open, read, write, and close and file descriptors The middle layer: VFS To separate FS generic operations from their implementation by defining a clean VFS interface The VFS is based on a file-representation, called a vnode, that contains a numerical designator for a network-wide unique file. (UNIX inodes are unique within only a single file system) The bottom layer Various FS implementation Ext3 NFS

File-System Implementation: VFS

DIRECTORY IMPLEMENTATION
Linear list

To use a linear list of file names with pointers to the data blocks. To find a file To require a linear search. To create a file To search the directory to make sure no existing file has the same file name. To add a new entry at the end of the directory. To delete a file To search the directory for the file. To remove the entry. To free the space allocated to this file.

Directory Implementation
Linear list

To reuse a directory entry To mark the entry as used. To attach it to a list of free directory entries. To decrease the length of the directory. Discussion: Simple to program, time-consuming to execute The searching is expensive Binary search To keep it sorted list To use B-tree

Directory Implementation
Hash Table

To use a linear list to store the directory entries and to use a hash table to quickly find out the directory entry given a file name. No collisions allowed (Each hash entry has a single value) Use a hash function to map file name to a hash value. Hash function should be dynamically changed. Fastest. Collisions allowed (Each hash entry has a list of multiple values) Use a hash function to map file name to a hash value and use this value to index the hash table and then search the list to find out the directory entry. Faster.

ALLOCATION METHODS

An allocation method refers to how disk blocks are allocated for files:

Contiguous allocation Linked allocation

Indexed allocation

Allocation Methods: Contiguous Allocation

The contiguous-allocation methods requires each file to occupy a set of contiguous blocks on the disk The directory for a file consists only its starting location (block #) and length (number of blocks) (See the next slide) Discussions: Supports both sequential access and direct access. Simple to implement. External fragmentation (dynamic storageallocation problem). How to specify an initial size for a file Under estimating its size Over estimating its size

Allocation Methods: Contiguous Allocation

Allocation Methods: Contiguous Allocation

Extent-based file system Extent-based file systems allocate disk blocks in extents. An extent is a contiguous block of disks. Extents are allocated for file allocation. A file consists of one or more extents. Many newer file systems (I.e. Veritas File System) use a modified contiguous allocation scheme. Contiguous allocation can be combined with other allocation methods. Contiguous allocation for small files Other allocations for large files.

Allocation Methods: Linked Allocation

With linked allocation, Each file is a linked list of disk blocks; The disk blocks may be scattered anywhere on the disk. The directory contains a pointer to the first and last blocks of the file. (See the next slide) Disk block
block = pointer

An example linked file (see the next slide)

Allocation Methods: Linked Allocation

Allocation Methods: Linked Allocation

Simple need only starting address Free-space management system no waste of space No random access Pointers waste space: to use clusters rather than sectors To improve usage To speed up Poor reliability Imagine the pointer is messed up.

Allocation Methods: Linked Allocation

FAT FS FAT (File Allocation Table) duplicated Supports direct access Cached Poor disk utilization

Allocation Methods: Linked Allocation

Allocation Methods: Indexed Allocation

Problems: External fragmentation and size-declaration for contiguous allocation Direct access for linked allocation Indexed allocation Bringing all the pointers together into one location: index block (See the next slide) Logical view

index table

Allocation Methods: Indexed Allocation

Allocation Methods: Indexed Allocation

Need index block Access methods: sequential access and direct access. Mapping from logical to physical in a file of maximum size of 256K words (or 1024KB) and block size of 512 words. We need only 1 block for index table. (512x2KB=1024KB) Index blocks waste space How large should the index block be? Linked scheme Multilevel index Combined scheme

Allocation Methods: Indexed Allocation

Linked scheme For a small file, one index block For a large file, more index blocks can be linked together. Multilevel index 1-level index block: 1024*4KB 2-level index block: 1024x1024*4KB 3-level index block: 1024x1024*1024*4KB (similar to paging)

Allocation Methods: Indexed Allocation

outer-index

index table

file

Allocation Methods: Indexed Allocation

Combined scheme 12 for direct pointers 1 single-indirect block 1 double-indirect block 1 triple-indirect block

Allocation Methods: Indexed Allocation

Allocation Methods: Performance

Two criteria: Storage utilization efficiency Data block access time Contiguous allocation: Good for known-size file Linked allocation: Good for storage utilization Indexed allocation: Access time depends on index structure, file size, block position Conclusion: Combining contiguous allocation and linked allocation (Some OS) Combining contiguous allocation and index allocation (SUN)

FREE-SPACE MANAGEMENT
Bit vector Linked Lists Grouping Counting

Free-Space Management: Bit vector

The free-space list is implemented as a bit map or bit vector. Each block is represented by 1 bit. If the block is free, the bit is 1; if the block is allocated, the bit is 0.

An example 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 0 1 1 1 1 0 0 1 1 1 1 1 1 0 0

Free-Space Management: Bit vector

Discussion: Simple to implement Efficient to find the first free block Easy to get contiguous files Block number calculation: (number of bits per word) *(number of 0value words) +offset of first 1 bit Bit map requires extra space. Example: block size = 212 bytes disk size = 230 bytes (1 gigabyte) n = 230/212 = 218 bits (or 32K bytes)

Free-Space Management: Linked Lists

Linked list (free list) Cannot get contiguous space easily No waste of space

Free-Space Management: Others

Grouping To linked blocks to store the addresses To group n-1 addresses into an address block To use the last address in an address block to point to next address block. Easier to find a large number of free blocks. Counting Every entry is a pair of (starting address, contiguous block number) rather than just a address The total list is smaller.

EFFICIENCY AND PERFORMANCE

Efficiency Preallocating the i-node on a partition (UNIX FS) Different cluster size (BSD UNIX) File size To fix the parameters or dynamically change the parameters.

Efficiency and Performance

Page Cache I/O Without a Unified Buffer Cache I/O Using a Unified Buffer Cache Reads/Writes Synchronous reads (initially) /Aynchronous reads (later on) Aynchronous writes (normally) / Synchronous write/Aynchronous (sometimes) RAM Disks vs OS caching.

Efficiency and Performance

I/O Without a Unified Buffer Cache

Efficiency and Performance

I/O Using a Unified Buffer Cache

Efficiency and Performance: Various Disk-Caching Locations

Recovery

Consistency checking Metadata are more important than actual data UNIX caches directory entries for reads, Not any data write that results in space allocation, or other metadata changes. Backup and restore Backup scheme Day 1: complete backup Day 2,3,4,n: incremental backup Save the backup in different places.

LOG STRUCTURED FILE SYSTEMS

An file operation, such as file create, can involve many structural changes within the file system on the disk. Directory structures are modified FCBs are allocated Data blocks are allocated The free counts for all of these blocks are decreased, An file operation can be interrupted, can cause inconsistency, difficult to recover Transaction (DBMS).

Log Structured File System

Log structured (or journaling) file systems record each update to the file system as a transaction. All transactions are written to a log. A transaction is considered committed once it is written to the log. However, the file system may not yet be updated. The transactions in the log are asynchronously written to the file system. When the file system is modified, the transaction is removed from the log. If the file system crashes, all remaining transactions in the log must still be performed.

NFS
An implementation and a specification of a software system for accessing remote files across LANs (or WANs). Exports mount

Homework
12.1 12.4 12.5 12.6