Storage Area Network Basics

2004 Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr
1 2004 Cisco Systems, Inc. All rights reserved.
OPT-2T01
9899_06_2004_X
STORAGE AREA NETWORKING
PROTOCOLS AND ARCHITECTURE
SESSION OPT-2T01
OPT-2T01
9899_06_2004_X
Morning Schedule
9:00am10:30am Introduction to Storage Area Networking
Storage Terms and Acronyms
Storage Networking Devices (Switches, HBAs, Disk)
Storage Networking Applications
Storage Networking Topologies
Intro to Storage Protocols (SCSI, FC, FCIP, iSCSI)
10:30am10:45am Break
10:50am12:30pm Storage Protocols in-depth
Introduction to the Standards
SCSI
Fibre Channel
12:30pm1:30pm Lunch
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Afternoon Schedule
1:45pm3:30pm Storage Protocols In-Depth (Cont.)
Fibre Channel Services
iSCSI
FCIP
iFCP
iSNS and SLP
3:30pm3:45pm Break
3:50pm6:00pm Storage Network Troubleshooting
Required Tools
Required Technical Skill Sets
Storage Network Architecture
Design Practices
FC Network Designs
IP SANs
SAN Extension
Implementation and Management
OPT-2T01
9899_06_2004_X
Associated Sessions
OPT-1051 Introduction to Storage Topologies
and Applications
OPT-2051 Fibre Channel Storage Area Network Design
OPT-2052 FCIP Design and Implementation
OPT-2053 iSCSI Design and Implementation
OPT-2054 Storage Networking Security
OPT-3051 Troubleshooting MDS9000 Fibre Channel SAN
OPT-3052 Troubleshooting MDS9000 IP Storage
Area SAN
OPT-4051 Design and Architecture of Storage
Networking Platforms
OPT-4052 Case Study: Cisco IT Storage Strategy
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Reference Materials
Cisco Storage Networking
www.cisco.com/go/storagenetworking
Cisco AVVID Storage Networking Partner Program
www.cisco.com/go/partners
Cisco Metro Optical Product Information
www.cisco.com/go/comet
Storage Network Industry Association (SNIA)
www.snia.org
IETFIP Storage
www.ietf.org/html.charters/ips-charter.html
ANSI T11Fibre Channel
www.t11.org/index.htm
INTRODUCTION TO STORAGE
AREA NETWORKING
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Section Agenda
Storage Terms and Acronyms
Storage Networking Devices
Storage Networking Applications
Storage Networking Topologies
Introduction to Storage Protocols
OPT-2T01
9899_06_2004_X
STORAGE TERMS
AND ACRONYMS
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Technologies Overview
(or Storage in a Nutshell )
FSPF
Databases
Servers and
Mainframes
Backup
Apps
Mirroring
IP
CLOUD
IP
CLOUD
IP
CLOUD
FC Switch
FC Switch
Virtual SAN
JBODs
and NAS
Storage Area
Network (SAN)
Technologies
TAPE
Embedded
Management
SAN
Applications
SAN
Protocols
RAID &
VirtualRAID
iSCSI Drivers
FCIP
iSCSI
iSCSI
Generic
Fibre Channel
Enhanced
Fibre Channel
SAN
FC
HA
C
a
ll H
o
m
e
Support
Center
Storage
Virtualization
10 10 10 2004 Cisco Systems, Inc. All rights reserved.
OPT-2T01
9899_06_2004_X
Introduction to SAN Terminology
Block Level I/O
File Level I/O
SCSISmall Computer Systems Interface
FCFibre Channel
RAIDRedundant Array of Inexpensive Disks
iSCSIInternet SCSI
FCIPFibre Channel over TCP/IP
iFCPInternet Fibre Channel Protocol
iSNSInternet Storage Name Service
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
RAID Levels
3 without log 4
with log
Stripe with distributed parity with Random
I/O
5
N/A Fix parity with Random I/O 4
N/A Fix parity with concert I/O 3
N/A Hamming Code 2
4 Mirror then Striping/Concatenation 1+0
4 Striping/Concatenation then Mirror 0+1
2 Mirror 1
2 Striping/Concatenation 0
Min Disks Description RAID Level
OPT-2T01
9899_06_2004_X
Terminology
Direct Attached Storage (DAS)
Block level I/O
Can be internal or external
Typically SCSI or FC
Limited scalability
High cost due to management
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Terminology
Network Attached Storage (NAS)
File level I/O
Used for file sharing
applications
IP-based
Deployed over existing
low-cost Ethernet networks
Redundant links
Scalable
Multiple servers can share
same file system
IP
NAS NAS NAS
OPT-2T01
9899_06_2004_X
Terminology
Storage Area Network (SAN)
Block level I/O
Deployed as separate
network
Servers share storage
subsystem
Scalable
Multiple paths for high
availability
Presentation_ID.scr
STORAGE
NETWORKING DEVICES
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
SAN Components
Host Bus Adapter (HBA)
Interface between host and storage
Supports copper or optical
Typically one port; Can be multiple ports
1Gb, 2Gb and 4Gb
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SAN Components
Fabric Switch
1Gb, 2Gb, and 4Gb
8-40 ports
Low latency
Can be copper or optical
OPT-2T01
9899_06_2004_X
SAN Components
Director Class Switch
1Gb, 2Gb, 4Gb and 10Gb
FC and FICON
256 ports and growing
Low latency
Can be copper
or optical
Multi-service
platforms
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SAN Components
JBOD
Just a bunch of disks
Limited scalability
Typically 2 FC ports
SCSI or FC disks
Basic controllers
No caches
OPT-2T01
9899_06_2004_X
SAN Components
Storage Arrays
36GB to many TB
Typically 2 to many interfaces
Subsystems may mix interfaces
ESCON/FICON, SCSI, FC, or iSCSI
SCSI or FC disks
Intelligent controllers
Large caches
iSCSI
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SAN Components
Tape Arrays
Tape speed vary 5MBs30MBs+
Capacity vary 20GB300GB+
Deployed in servers or external
libraries
SCSI, FC, Ethernet interface
DLT most common; LTO
gaining traction
STORAGE NETWORKING
APPLICATIONS
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
IT Storage Requirements
Scalability
Meet high growth demand for storage capacity (>80% per year)
Increase capacity utilization rates
Availability
Share data across distributed data centers via fast speed,
long distance connectivity links
Provide effective disaster recovery
Improve interoperability across heterogeneous equipment
Enhance security
Manageability
Automate storage management functions
Provide cross-vendor management tools
Managing heterogeneous environments
OPT-2T01
9899_06_2004_X
Storage Network Build-Out
Application-specific
islands of networked
storage
iSCSI
Convenient extension
of existing FC SAN to
IP-attached servers
Extensive IP services
for NAS environments
Homogenous Infrastructure
Isolated Islands
Starting Point
DAS
SAN
NAS
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Storage Network Interconnection
SAN interconnection for
Business continuance
Unified management
Remote backup
Metro DWDM solutions
Low-latency option for
synch replication
FCIP
Lower-cost option for
asynch replication and
backup consolidation
SAN Interconnectivity
Present Trend
Optical
FCIP
FCIP
OPT-2T01
9899_06_2004_X
Intelligent SAN
Intelligent services
into the network
Common
management
framework
Content, file, and
block awareness
Transport
independent
Storage Utility SAN
Data Mgmt
Services
Storage
Routi ng
Host
Awareness
Storage
Management
Storage
Switching
Content
Delivery
Storage
Virtualization
Presentation_ID.scr
STORAGE NETWORKING
TOPOLOGIES
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
SCSI I/O Topology
SCSI is the protocol used to communicate
between servers and storage devices
SCSI I/O channel provides a half-duplex
pipe for SCSI commands and data
Parallel implementation
Bus width: 8, 16 bits
Bus speed: 580 Mhz
Throughput: 5320 MBps
Devices/bus: 216 devices
Cable length: 1.5m25m
A network approach can scale the
I/O channel in many areas (length,
devices, speed)
SCSI
Target
SCSI Adapter
Host System
Initiator
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fibre Channel Topology
Very common method for
networking SCSI
Fibre Channel provides high-speed
transport for SCSI payload
Fibre Channel overcomes many
shortcomings of DAS including:
Addressing for up to 16 million nodes (24 bits)
Loop (shared) and Fabric (switched)
transport
Speeds of 100 or 200 Mbps (1 or 2 Gbps)
Distance of up to 10km (without extenders)
Support for multiple protocols
Combines best attributes of a channel
and a network
SCSI
Target
Fibre
Channel HBA
Host System
Initiator
Fibre Channel
Fabric
OPT-2T01
9899_06_2004_X
iSCSI Storage Topology
IP access to open
storage sub-systems
iSCSI driver is loaded
onto hosts on ethernet
network
Able to consolidate servers
via iSCSI onto existing
storage arrays
Able to build ethernet-based
SANs using iSCSI arrays
Storage assigned by iSCSI
instance
iSCSI-Enabled
Hosts (Initiators)
iSCSI
Array
(Target)
Storage
Pool (Target)
iSCSI
Router
IP
Network
FC
Fabric
FC HBA
Attached
Host
(Initiator)
iSCSI
iSCSI
iSCSI
iSCSI
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FCIP SAN Extension Topology
FCIP gateways perform Fibre Channel encapsulation process into
IP packets and reverse that process at the other end
FC Switches connect to the FCIP gateways through an E_Port for
SAN fabric extension to remote location
A tunnel connection is set up through the existing IP network
routers and switches across LAN/WAN/MAN
Standby
Storage
Production
Storage
Production Site Production Site
Database Servers
EMC SRDF
Existing IP
Network
LAN/WAN/MAN
FC Switch
FC
SAN
Backup
Server
FCIP FCIP
Gateway Gateway
FCIP FCIP
Gateway Gateway
Servers
FC
SAN
FC Switch
Backup, R&D, Shared Storage,
Data Warehousing, Etc.
Backup, R&D, Shared Storage,
Data Warehousing, Etc.
OPT-2T01
9899_06_2004_X
FCIP and iSCSI: Complementary
FCIP: SAN-to-SAN over IP
iSCSI: Host to storage over IP
IP Network
FC SAN
Storage
Router
FC SAN
i
S
C
S
I
i
S
C
S
I i
S
C
S
I
i
S
C
S
I
Storage
Router
FCIP
Gateway
FCIP
Gateway
FCIP FCIP
Presentation_ID.scr
INTRODUCTION TO
STORAGE PROTOCOLS
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Introducing SCSI
SCSI = Small Computer System Interface
SCSI is a standard that defines an interface
between an initiator (usually a computer) and
a target (usually a storage device such as a
hard disk)
INTERFACE refers to connectors, cables,
electrical signals, optical signals and the
command protocol that allow initiators
and targets to communicate
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sun Sun
Opcode (2A =Write 10)
Reserved
LBA
LBA
LBA
(0010E43)
Reserved
LBA
LBA
Len
(128)
Control
SCSI Example
Initiator Target 2 Target 1
SCSI Connector SCSI Cable
The SCSI Command
Protocol Is Used to
Communicate Between
SCSI Devices
SCSI Command
In this Case, a
File is Being
Written to the
Hard Drive By an
Application on
the Workstation
Tape
Disk
OPT-2T01
9899_06_2004_X
Why Is SCSI Important for SANs?
SCSI command protocol is the de facto standard
that is used extensively in high-performance
storage applications
The command part of SCSI can be encapsulated
in FCPFibre Channel Protocol or IP and
carried across internetworks; This is the core
concept behind storage area networking
To understand the finer points involved with
transporting SCSI across a network with FC
or ethernet, the basics of SCSI must be well
understood
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Standards
SCSI has evolved since it was introduced as
SASI in 1979 by Shugart Associatesit was
approved as a standard by ANSI in 1986 and
is now referred to as SCSI-1
SCSI-2 was approved by X3 in 1990 and by
ANSI in 1994
SCSI-3 refers to a collection of standards, each
of which defines a very specific part of SCSI:
physical interface, transport interface, command
interface, architecture model, programming
interface, etc.
OPT-2T01
9899_06_2004_X
Sample SCSI Standard Components
SCSI Parallel Interface: SPI
Sun
Sun
Initiator
Target 1
Target 2
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sample SCSI Standard Components
SCSI Primary Commands: SPC
Sun Sun
Initiator Target 2 Target 1
SCSI Primary Commands
(SPC-2)
SCSI Block Commands
(SBC)
SCSI Stream Commands
(SSC)
OPT-2T01
9899_06_2004_X
SCSI Standards: The Big Picture
FC-xx 1394
SPI-x
FCP SBP
ATAPI
SPC-2 / SPC-3
CAM ASPI Generic
SBC SSC SES More
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
This specification describes a reference model
for the coordination of standards applicable to
SCSI-3 I/O systems and a set of common
behavioral requirements which are essential for
the development of host software and device
firmware that can interoperate with any SCSI-3
interconnect or protocol.
SCSI Architecture Model
November 1995
OPT-2T01
9899_06_2004_X
The SCSI architecture
model defines generic
requirements and
implementation
requirements
Each SCSI
implementation
standard must fulfill
the requirements set
forth by SAM
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SAM Highlights: Client-Server
SCSI is a client-server
protocol
The client is called the
initiator (this is usually
the OS I/O subsystem)
and issues requests to
the server
The server is called the
target (this is usually the
SCSI controller that is
part of a storage device)
and receives, executes
and returns initiator
requests and their
associated responses
OPT-2T01
9899_06_2004_X
SAM Highlights: Initiator: Target
A single initiator can
have multiple
application clients
Targets have ONE task
manager and one OR
MORE Logical Units
(LU), which are
numbered (LUN)
The task manager has
the authority to modify
service requests that
have already been
received by the target
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SAM Highlights: Logical Units
Each logical unit within
a target is numbered;
that number is called a
LUN and is the only
way to refer to that
logical unit
The device server is the
entity that receives,
executes and returns
requests that are made
to its logical unit
The concept of task set
is beyond the scope of
this presentation
OPT-2T01
9899_06_2004_X
SAM Highlights: Command Model
SAM defines two categories of protocol services:
Execute command/confirmation services;
Data transfer services
This leads to the three main phases of a data
transfer:
1. Execute: Send required command and parameters
via CDB;
2. Data: Transfer data in accordance with the command;
3. Confirmation: Receive confirmation of command
execution
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SAM Highlights: Sample Data READ
1. Send SCSI Cmd issued
by initiatorthe
command sent is READ;
2. SCSI command received
by target;
Data transfers occur
during the working
phase between initiator
and target;
3. Send command complete
is returned by the target;
4. Command complete
received by target

OPT-2T01
9899_06_2004_X
SAM Highlights: Parameters
The data transfer model reflects parameters that
will be used by SCSI commands
This model illustrates that a complete data transfer
(right) can be broken up into multiple parts (left)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SAM Highlights: Communication Model
SAM Defines a Hierarchy of Protocols
Lets Expand on
this Portion
OPT-2T01
9899_06_2004_X
SCSI Transport Protocol
IP IP
TCP TCP
iSCSI iSCSI
SCSI Protocol SCSI Protocol
FibreChannel FibreChannel
FCP FCP Parallel Bus Parallel Bus
Ethernet Ethernet
iFCP iFCP
FCIP FCIP
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SCSI Transport Protocol
IP IP
TCP TCP
iSCSI iSCSI
SCSI Protocol SCSI Protocol
FibreChannel FibreChannel
FCP FCP Parallel Bus Parallel Bus
Ethernet Ethernet
iFCP iFCP
FCIP FCIP
Todays In-Depth
Protocol
Discussions
STORAGE PROTOCOLS
IN-DEPTH
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Section Agenda
Introduction to Standards
SCSI Protocol
Fibre Channel Protocol
Internet SCSI (iSCSI)
Fibre Channel over IP (FCIP)
Internet Fibre Channel Protocol (iFCP)
iSNS and SLP
INTRODUCTION TO STANDARDS
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Standards Groups: Storage
ISO / IEC
J TC-1
InterNational Committee for Information
Technology Standards
(INCITS)
American National
Standards Institute
(ANSI)
Techincal Committee on
Lower-Level Interfaces
(T10)
Information Technology
Industry Council
(ITI)
Techincal Committee on
Device-Level Interfaces
(T11)
C
J 11
C++
J 16
Techincal Committee on AT
Attachment Interfaces
(T13)
SCSI Fibre Channel
HIPPI
IPI
ATA (IDE)
ATAPI
www.t10.org www.t11.org www.t13.org
OPT-2T01
9899_06_2004_X
Standards Process
Technical Committees (T10) write drafts
Drafts are sent to INCITS for approval
Once approved by INCITS, drafts become standards
and are published by ANSI
ANSI promotes american national standards to ISO
as a Joint Technical Committee member (JTC-1)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Standards Work Group: IP Storage
IP Storage Technical Work Group Acts as Primary
Technical Focal Point of the Storage Networking
Industry Association (SNIA) on IP Storage Issues,
Coordinating with the SNIA IP
Storage Forum
Transport AreaHas 23
WGs, One which Is the IP
Storage WG
ISOC
Internet Society
IESG
Internet Engineering
Steering Group
IETF
Internet Engineering
Task Force
Transport Area
IEFT Is the Organization
Ratifying the IPS Standards
FIBRE CHANNEL IN-DEPTH FIBRE CHANNEL IN-DEPTH
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fibre Channel Protocol Agenda
FC Introduction
Fibre Channel Communications Model
Protocol Constructs
FC-PH (Fibre ChannelPhysical and Signaling
Interface)
Login Parameters
Frame Processing
Arbitrated Loop
Switch Fabric Operation
Switch and Hub Mixed Topology Network Operations
FC Error Management
OPT-2T01
9899_06_2004_X
Fibre Channel Environment
Channel reliability
Multiprotocol support
Overshared serial media
With networking capability and functionality
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fibre Channel Environment
High bandwidth
High data integrity
Highly reliable
Destination paced
Buffer credits
Scalable
High availability
Shared media
Circuit/packet
Multiple protocol
support
Transport flexibility
Dedicated connClass 1
MultiplexedClass 2
DatagramClass 3
Configuration flexibility
Switch
Loop
OPT-2T01
9899_06_2004_X
What Is It?
Channels
Connection service
Physical circuits
Reliable transfers
High speed
Low latency
Short distance
Hardware intense
Networks
Connectionless
Logical circuits
Unreliable transfers
High connectivity
Higher latency
Longer distance
Software intense
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
What Is It?
Fibre Channel
Channels
Connection service
Physical circuits
Reliable transfers
High speed
Low latency
Short distance
Hardware intense
Networks
Connectionless
Logical circuits
Unreliable transfers
High connectivity
Higher latency
Longer distance
Software intense
Fibre Channel
Circuit and packet
switched
Reliable transfers
High data integrity
High data rates
Low latency
High connectivity
Long distance
OPT-2T01
9899_06_2004_X
Fibre Channel Protocol Levels
Levels
HIPPI
Cluster 370
OEM
SCSI IP
ATM
Common Services
FC-4s
FC-3
Signaling Protocol
Transmission Code
Physical Interface
FC-2
FC-1
FC-0
FC-PH
N_Port
F_Port
PC-PH = Physical and Signaling Layer
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fibre Channel Functions
Structure Is Divided into 5 Levels of Functionality
FC-0 defines the physical interface characteristics
Signaling rates, cables, connectors, distance capabilities, etc.
FC-1 defines how characters are encoded/decoded
for transmission
Transmission characters are given desirable characters
FC-2 defines how information is transported
Frames, sequences, exchanges, login sessions
FC-3 is a place holder for future functions
FC-4 defines how different protocols are mapped to use
Fibre Channel
SCSI, IP, virtual interface architecture, others
OPT-2T01
9899_06_2004_X
Fibre Channel Topologies
Point to point
Arbitrated loop
Switched fabric
FC
N
N N
L
L
L L
L
L
N
N
N
F F
F F
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Point to Point
Dedicated connection between N port Fibre
Channel devices
All link bandwidth is dedicated to communication
between the two nodes
Suitable for small scale scenarios when storage
devices are dedicated to file servers
N
N
N
N
OPT-2T01
9899_06_2004_X
Arbitrated Loop (FC-AL)
TX of each node is connected to the
RX of the next node until a closed loop
is formed
Maximum bandwidth: 100 MB/sec.
(shared amongst all nodes on loop)
126 nodes max on loop
Not a token passing schemeno limit
on how long a device may retain control
Operational sequence:
Arbitrate for control of loop
Open channel to target
Transfer data
Close
Number of nodes on loop directly
affects performance
Fibre Channel Hub
FC
L
L
L
L
L
L
L
L
L
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Data Integrity
Upper Level Protocol
Signaling Protocol
Transmission Code
Physical Media
Operation Control
and Byte Counts
Operation
Frame counts
CRC (32 bit)
Frame delimiters
8b/10b Code
Fibre Reliability
OPT-2T01
9899_06_2004_X
Flow Control
Back pressure technique
Frame credit
Established by receiver during LOGIN
Transmitter
Must have credit to transmit
Receiver
Reinstates credit with ACK
Presentation_ID.scr
FIBRE CHANNEL
COMMUNICATIONS MODEL
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
The Model
The Fibre Channel communications model
is based on the definition of:
Physical objects
Protocol construct
These objects and constructs:
Define the behavior of the physical elements
Control the transfer on information
Provide for link management
Provide the basis for:
Hardware
Firmware
Software
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Physical
The fundamental physical objects in Fibre
Channel are:
Ports
Link
Nodes
Fabric
Some Logical Items Used in These Discussion Are:
Addressing
Communications Model
OPT-2T01
9899_06_2004_X
Fibre Channel: Port Types
N port: Node ports used for connecting peripheral
storage devices to switch fabric or for point to point
configurations; can be considered the end port
F port: Fabric ports reside on switches and allow
connection of storage peripherals (N port devices)
L port: Loop ports are used in arbitrated loop
configurations to build storage peripheral networks
without FC switches; these ports often also have N
port capabilities and are called NL ports
E port: Expansion ports are essentially trunk ports
used to connect two Fibre Channel switches
G port: A generic port capable of operating as either
an E or F port; its also capable of acting in an L port
capacity; Auto Discovery
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
N_Port
Host / Device
N_Port
Host/ Device
Interface
Serial Data Out Serial Data In
OPT-2T01
9899_06_2004_X
Link
A link consists of
2 unidirectional fibers transmitting in opposite directions
May be either:
Optical fiber
Copper
Transmitters may be:
Long wave laser
Short wave laser
LED
Electrical
Link Transfer Rates
25 265.5
100 106.25
Mbytes/sec Clock
Mbaud/sec
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Link
Link
Host / Device
N_Port
Host/ Device
interface
Serial Data Out Serial Data In
OPT-2T01
9899_06_2004_X
Node
The equipment which contains one or more
N_Port or NL_Port (topology dependent)
May be
Computer
Controller
Device
Is NOT a switch fabric
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Node
N_Port
Link
N_Port
Link
N_Port
Link
N_Port
Link
Controller
OPT-2T01
9899_06_2004_X
Point to point
N_Port to N_Port
Flow control
Acknowledged
Node Node
Transmitter
Receiver
Transmitter
Receiver
N_Port
Node Node
Link
Transmitter
N_Port
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fabric
Fabric
The entity which interconnects N_Ports
Provides routing based on destination address
Fabric may be:
Point to pointNo routing required
SwitchedRouting provided by switch
Arbitrated loopRouting is distributed throughout
attached L_Ports
OPT-2T01
9899_06_2004_X
Terms
Topology
The physical structure of the interconnect of ports
Defines the logical behavior of transactions
Fibre channel has 3 topologies
Pt to Pt
Switched
Arbitrated loop
Fabric
The fabric is the generic item that interconnects nodes
A fabric is made of Fibre Channel topologies like Pt to Pt,
switches and loops
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Point to Point
Source to destination
Based on address routing through the fabric
Node Node
Transmitter
Receiver
Receiver
Node Node
Fabric
N_Port
N_Port
Transmitter
OPT-2T01
9899_06_2004_X
Switched Fabric
N_Port N_Port N_Port N_Port N_Port N_Port
Fabric
Switch
Communications ModelSourceto Destination Based on
Address Routing through the Fabric
A B
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Communications ModelSource to Destination Based on
Address Routing Distributed in the NL_Ports on the Loop
Arbitrated Loop
NL_Node
A
NL_Node
B
Link
A B
FC PROTOCOL CONSTRUCTS
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
What Are Protocol Constructs
The fundamental protocol structures in the Fibre
Channel are called constructs, and they are:
Frames
Sequences
Exchanges
Information Units (IU)
Procedures
Upper Layer Protocols (ULPs)
OPT-2T01
9899_06_2004_X
Construct Introduction
FC-2 defines these constructs that allow the related
information to be:
Grouped together
Coordinated
Handled in an efficient manner
To accomplish this we define the notion of:
Frames
Sequences
Exchanges
Also defined are means for the Upper Level Protocols ULPs
to communicate with FC-2:
A procedure called the login defines the operating
environment between the N_Ports
Exchange of the data describing the characteristics of the ports
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Chunks
The ULPs deal with chunks of data that are
moved across the network
These chunks of data may be either
Control
Status
Real data
OPT-2T01
9899_06_2004_X
Frames
Frame
FC-2 layer will take this
chunk of data and move
it from
Transmitting node to
recei ving node In the units
of what Fibre Channel calls
frames
Frame Size
FC-2 will determine the size
of the frames based on
operating environment
established between the
two communicating nodes
Common Services
Signaling Protocol
Transmission Code
Physical Interface
FC-3
FC-2
FC-1
FC-0
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Frame Structure
24* 4 24 0-2114 4 4 Bytes
Idles SOF
Frame
Header
Data Field CRC
EOF
Idles
* 6 Idle Words (24 bytes) Requires by TX
2 Idle Words (8 bytes) Guaranteed to RX
General FC-2 Frame Format
Frame Format
CRC Calculated on Frame
Header and Data Field Only
OPT-2T01
9899_06_2004_X
Frame Header
R_CTL Routing D_ID 24bits Destination
0 7 8
1
5
1
6
2
3
2
4
3
1
Word
0
1
2
3
4
5
S_ID 24 bits Source
CS_CTL 8 bits
Class Spec
TYPE 8 bits
Data structure
SEQ_ID 8 bits
DF_CTL 8 bits
Data field
F_CTL 24 bits Frame Control
SEQ_CNT 8 bits Sequence Count
OX_ID 16 bits Orig Exch ID
RX_ID 8 bits Respon Exch ID
Parameter Specific to frame type
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Data Field
Optional Headers
Payload
F
I
L
L
Data Field 0-2114
0 - 64 0 - 2112
1 - 3
02048 Typical MTU
OPT-2T01
9899_06_2004_X
Sequence
Sequences
Each chunk of Upper Level Protocol (ULP) data is
moved within the envelope of what Fibre Channel
calls a Sequence (SEQ)
A sequence consists of a set of related frames
As expected there are lots of rules governing sequences
The ULP tells the FC-2 how to transfer theses chunks of
data through a structure called a information unit
Very few rules for IUs
IU is a convention defined outside of FC-PH
IUs are unique to each upper level protocol
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sequence
Sequence Initiator (SI)
The N_Port which is
transmitting the data
frames
Sequence Recipient (SR)
The N_Port which is
receiving the data frames
Fabric
SI N_Port
SR N_Port
Data
Chunk
Data
Chunk
D
a
t
a
F
r
a
m
e
OPT-2T01
9899_06_2004_X
Sequence
Fabric
Initiator (SI)
Target
Data Frame
Sequence
Read Command
(Chunk)
(SR)
(SI)
Sequence
(SR)
Data
(Chunk)
Status
Data Frame
Data Frame
Sequence
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sequence Identifier
Sequence initiator assigns an identifier to each
sequence
This identifier is called the Sequence_Identifier or Seq_ID
The Seq_ID uniquely identifies a given sequence within
the context of the operation
Each frame is identified within this operation by Seq_ID
and Seq_CNT
OPT-2T01
9899_06_2004_X
Sequences: Active and Open
A sequence is ACTIVE
From the time the first frame of the sequence is
transmitted until the frame with the end sequence
flag is sent
A Sequence is OPEN
From the time the first frame is transmitted until the
reception of the ACK to the last frame
Sequence Recipient (SR)
A sequence is ACTIVE and OPEN
From the time of the first frame of the sequence is
received until the transmission of the ACK to the last
frame of that sequence
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sequences: Active and Open
Originator (SI)
Responder (SR)
First Data_Frame
SOF Received
EOF Transmitted
Frame with End_SEQ set
ACK to first Frame
ACK to last Frame
EOT Received
Open
Acti ve
Acti ve
&
Open
OPT-2T01
9899_06_2004_X
Sequence Streaming
Sequence streaming is the ability to
Begin transmission of the next sequence while one or more
previous sequences are OPEN
Sequence Recipient (SR) grants permission to have
up to n streaming sequences; This is determined
at N_Node login time
Must Support n=1 sequence status blocks (state info)
(This Allows for More Data in the Pipe for Distant Connections)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Exchange
Upper level protocols frequently deal with related
bits of data as:
Request/reply
Command/data/status
These relationships are called operations
Exchanges
Operations of data grouped together into what
Fibre Channel call exchanges
An exchange consists of a set of related sequences
Exchanges are bi-directional
Sequences are unidirectional and sequential
There are other rules that govern exchanges
OPT-2T01
9899_06_2004_X
Exchange
Fabric
Initiator (SI)
Target
Data Frame
Sequence
Read Command
(Chunk)
(SR)
(SI)
Sequence
(SR)
Data
(Chunk)
Status
Data Frame
Data Frame
Sequence
Exchange
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Exchange
Exchange originator
The N_Port which transmitted the FIRST data frame for
this exchange
Exchange responder
The N_Port which is the destination of the FIRST data
frame of this exchange
The designation for the originator and responder are
fixed for the duration of the exchange
Unlike the SI and SR Which Change Roles Within the Exchange
OPT-2T01
9899_06_2004_X
Exchange Identifiers X_ID
An exchange has two identifiers
associated with it
Exchange originator:
Assigns an OX_ID which is meaningful to it
Exchange responder:
Assigns a RX_ID which is meaningful to it
In general terms it is called the X_ID
Meaningful is that in the exchange there is
context with information like state, control,
and status with regards to the exchange
An N_Port will save, create and update this
information throughout the exchange based on
the assigned X_IDs
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Information Unit
Upper Level Protocols (ULPs) know about
Information Units (IUs) but know nothing about:
Frames
Sequences
Exchanges
A ULP deals with units like:
Order of events within the operation
Which node will transmit in the next phase
(Command phase, data phase, status phase)
Is required to have some knowledge about Fibre Channel
An information unit is a Fibre Channel sequence
OPT-2T01
9899_06_2004_X
Information Unit
The IU contains information sets with such items
as LUN, task attributes, CDB and the command
byte count
The IUs are used in protocol mapping from FC-4
to FC-2 and are assigned an identifier that is useful
to humans not used by the machine
All the information needed to support a ULP is
formed into a IU table and is listed as a first ,
middle or last IU in the exchange
We Will See More of these Tables when We
Cover SCSI Mapping onto Fibre Channel
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FC-2 Hierarchy
Construct Meaning
OX_ID / RX_ID
Exchange
SEQ_ID
Information
Unit
Sequence
Frame
SEQ_CNT
Frame Fields
Consists of one or more
Sequences for ULP Operation
The structure used by the ULP to
define a Sequence (not visible over link)
Consists of one or more related
Frames
Contains in its Payl oad a ULP
chunk of data
The Hierarchy of Constructs
OPT-2T01
9899_06_2004_X
FC-2 Hierarchy
Frame Fields
OX_ID & RX_ID
...
...
EXCHANGE
SEQ_ID
SEQ_CNT
SEQUENCE
Frame Frame Frame
Information Unit Per ULP Terms
Presentation_ID.scr
FC-PH (FIBRE CHANNEL:
PHYSICAL AND SIGNALING
INTERFACE) STRUCTURE,
PROCEDURES, AND PROTOCOLS
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Transmission Code
Fibre Channel uses a 8b/10b transmission code
Each 8 bit data byte to be transmitted is converted into a
10 bit quantity
The 10 bit quantity is then transmitted over the FC media
The 10 bit quantity is then converted back to the 8 bit data
byte by the receiving node
The 10 bit quantities are called transmission
characters
Transmission characters come in two forms
Data charters
Special characters
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
8b/10b Code
Why 8b/10b
1. To ensure the sufficient transitions are present
in the serial bit stream to make clock recovery
possible at the receiver
2. Increase the likelihood of detecting any single
or multiple bit errors
3. To provide special characters with distinctive
and easily recognizable characters to achieve
word alignment on the incoming bit stream
OPT-2T01
9899_06_2004_X
8b/10b Code
Characteristics of 8b/10b
The 10 bit transmission code
Supports all 256 values of the 8 bit data byte
Contains unused code points
Illegal codes(called code violations)
Detection of code violations
May occur on the transmission character in which the error
occurred or may be detected on a subsequent character
Contains special characters
Running disparity with DC balance
(Count of 0s and 1s Equal the Same over Time)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
8b/10b Code
Running Disparity
Disparity: The Difference Between the Number of Ones and
Zeros in a Transmission Character
Running Disparity: A Binary Parameter Indicating the Cumulati ve
Disparity of All Previously Issued Transmission
Characters
Transmission Characters Always Have Either:
6 Ones and 4 Zeros = Positi ve Disparity
4 Ones and 6 Zeros = Negative Disparity
5 Ones and 5 Zeros = Neutral Disparity
Rules: A Positive Disparity Transmission Character Can Not Be Followed By
Another Positi ve Transmission Character
A Negati ve Disparity Transmission Character Can Not Be Followed By
Another Negative Transmission Character
At Transmission Character Boundaries the Difference between the Number of
Ones and Zeros is + or 1
OPT-2T01
9899_06_2004_X
8b/10b Code
Code Notation
Each valid transmission character has been
assigned a name in the form of:
Zxx.y
Z = K or D D=Data K=Special Character
xx = Decimal Value of the 5 LSb bits
y = Decimal Value of the 3 MSb bits
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Conversion Table
A B C D E i F G H j
0 1 2 3 4 5 6 7
FC-2 Bits
FC-1 Code Bit
LSB
MSB
D or K
.
j and i are add as part of the 10b conversion process
0 0 0 0 1 0 0 0
Example
D1.0
FC-1 Transmission
Character
0 1 1 1 0 1 0 1 0 0
Neg Disp Value
Transmission Order
OPT-2T01
9899_06_2004_X
Special Characters
K28.5 only special character used in Fibre
Channel out of the 12 set aside
Has no 8 bit representation
The only FC transmission character with 5
consecutive 1s or 0s
Used to find word boundaries and sync
Used in ordered sets
0 0 1 1 1 1 1 0 1 0 + Current Running Disparity
1 1 0 0 0 0 0 1 0 1 - Current Running Disparity
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Transmission
Transmission word consists of 4 continuous
transmission characters treated as a unit
40 bits long
Aligned on a word boundary
There is a ordered set and a data word
Encoded Data
Byte
Encoded Data
Byte
Encoded Data
Byte
K28.5
Byte
0
1 2 3
Ordered Set
Encoded Data
Byte
Encoded Data
Byte
Encoded Data
Byte
Encoded Data
Byte
Data Word
Transmission Order
OPT-2T01
9899_06_2004_X
Ordered Set
Transmission word starting with the K28.5 special
character
Three classifications of ordered sets are defined
Delimiters
Primitive signals
Primitive sequences
Dxx.y Dxx.y Dxx.y K28.5
MSB LSB
The Three Data Characters Define the Meaning of the Ordered
Set and Are Repeated for the Third and Fourth Character
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Primitive Signals
Primitive signals are ordered sets
Transmission of primitive signals are interrupted
occasionally to transmit frames
Three basic types
Receiver_Ready (R_Rdy)
Idle (idle or I)
Arbitrate (ARBx)
OPT-2T01
9899_06_2004_X
Delimiters
Delimiters are ordered sets that delineate a frame
Immediatel y preceding and following the contents of a frame
Two basic types
Start_of_Frame (SOF)
End_of_Frame (EOF)
SOF delimiters
Identify the start of a frame
Identify the transmission class
Used to establish a Class_1 connection
Identify the beginning and continuation of a sequence
EOF delimiters
Terminate frames
Identify the end of a sequence
Terminate connections
Indicate known frame errors
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FC-1 Synchronization
Procedures
Sync acquire
Initialization
Loss of sync procedure
Primitive sequences
OPT-2T01
9899_06_2004_X
Sync Procedures
Bit synchronization
The state in which a receiver is delivering retimed serial data at
the required bit error rate
Transmission word synchronization
Achieved when the receiver identifies the same transmission
word boundary on the receive bit stream as the established by
transmitter at the other end of link
Acquired by detection of three consecuti ve ordered sets
without errors
Loss of synchronization procedure
The recei ver shall enter the loss-of-sync state upon detection of
the fourth invalid transmission word
Synchronization acquired procedure
The recei ver shall enter the synchronization-acquired state when
it has achieved both bit and transmission word sync
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Synch Acquired
Loss of Sync State
Bit Sync Acquired
Rx Ordered set #1
Rx Ordered set #2
Rx Ordered set #3
Sync Acquired
Data Word
Data Word
Data Word
Waiting on Bit Synchronization
OPT-2T01
9899_06_2004_X
Loss-of-Sync Procedure
Sync Acquire State
No Invalid
Words Detected
First Invalid Word
Second Invalid Word
Third Invalid Word
Loss Of Sync
Fourth Invalid Word
One Invalid
Word in Next
2 Words
One Invalid
Word in Next
2 Words
One Invalid
Word in Next
2 Words
One Invalid
Word in Next
2 Words
Two
Consecutive
Valid Words
Two
Consecutive
Valid Words
Two
Consecutive
Valid Words
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FC-1 Constructs
Port states
Primitive sequences
NOS/OLS/LR/LRR
Primitive sequence protocols
Sequence flows
Relationships
Port state transition table
OPT-2T01
9899_06_2004_X
Port States
Four primary operational states
Active state
Link recovery state
Link failure state
Offline state
Operational states of a port
N_Ports
F_Ports
Port state changes occur as a result of
Conditions detected within the port
In response to reception of primitive sequences
In response to upper level controlling entity
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Primitive Sequences
Ordered set that is transmitted continuously to
indicate that specific conditions within the port
are encountered
Transmitted while the condition exist
Four primitive sequences
Not Operational Sequence (NOS)
Offline Sequence (OLS)
Link Reset Sequence (LR)
Link Reset Response Sequence (LRR)
OPT-2T01
9899_06_2004_X
Primitive Sequence NOS
Not_Operational Sequence
Transmitted by the port to indicate that
Link failure had been detected
Loss of sync
Loss of signal
Port is offline
D5.2 D31.5 D21.1 K28.5
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Primitive Sequence OLS
Offline Sequence
Transmitted by port to indicate that it is:
Initiating the link initialization protocol
Recei ving NOS
Entering the Offline state
D21.2 D10.4 D21.2 K28.5
OPT-2T01
9899_06_2004_X
Primitive Sequence LR
Link Reset Sequence
Transmitted by port to indicate that it is:
Initiating the link reset protocol
To recover from a link timeout
To remove a Class_1 connection
D9.2 D31.5 D9.2 K28.5
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Primitive Sequence LLR
Link Reset Response Sequence
Transmitted by port to indicate that:
Link reset is being received
D9.2 D31.5 D21.1 K28.5
OPT-2T01
9899_06_2004_X
Primitive Sequence Protocols
Link Initialization Protocols
Required after
Port power-on
Port internal reset
Port has been in offline state
Online to offline protocols
Required to enter offline state
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Primitive Sequence Protocols
Link Failure Protocol
Required after
Detection of loss of synchronization for a period of time
greater than 100ms which is the receiver-transmitter time-
out value (R_T_TOV)
Loss of signal while not in the offline state
Link Reset Protocol
Required after
Link reset
Link timeout
OPT-2T01
9899_06_2004_X
Primitive Sequence Flows
Link Failure
State (LF)
Offline
State (OL)
Link Recovery
State (LR)
Active State
(AC)
NOS
L
i
n
k

F
a
i
l
u
r
e

P
r
o
t
o
c
o
l
L
i
n
k

I
n
i
t
i
a
l
i
z
a
t
i
o
n

P
r
o
t
o
c
o
l
L
i
n
k

R
e
s
e
t

P
r
o
t
o
c
o
l
Idle
O
n
l
i
n
e

t
o

O
f
f
l
i
n
e

P
r
o
t
o
c
o
l
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Primitive Sequence Meanings
Idles or
R_RDY
Operational Link
Idles and R_RDY recognized
IDLE
Idles
Link Reset Response
Link Reset Recognized
LRR
LRR
Link Reset
Remove class_1 Conn
Reset F_Port
OLS recognized
LR
LR
Offline State
Internal port failure
Transmitter power down, performdiags, or
performinitialization
Receiver shall ignore Link error or Link
Failure
OLS
OLS
Not Operational
Link Failure
NOS
Transmit in
Response
Meaning Currentl y
Transmitting
OPT-2T01
9899_06_2004_X
AC = Activity State
LR = Link Recovery State
LF = Link Failure State
OL = Offline State
Link Failure Condition
AC AC
LF
LF
LR
OL
LR
AC
AC
NOS
OLS
LR
Idle
LRR
Idle
Port A Port B
Link Failure
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Offline
AC AC
OL
OL
LR
LR
AC
AC
OLS
LR
LRR
Idle
Idle
Idle
Port A Port B
Request to Go
Offline
After 5ms Minimum
Diags May Be
Preformed
Request to Go
Online
AC = Activity State
OL = Offline State
OPT-2T01
9899_06_2004_X
Frame Header Detail
Routing control (R_CTL)
Addressing (D_ID) (S_ID)
Type (TYPE)
Frame control (F_CTL)
Sequence identifier (SEQ_ID)
Sequence count (SEQ_CNT)
Exchange identifiers (OX_ID) (RX_ID)
Parameter field
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Frame Detail: Routing Control
0 7 8
1
5
1
6
2
3
2
4
3
1
Word
0
1
2
3
4
5
S_ID 24 bits Source
CS_CTL 8 bits
Class Spec
TYPE 8 bits
Data structure
SEQ_ID 8 bits
DF_CTL 8 bits
Data field
OPT-2T01
9899_06_2004_X
Routing Control
The Routing control field is an 8 bit field
R_CTL consist of two 4 bit sub-fields
Routing
Information category
Info Category Routing
31 24 27 28
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Routing Control
The R_CTL is used to direct the frame to the
process the frame is directed to; For example:
Frames directed to the fabric for extended link
services (0x22)
Indication of the function or purpose of the frame
payload from the upper level protocol at FC-4 (0x01)
OPT-2T01
9899_06_2004_X
Port Addressing
R_CTL Routing D_ID 24 bits Destination
0 7 8
1
5
1
6
2
3
2
4
3
1
Word
0
1
2
3
4
5
S_ID 24 bits Source
CS_CTL 8 bits
Class Spec
TYPE 8 bits
Data structure
SEQ_ID 8 bits
DF_CTL 8 bits
Data field
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Port Addressing
D_ID and S_ID fields are 24 bits each
They provide the address or identifier of the
Source and destination port of a frame
Although the address map is flat, there are
several formats depending on:
Topology
Location
OPT-2T01
9899_06_2004_X
Port Address Identifiers
Applicable to all topologies
Point to point
Switched
Loop
Dynamically assigned or administratively assigned
Used for frame routing
Unique within Fibre Channel network
Assigned by the fabric
Some address reserved for special functions
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Port Address Identifiers
Acquired During Loop Initialization Arbitrated Loop
By Switch During Fabric Logon
Bound to Physical Port on Switch
Switched
By N_Port with Higher Worldwide
Name (MAC)
Point To Point
Assignment Topology
OPT-2T01
9899_06_2004_X
Address Identifiers
Arbitrated Loop Physical
Address (AL_PA)
00 00
AL_PA Area Domain
Device Area Switch Domain
8 bits 8 bits 8 bits
Switch Topology
Model
Private Loop (Not
Connected to a
Switch)
Public Loop
(Connected to
Switch)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Reserved Addresses
FC-PH has defined a block of addresses for
special functions:
High order 16 addresses in the 24 bit address space
Called the well known addresses
Main Address Used Today
FF FF FC Directory Server
FF FF FD Fabric Controller
FF FF FE Fabric F_Port which N_Port is
attached to
OPT-2T01
9899_06_2004_X
Data Structure Type
0 7 8
1
5
1
6
2
3
2
4
3
1
Word
0
1
2
3
4
5
S_ID 24 bits Source
CS_CTL 8 bits
Class Spec
TYPE 8 bits
Data structure
SEQ_ID 8 bits
DF_CTL 8 bits
Data field
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Type
The TYPE is a 8 byte field
Indicates the upper level carried in the payload
of the frame
Examples:
SCSI 08h
IP 05h
SNMP 24h
Fibre Channel services 20h
OPT-2T01
9899_06_2004_X
Frame Control
0 7 8
1
5
1
6
2
3
2
4
3
1
Word
0
1
2
3
4
5
S_ID 24 bits Source
CS_CTL 8 bits
Class Spec
TYPE 8 bits
Data structure
SEQ_ID 8 bits
DF_CTL 8 bits
Data field
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Frame_Control
The frame control is a 24 bit field
It contains a number of flags that are used to
control the flow of the sequence
The more common flags are exchange and
sequence management, acknowledgement
control and error conditions
Bits 16-23 deal with the sequence and exchange settings
Bits 14-15 deal with X_ID
Bits 13-12 form the ACK level for class 1 & 2
Bits 5-4 used for aborting the sequence
OPT-2T01
9899_06_2004_X
Frame Control Bits 12-13
Acknowledgment Capability
Provide assistance to Sequence Recipient (SR) by
translating the ACK capabilities bits in the N_Port
class parameters
Meaningful only in Class 1 and 2 data frames
0 0 = No ACK
0 1 = ACK level 1 one for every frame
1 0 = ACK level N N = number of frames
1 1 = ACK Level 0single ACK for complete
exchange, used in video streaming
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sequence Identifier
0 7 8
1
5
1
6
2
3
2
4
3
1
Word
0
1
2
3
4
5
S_ID 24 bits Source
CS_CTL 8 bits
Class Spec
TYPE 8 bits
Data structure
SEQ_ID 8 bits
DF_CTL 8 bits
Data field
OPT-2T01
9899_06_2004_X
Sequences
Sequences
Deal with chunks of upper level protocol
Are made up of one or more frames which
transport the ULP
The data phase may be subdivided into
multiple sequences
Uniquely identifiable with SEQ_ID
The command, data, and status phases of
SCSI are examples of sequences
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sequence Identifier
The Sequence Identifier (SEQ_ID) is a 8 bit field
All Frames of a sequence will carry the same
SEQ_ID value
Data content of these frames are related in some way
by the ULP
OPT-2T01
9899_06_2004_X
Sequence Count
0 7 8
1
5
1
6
2
3
2
4
3
1
Word
0
1
2
3
4
5
S_ID 24 bits Source
CS_CTL 8 bits
Class Spec
TYPE 8 bits
Data structure
SEQ_ID 8 bits
DF_CTL 8 bits
Data field
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sequence Count
Sequence count (SEQ_CNT) is a 16 bit field
Identifies the order of the transmission of frames
within this sequence
Used by Sequence Recipient (SR) to account for
all transmitted frames
Used by Sequence Initiator (SI) to account for all
transmitted acknowledges (ACKs) in Class 1 and 2
OPT-2T01
9899_06_2004_X
Sequence Count
Within a Sequence_Initiative
The SEQ_CNT of the first data frame will be zero
The SEQ_CNT of each subsequent data frame in the
sequence will be incremented by 1
The first data frame of the next sequence may be either
zero or one more then the last data frame, this is called
continuously increasing SEQ_CNT
If streamed sequences is used, continuously increasing
SEQ_CNT is required
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sequence Count
Sequence initiator
Assigns SEQ_CNT to data frames
Keeps a record of ACK frames received
Sequence recipient
Records SEQ_CNT of data frames
Transmits an ACK frame for each valid data frame
when Rx buffer is available
Knows that sequence was received without error if all
Frames are Rx without errors and are accounted for
Sequence initiator
Knows the sequence was received without error if it
has Rx an ACK frame to all frames within the sequence
OPT-2T01
9899_06_2004_X
Exchange Identifiers
0 7 8
1
5
1
6
2
3
2
4
3
1
Word
0
1
2
3
4
5
S_ID 24 bits Source
CS_CTL 8 bits
Class Spec
TYPE 8 bits
Data structure
SEQ_ID 8 bits
DF_CTL 8 bits
Data field
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
OX_ID and RX_ID
2 byte fields each
Contain the originator exchange identifier
and responder exchange identifier
They point to state and context information
regarding the exchange in the originator port
and responder port
OX_IDs are reused after each exchange is over
OPT-2T01
9899_06_2004_X
Parameter Field
0 7 8
1
5
1
6
2
3
2
4
3
1
Word
0
1
2
3
4
5
S_ID 24 bits Source
CS_CTL 8 bits
Class Spec
TYPE 8 bits
Data structure
SEQ_ID 8 bits
DF_CTL 8 bits
Data field
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Parameter Field
The parameter is a 4 byte field
The content of the parameter field is dependent
on the specific frame type as identified in the
routing field
FC-4 data frames
ACK link control
Port reject and frame reject frames
Port busy and fabric busy frames
LOGIN PARAMETERS
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Login
Exchange service parameters done with login
frame PLOGI or FLOGI
Required before communications can be
established between the two ports
Applies to all topologies
Applies to all ports, node and fabric
Bi-directional
ACCEPT Frame contains service parameters of the
port addressed
Procedure to Determine the Operating Environment
for Communications between Two Ports
OPT-2T01
9899_06_2004_X
Login
Version of Fibre Channel support
N_Port or F_Port functionality
Service classes supported
Size of receive buffers
Number of sequences supported
Support for Intermix
ACK capability
Error policy supported
Others
Service Parameters Contain the Following
Type of Information
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
ACKs
Informs Transmitter that:
One or more valid data frames were received by the
sequence recipient for the corresponding sequence
qualifier
Interface buffer is available for another data frame,
this only applies to class 1 and class 2
Class 3 are not ACKed
Flow control
Re-instates end-to-end credit
OPT-2T01
9899_06_2004_X
ACKs
Frame Header
Constructed from the data frame which is being
acknowledged
S_ID and D_ID are swapped
F_CTL with both exchange and sequence context
bit inverted
SEQ_ID is unchanged
SEQ_CNT is set to the sequence count of the highest
data frame being replied to by the ACK
Parameter Field
Bit 16 = History bit
Bits 0-15 are ACK type specific
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
ACKs
Again there are three types of ACKs
ACK_1 default for class 1 and 2 one ACK sent for each
SEQ_CNT
ACK_N Class 1 or 2 N=ACK sent by recipient for the
support indicated during port login
ACK_0 class 1 or 2 single ACK sent at end of sequence
We could spend a lot more time discussing ACKs but
there is little or no class 1 or 2 used in networks today
and doubt if we will see any soon
OPT-2T01
9899_06_2004_X
Busy and Reject
Port Reject P_RJT Fabric Reject F_RJT
Transmitted by destination port or fabric in
response to a specific data frame
Applicable to only Class 1 and 2
Sent in reply to valid frames
Transmitted by the receiver of the data frame
with reason code
Indicated that the corresponding data frame was
NOT delivered to the ULP
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Busy and Reject
Busy sent by fabric if unable to deliver frame due
to busy condition
Busy sent by port if temporarily busy and unable
to process a frame
If F_BSY or P_BSY is sent, fabric or port give
reason code
Class 1 busy only allowed on the connection request
Class 2 any frame may Rx busy
Class 3, busy is not sent; If a frame can not be delivered
it is discarded without notification
OPT-2T01
9899_06_2004_X
Flow Control and Credit
Flow model
Frames are moved:
From one Buffer
To another Buffer
Frame Flow is:
From the Source buffer and
To the destination buffer
Depending on the class of service
Multiple intermediate buffers may be involved
Applies to:
All topologies
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Frame flow is controlled by the receiver
Back-pressure mechanism
ACKs class 1 and 2, RDYs class 3
Flow control is based on frame flow
Which frames are flow controlled is dependent
on class of service
Receiver defined parameters during the
login procedure
Maximum frame size
Number of buffers
OPT-2T01
9899_06_2004_X
Receiver
Establishes operating environment through login
Size of buffers
Number of buffers (credits) allocated to this
transmitting port
Pumps-up these credits
By ACKs when buffer is available
A receive Buffer is available after
The frame was verified to be valid, no errors
And the frame has been moved off the interface buffer
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Transmitter
Keeps
Credit maximum value
Credit_Count
Consumes one credit for each frame it transmits
Credit_CNT = Credit_CNT 1 for each Data_Frame Tx
Regenerates credit for each ACK Rxed
Credit_CNT = Credit_CNT + N
Stops transmitting when
Credit_CNT = 0
OPT-2T01
9899_06_2004_X
FC-2 defines two type of credit
Buffer to Buffer (BB)
End-to-End (EE)
BB credit is the flow of connectionless traffic
Over a LINK from Tx to Rx
Class 2 and 3
Signal used = R_RDY
EE credit is the flow on connection traffic
Source to destination node
Class 1 and 2
Signal used = ACK
Both based on
Credit
Credit_CNT
Differ in
Frames controlled and acknowledgement signal
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sequence
Initiator
Sequence
Recipient
Fabric
ACK
R_RDY
R_RDY
R_RDY
R_RDY
ACK
EE_Credit
BB_C BB_C
TX
Buf
TX
Buf
TX
Buf
TX
Buf
RX Buf
RX Buf
RX Buf
RX Buf
OPT-2T01
9899_06_2004_X
Class of Service
Applicable to all fabric topologies
Switched
Point to point
Arbitrated loop
These three classes of service are
Class 1 dedicated connection
Class 2 connectionless multiplexed
Class 3 datagram
Delimiters used to set required class for a
sequence
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Class of Service
SOF delimiter
The required class of service along with basic sequence
management are specified in the SOF delimiter of
every frame
The SOF delimiter dedicate basic link management
functions within the fabric
The SOF delimiter identifies basic Sequence management
functions within the destination N_Port in the initial frame
of the sequence and the last frame of the sequence
EOF delimiter
Last frame of a sequence is terminated by a special EOF
Dedicated connections are removed by a special EOF
OPT-2T01
9899_06_2004_X
Class of Service
Class 1
Dedicated connection service
Connection oriented service between two N_Ports
Frames received in order transmitted
Guaranteed delivery with notification of non-delivery
Guaranteed throughput
Optional Intermix
Can mix Class 2 and 3 frames if allowed
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Class of Service
Class 1
Requires explicit connection establishment
SOF(C1) delimiter
Requires explicit removal of connection
ACK with EOF(DT) delimiter
Once connection is established
BSY and RJT will not occur
Flow control
Buffer to buffer on SOF(C1) frame: R_RDY
End to end for all other data frames: ACK
OPT-2T01
9899_06_2004_X
Class of Service: Class 1 Flow
Fabric
Initiator Recipient
SOF(C1)
R_RDY
ACK
SOF(n1)
SOF(n1)
SOF(n1) ACK
ACK
R_RDY
Connection Requested
Connection Established
EOF(t)
Conn Removed
ACK
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Class of Service
Class 2
Multiplexed connectionless service
Connectionless oriented service between two N_Ports
Order of frame reception not guaranteed
Guaranteed delivery
Notification of non-delivery
No throughput guarantees
Optional intermix
OPT-2T01
9899_06_2004_X
Class of Service
Class 2
Multiplex on a frame-by-frame basis
Between different destination N_Ports
Among different sequences
BSY and RJT may occur on any frame
Flow Control
Buffer-to-buffer for all frames: R_RDY
End-to-end for all data frames: ACK
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fabric
Initiator Recipient
SOF(C2)
R_RDY
ACK
SOF(n2)
SOF(n2)
ACK
ACK
R_RDY
R_RDY
R_RDY
R_RDY
R_RDY
R_RDY
R_RDY
R_RDY
OPT-2T01
9899_06_2004_X
Class of Service
Class 3
Datagram multiplexed connectionless service
Connectionless oriented service between two N_Ports
Order of frame reception not guaranteed
Unacknowledged
Delivery NOT guaranteed
No throughput guarantees
Optional intermix
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Class of Service
Class 3
Multiplex on a frame-by-frame basis
Between different destination N_Ports
Among different sequences
BSY and RJT will not occur on any frame
Flow control
Buffer-to-buffer for all data frames: R_RDY
OPT-2T01
9899_06_2004_X
Fabric
Recipient Initiator
Data Frame
Data Frame
R_RDY
R_RDY
R_RDY
R_RDY
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
EE Credit
NL_Node A
NL_Node B
EE_Credit
Switch
EE_Credit
EE_Credit
Applies Onl y to Class 1 and Class 2
Frames for All Topologies
OPT-2T01
9899_06_2004_X
For All Class 2 and Class 3 Frames
for All Topologies
BB Credit
BB_Credit
Switch
NL_Node A
NL_Node B
BB_Credit
BB_Credit
Presentation_ID.scr
FRAME PROCESSING
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Tables
The N_Port will keep the following information
Available X_ID table
Exchange context table
Login table
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Tables
Available X_ID Table
This table contains a list of available X_IDs
Can be used for OX_IDs or RX_IDs
A given implementation may choose to keep two tables
one for OX_ID and RX_ID
When a device driver sends a request to transmit
a frame, a value will be taken for the OX_ID
When a port receives a frame for a new exchange,
a value will be taken for the RX_ID
OPT-2T01
9899_06_2004_X
Tables
Exchange Context Table
Each exchange ID points to a unique entry in the
exchange context table
Each entry contains the context and state
information for the particular exchange
Port_ID involved in exchange
X_ID it assigned to exchange
ULP and phase within the operation
Data source or destination address
Data frames transmitted or received (SEQ_CNT)
ACK frames transmitted or received (SEQ_CNT)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Tables
Login Table
This table contains one entry for each port to
which this port is logged in with
Each entry contains service parameters and
working EE_Credit count value
OPT-2T01
9899_06_2004_X
Data Frames: Putting It All Together
Data Frame Transmission
Request for a ULP
Initiate some operation with a specific destination port
Login process
If you are not logged in, initiate login process
Build logging table entry for destination port
Assign OX_ID if needed
Get a value from the available X_ID Table
Build the exchange context table
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Data Frames
Data Frame Transmission (Cont.)
Gather information
Exchange context table
Receive buffer size and destination port
Login table
Working credit count of destination port
Set-up frame header
Data frame transmission
Segmentation process
Credit management
OPT-2T01
9899_06_2004_X
Data Frames
Transmit Request
ULP passes a request to transmit a chunk
of data to the N_Port
Destination Port D_ID is made
The N_Port must access the login table
to determine the service parameters on the
destination port
Number of Rx buffers
Value of the working credit count
And the rest
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Data Frames
The Data Transmission
ULP data chunk is moved in frames with the
use of the sequence
All within the context of the exchange
A number of processes are involved
Initialization of the frame header fields
Segmentation and reassemble
OPT-2T01
9899_06_2004_X
First and Last Data Frames
The first data frame of a sequence is identified by
SOF(Ix) Delimiter, where x is the Class of Service
The last data frame of a sequence is identified by
F_CTL bit 19, End_SEQ=1
A sequence consists of all data frames
Starting at the SEQ_CNT for the first frame through
the SEQ_CNT of the last frame
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sequence Processing
Sequence Count
ULP chunk of data is transmitted IN ORDER
All Frames are sent in order
Sequence_Count (SEQ_CNT)
Frames are assigned sequentially increasing numbers
as they are sent
The receiving N_Port will use the SEQ_CNT to insure
that Frames are reassembled in order and back in its
original chunk
OPT-2T01
9899_06_2004_X
Sets F_CTL bit 23
0 If it is the exchange originator
1 If it is the exchange responder
OX_ID and RX_ID set to assigned values
RX_ID = FFFF if first sequence of exchange
Routing field (R_CTL) set to 0000 to indicate
FC-4 data frame
Information category field of R_CTL set according
to payload
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sequence Initiator: Frame Header
Sequence ID (SEQ_ID)
Any value select that is not used
Sequence count (SEQ_CNT)
Assign sequentially as frames are sent
Starts with 0 on first frame of sequence
Increments by 1 while sequence initiative is held
Parameter
Set to offset of the first byte of payload with respects
to entire chunk
Offset = 0 on first frame and 1 + for second and
subsequent frames
OPT-2T01
9899_06_2004_X
Sequence Initiator: Frame Header
Other important F_CTL bits
Bit 23, exchange context
Bit 21, first sequence
Bit 20, last sequence
Bit 19, end sequence
Bit 16, sequence initiative
Used to pass initiative to other device
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Automatic Processes
These processes are automatic and are
performed by the protocol chip
Segmentation and reassembly
SEQ_CNT assignment
Higher layers are unaware of these processes
OPT-2T01
9899_06_2004_X
ULP Processing
The Upper Level Protocol (ULP) uses these fields
Routing 0000 = FC-4 data frame
Type 08 = SCSI/FCP
Info category
Identifies Specific Function of Payload
01 = Solicited Data
06 = Unsolicited Command
05 = Data Descriptor
07 = Command Status
Presentation_ID.scr
ARBITRATED LOOP
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Fibre Channel Arbitrated Loop (FC-AL)
Maximum bandwidth: 100
MB/sec. (shared amongst all
nodes on loop)
126 nodes max on loop
Can be combined with
switches
Attaches NL_Ports
Number of nodes on loop
directly affects performance
Defined in its own standard
Fibre Channel Hub
FC
L
L
L
L
L
L
L
L
L
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Loop Advantages
Low cost solution with copper transceivers
Eliminates the need for a discrete fabric
Fabric routing decision distributed around the loop
Compatible with all FC- 0 variants
Copper within a box
Optical between boxes
Self discovery procedure
Simple additions to FC-PH
OPT-2T01
9899_06_2004_X
Loop Advantages
Port bypass network
High availability configurations possible
Supports both public and private loops
Provides access fairness
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
NL_Port
N_Port
Attaches to the physical transport media
Provides the Fibre Channel control and protocol
Provides the termination point for Fibre Channel
Resides within the node
NL_Port
Provides all functionality on N_Port with additional
function of the loop
An NL_Port can function as a N_Port
OPT-2T01
9899_06_2004_X
FL_Port
F_Port
Attaches to the physical transport media at the edge of
the switched fabric
FL_Port
The switched fabric port which attaches to a loop
F_Port functionality with additional function of the loop
G and GL Ports Will Do Both N and F
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Private and Public
Private Loop
Contains no FL_Port
Communications outside the loop via Fibre Channel
is not possible
Public Loop
Contains an FL_Port
Communications outside loop via Fibre Channel
is possible
Private Devices
Devices on a public loop may be private, i.e. do not login
OPT-2T01
9899_06_2004_X
Addressing
Arbitrated Loop Physical Address (AL-PA or PA)
Assigned during the loop initialization (soft addressing)
A unique 8 bit value
127 valid values
Arbitrated Loop Destination Address (AL-PD or PD)
The AL_PA used to identify the destination L_Port
Target of a primitive signal or D_ID of a frame
Arbitrated Loop Source Address (AL_PS or PS)
The AL_PA used to identify the source L_Port
Source of a primitive signal or S_ID of a frame
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
The Fabric Definition
The entity that interconnects attached N_Ports
Provides routing based on destination address
Fabric may be:
Point to pointNo routing required
SwitchedRouting provided by the Switch
Arbitrated loopRouting is distributed throughout
the attached NL_Ports
OPT-2T01
9899_06_2004_X
Switched Fabric
N_Port N_Port N_Port N_Port N_Port N_Port
Fabric
Switch
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Loop
NL_Port
Node
NL_Port
Node
NL_Port
Node
Node
NL_Port
Node
NL_Port
Node
NL_Port
Fabric
LOOP
= Arbitrated Loop
Additional Function
OPT-2T01
9899_06_2004_X
Routing Process: Loop
The routing function is distributed
Each L_Port performs a portion of routing
Routing is performed through out-of-band
signaling using primitive signals
Connection oriented independent of class
of service
Obtain ownership of the loop (Arbitration)
Establish a connection (Open)
Transfer frames (Data)
Remove the connection (Close)
Relinquish the loop
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Processes and Procedures
Initialization
The process by which addresses are assigned
and recovery is performed
Arbitration
The process by which an L_Port acquires ownership
of the loop
Open
The process by which the L_Port which owns the Loop
uses to select the L_Port to which it wants to
communicate with
Close
The process by which the L_Port which owns the Loop
releases control
OPT-2T01
9899_06_2004_X
Fill Words
FC-PH defines two signals that may be transmitted
between frames (when no other information is
being transmitted)
Idle
R_RDY
FC-AL defines several additional signals that may
be transmitted between frames
FC-AL defines the fill word to be
ARB(F0)
ARB(x)
Idle
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Primitive Signals and Sequences
Primitive signals
Arbitrate
Open
Close
Mark
Primitive sequences
Port bypass enable
Port bypass disable
Loop initialization
FC-AL Defined the Following Unique
Signals and Sequences
OPT-2T01
9899_06_2004_X
Credits Buffers
Alternate BB_Credit management requested
during login
When activated service parameter BB_Credit = number
of buffers available when circuit is established
The receiving L_Port shall transmit R_RDYs for the
additional buffers at anytime when opened
Used to pump up BB_Credit_CNT
Transmitting L_Port
Decrements BB_Credit by 1 for each data frame Tx
Increments BB_Credit by 1 for each R_RDY Rx
Stops transmitting when BB_Credit =0
Loop Uses Same Credit Method as Previously
Discussed But Also Has an Alternate Credit Model
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Arbitrated Loop Initialization Procedure
Purpose
An L_Port will perform the loop initialization
procedure to:
Determine the Operating environment for the L_Port;
Is this a loop?
Acquire an address. AL_PA (Physical Address)
Report that an error has been detected
OPT-2T01
9899_06_2004_X
Loop Commands
Command & AL_PA
Distribute Position map
132
Link Initialization Loop Position
LILP
Command & AL_PA
Collect Position map
132
Link Initialization Report Position
LIRP
Command & AL_PA bit
map
20
Link Initialization Soft Assigned
LISA
Command & AL_PA bit
map
20
Link Initialization Hard Assigned
LIHA
Command & AL_PA bit
map
20
Link Initialization Previousl y Assigned
LIPA
Command & AL_PA bit
map
20
Link Initialization Fabric Assigned
LIFA
Command & WWN
12
Link Initialization Select Master
LISM
Payload Contents Bytes Command
Loop Initialization ProcedureLIP Is an Ordered Set
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
LIP: Initialization Procedure
Phase A
Phase B
Phase C
Phase D
Phase E
Close
Start The
Initialization
Procedure
Select
Temporary Loop
Master
AL_PA
Mapping
Phase
Reporting Phase
Distribute AL_PA
Map Phase
LIP
LISM
LIFA,
LIPA,
LIHA,
LISA
LIRP
LILP
Start The
Initialization
Procedure
FL_Port Wins if Present
Otherwise Lowest WWN Wins
Build the AL_PA bit
Map in 4 Steps
Collect the AL_PA
Position Map
Distribute the AL_PA
Position Map
OPT-2T01
9899_06_2004_X
LIP: Phase A
Loop Initialization Primitive Sequence
Transmitted continuously by L_Port until it
receives the same LIP configuration
LIP (F7F7) the L_Port is attempting to determine if this
is a loop and to acquire an AL_PA
LIP (F8F7) the L_Port has detected a loop failure at its
receiver prior to acquiring an AL_PA
LIP (F8) the L_Port (AL_PS) had detected a loop failure
at its receiver
LIP (F7) the L_Port (AL_PS) has detected a performance
degradation
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
LIP: Phase B
Each L_Port will build the LISM with:
AL_PA = 00 hex if FL_Port
EF hex if NL_Port
D_ID = 0000hex + AL_PA Example (0000EF)
S_ID = 0000hex + AL_PA
Payload = Command + WWN
Current Fill Word = Idle
Each L_Port will continuously transmit a LISM
Normal flow control rules are not in effect during
initialization
OPT-2T01
9899_06_2004_X
LIP: Phase B (Cont.)
Each L_Port monitors its receiver
Will continue to transmit LISM if Your AL_PA + WWN is
less then received AL_PA + WWN
Otherwise pass the received LISM
You are temporary loop master
If the device receives a LISM identical to the
one transmitted
FL_Ports always win; If two or more FL _Ports;
Lowest WWN wins and the others go non-participating
If no FL_Port the NL_Port with lowest WWN wins
Loop master
Current fill word would be ARB(F0)
When ARB(F0)s are received, go to phase C
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
LIP: Phase C
Loop Master Will Form the Initial Bit Map as
Shown:
0000 0000 0000 0000 0000 0000 0000 0000 3
0000 0000 0000 0000 0000 0000 0000 0000 2
0000 0000 0000 0000 0000 0000 0000 0000 1
0000 0000 0000 0000 0000 0000 0000 L000 0
0
7
8
15
16
23
24
31 Word
Where L = 1 Requesting F_Login of all NL_Ports
Bit Positi on = 127 vector corresponding to valid AL_PAs
Word 0 bit 30 = lowest number 00hex
Word 3 bit 0 = hi gh number AL_PA value EFhex
Set the bit = 1 that corresponds to its Fabric Assigned AL_PA
Bit
Position
Lowest
AL_PA
Highest
AL_PA
OPT-2T01
9899_06_2004_X
LIP: Phase C
Loop master will transmit the following three
commands allowing an L_Port to choose a
desired AL_PA
LIFA bit map primed with initial value
LIPA bit map primed with results of LIFA
LIHA bit map primed with results of LIPA
Loop master will then transmit the LISA command
LISA bit map primed with results of LIFA allowing L_Ports
which were unable to obtain their desired AL_PA to get a
soft assigned AL_PA
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
LIP: Phase C
Each NL_Port will
Receive, possibly modify and retransmit the four
Initialization Command frames
Set the Current Fill Word (CFW) = ARB(F0)
Modify the AL_PA bit map as follows
Set one bit of the initialization command AL_PA bit maps
based on history of AL_PA assignment
If the bit map corresponding to a desired AL_PA has
been set by an up-stream L_Port, this L_Port assumes a
soft AL_PA by setting the first 0 bit=1 in the bit map of
the LISA frame
If no bit positions were available in the LISA bit map, the
L_Port will remain in non-participating mode
At most the bit map of one command will be modified
by each L_port
OPT-2T01
9899_06_2004_X
LIP: Phase D
The loop master will prime the AL_PA position
map to:
Byte 0 = 01 hex
Byte 1 = its AL_PA
Bytes 2-127 = FF hex
Then transmit the LIRP with this position map
Each NL_Port will:
Increment the offset by one and store the offset
Store its AL_PA at the offset
Retransmit the updated LIRP frame
The loop master will save the resulting loop
position map
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
LIP: Phase E
The loop master will transmit the LILP
command with
Payload = AL_PA position map
Each NL_port will
Save the loop position map
Retransmit the LILP command
When the loop master receives the LIILP
command it will
Transmit a CLS and go to monitoring state
When each NL_Port receives a CLS they will
Retransmit the CLS and go to monitoring state
Initialization Complete
OPT-2T01
9899_06_2004_X
LIP: Summary
A. LIP starts the initialization procedure
B. Select a temporary loop master
Lowest AL_PA | WWN wins
C. Build a AL_PA bit map
Each L_Port indicates the AL_PA it selected in one
of 4 requests by the loop master
D. Collect a AL_PA position map
Each L_Port reports its relative position from master
and its AL_PA
E. Distribute the resulting AL_PA position map
to each L_Port
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Arbitration
The process by which L_port request ownership of
the loop based on primitive signals
AL_PA AL_PA D20.4 K28.5
MSB LSB
ARB(x)
Ordered Set
OPT-2T01
9899_06_2004_X
Arbitration
Loop Owner
The current loop owner is responsible for
Seeds the arbitration process with ARB(F0)
Blocks propagation of the received ARB(x) until it
relinquishes the loop
Initiates a new arbitration window
If ARB(F0) is received by setting current fill word = IDLE
Fairness variables
Access
ARB_WON
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Arbitration Process
When a port is arbitrating it enters the
arbitrating state
The CFW is updated to the ports ARB(AL_PA)
if the CFW is:
1. IDLE
2. ARB(F0)
3. ARB(FF)
4. Lower-priority ARB (higher value AL_PA)
Arbitration occurs even if a loop circuit exists
between another pair of ports
Once a port starts arbitrating it
Must continue to arbitrate until it wins
Withdraw if it knows that another port is arbitrating
OPT-2T01
9899_06_2004_X
Fairness
Access Fairness
Ports with higher-Priority AL_PA values could lock out lower
priority ports
When they ARB they will always win
Lower Priority ports might never win Arbitration
Access fairness limits how often a port can arbitrate
This is done by not arbitrating the loop until all other ports on
the loop that are arbitrating have won; This is called a fair port
Access fairness is based on access not duration
of usage
Does not limit how long a port uses the loop
Fairness is recommended by the standard but not
mandatory
FL_Ports may be unfair but NL_Ports should be fair
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fairness
The fairness is controlled by the FC-AL fairness
algorithm called a fairness window
Window begins when the first port wins arbitration
Ends when a port discovers that it was the last arbitrating port
IDLE resets the fairness window
The variables used are
Access = 0 for fairness window open
Access = 1 when NL_Port has won arbitration
Fair ports can only arbitrate once per window
After winning arbitration they wait for the end of the window
before arbitrating again
Unfair ports can arbitrate at anytime
OPT-2T01
9899_06_2004_X
Open
If the Port Requires the Loop when It Wins ARB
It sends an OPN(yx) or OPN(yy)
y=destination port x=source port
Full-Duplex establishes a point to point like circuit
between the loop ports
Half-duplex restricts open recipient to transmit link
control frames only
Cannot transmit device data frames
Used by designs that can not support simultaneous
data frames Tx and Rx
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Open
Selecting the Destination Port
Is the intended destination port on same loop or
connected via fabric switch?
If the upper 16 bits of destination field (D_ID) are all zeros
the port is on this private loop
If the upper 16 bits of the source(S_ID)are all zeros then the
source port is a private port and can only talk to ports on
same loop
If the upper 16 bits of the D_ID are the same as the upper 16
bits of the S_ID then they are both on the same loop or both
are public and attached to the same FL_Port
If none of these are true, the destination port is not
on the same loop and must be accessed via FL_Port
OPT-2T01
9899_06_2004_X
Opening a Port on Same Loop
Open Originator inserts the destination AL_PD
in the OPN
The AL_PD is obtained from the low-order 8 bits
of the destination address in the frame header
This process can be entirely by hardware
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Opening a Port Off the Loop
Originator inserts AL_PD of the FL_Port 00
in the AL_PD field of the OPN
The FL_Port is opened and frames are sent
to the FL_Port
FL_Port and fabric forwards the frames using
the destination address field
FL_Port can send to multiple destination
ports on the loop during this OPN
SWITCH FABRIC OPERATION
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Switch Model
Port Port
Port Port
Connection
Matrix
Fabric Controller
Connectionless
Switch Matrix
OPT-2T01
9899_06_2004_X
Worldwide Names
Each switch element is assigned a WWN at time
of manufacture
Each switch port is assigned a WWN at the
time of manufacture
During FLOGI the switch identifies the WWN
in the service parameters of the accept frame
Fabric port and
Switch element
These address assignments can then correlate
each fabric port with the switch element
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Switch Ports
Four basic types of switch ports
F_PortUses NOS/LOS to attach to single N_Port
FL_PortUses LIP to attach 1 to 126 NL_Ports
E_PortUses NOS/LOS to interconnect switches
(inter-link switch ISL)
G_PortUses NOS/LOS can be a F or E port
OPT-2T01
9899_06_2004_X
Fabric Addressing
The 24 bit address is partitioned into 3 fields
Device
Area
Domain
This partitioning helps speed up routing
Switch element assigns the address to N_Ports
Address portioning is transparent to N_Ports
Device Area Switch Domain
8 bits 8 bits 8 bits
Switch Topology
Model
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Directory Server
Repository of information regarding the
components that make up the Fibre
Channel network
Located at address FF FF FC (Some readings
call this the name server)
Components can register their characteristics
with the directory server
An N_Port can query the directory server for
specific information
Query can be the address identifier, WWN and volume
names for all SCSI targets
OPT-2T01
9899_06_2004_X
Directory Server
Command Requests
Get objects
GA_NXTGet all next
GFT_IDGet FC-4 types
Register objects
RFT_IDRegister FC-4 types
Deregister objects
DA_IDDeregister all
These Are Some of the More Used Commands
Used to Query the Directory Server
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fabric Controller
Each switch has a fabric controller
Assigned address FF FF FD
Every fabric controller in the fabric has the same address
It is the N_Port within the switch
Responsible for managing fabric, initialization, routing,
setup and teardown of Class-1 connections
Responsible to receive request and generate
responses for the switch fabric
Information must be consistent independent
of which fabric controller responds to a request
OPT-2T01
9899_06_2004_X
Extended Link Services
Extended link services provide a set of protocol
functions used by the port to specify a function
or service at another port
Usually sent from N_Port to F_port to perform
needed request
The R_CTL field of the first word will be set to 0x22 to
indicate an extend link service request
Many ELS services will return a payload in response
some have no reply
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Extended Link Services
Some of the more important and most used ELS
commands are:
FLOGI F_Port Login
PLOGI N_Port Login
FAN Fabric Address Notification
PRLI Process Login
PRLO Process Logout
SCN State Change Notification
SCR State Change Registration
RSCN Registered State Change Notification
OPT-2T01
9899_06_2004_X
ELS: FLOGI
FLOGIFabric login
Issued by N_Port to destination FF FF FE to
Determine if fabric is present
Establish a session with the fabric
Exchange service parameters with the fabric
FLOGI assigns N_Ports 24 bit address to
N_Port or AL_PA to loop ports
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
ELS: PLOGI
PLOGIN_Port login
Established sessions between two N-Ports
Required before upper level protocol operations can begin
N_Port will register to the name server FF FF FC in fabric
with all required login parameters
N_Port will then query name server for other N_Ports
on the fabric
OPT-2T01
9899_06_2004_X
ELS: PRLI
PRLIProcess Login
Allows the FC-4 levels to exchange service parameters
for communications between each other
Process is protocol specific (type field)
SCSI-3 FCP mapping requires PRLI
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
ELS: FAN
FANFabric Address Notification
Used in fabric loop attached topology
Provides mechanism for FL_Port to notify
NL_Ports of addresses and names of FL_Ports
along with fabric name
Allows NL_Ports to verify configuration following
a loop initialization
OPT-2T01
9899_06_2004_X
ELS: SCN
SCNState Change Notification
Provides notification to ports of events that may
effect logins or process logins to ports on the fabric
SCN can be sent from
N_Port to N_Port
N_Port to fabric controller
Fabric controller to N_Ports
Notification may indicate login session is no
longer valid
Loss of signal (NOS, LOS, FLOGI)
LIP has occurred
SCN sent to fabric controller
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
ELS: RSCN
RSCNRegistered State Change Notification
Similar to SCN but only sends change notice
to those ports registered
SCN did not define a registration method
OPT-2T01
9899_06_2004_X
Class_F Service
Communications between switch elements use
Class_F Service
Unique SOF delimiter and normal EOF delimiter
Used to pass control information within the switch
Highest priority within switch
Connectionless service
Has no meaning outside switch, N_Port will
discard if received
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Inter-Switch Link
The interconnection between switches is called
the inter-switch link
E_Port to E_Port
Supports all classes of service
Class 1, 2, 3, and switch to switch control traffic, class F
FC-PH permits consecutive frames of a sequence
to be routed over different ISL links for maximum
throughput
OPT-2T01
9899_06_2004_X
Interswitch Links (ISLs)
Inter-switch link (ISL) connects switches
Fabric parameters must match on both switch
otherwise link would not come up and fabric
will be segmented
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Principal Switch Selection
Only one switch is
designated principal
switch in a fabric
The switch with the
lowest WWN becomes the
principal switch originally
Principal switch makes
sure that no new switch is
added to the fabric if it
has a domain ID conflict
with an existing switch in
the fabric
Switch 1
Switch 3
Switch 4 Switch 5
Switch 6
Switch 2
OPT-2T01
9899_06_2004_X
Fabric Configuration Process
The fabric configuration process enables a switch
port to determine its operating mode, exchange
operating parameters, and provides for distribution
of addresses
The process is summarized in the following steps
Establish link parameters and switch port operating mode
Principal switch selection
Domain ID distribution
Path selection
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fabric Configuration Stages
Switch is operations with
routes established
Path selection (FSPF) is
defined in the next section
Switch has a
Domain_ID
Path Selection
Switch has a Domain_ID Switch requests a Domain_ID
from the Domain Address
Manager
Domain Address
Manager had been
selected
Domain ID
Acquisition
The Principle Switch is
selected
Switch_Names are
exchanged over all ISLs to
select a Principle Switch,
which becomes the Domain
Address Manager
BF or RCF SW_ILS
transmitted or
received
Select
Principle
Switch
Switch Port mode is
known. If a Port is an E
port, link parameters have
been exchanged & Credit
has been initialized.
The Switch Port attempts to
discover whether it is an FL,
F, or E port.
Switch Port has
achieved word
synchronization
Establish Link
Parameters
and Switch
Port Operating
Mode
Ending
Condition
Process Starting
Condition
Operation
OPT-2T01
9899_06_2004_X
Fabric Configuration: PS Selection
A principal switch shall be selected whenever at
least one inter-switch link (A link between two
E_Port) is established
The selection process chooses a principal switch,
which is then designated to assign domain
identifier to all the switches in the fabric, and any
who join later the fabric later on
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fabric Configuration: PS Selection
The principal switch selection can be triggered by
anyone of the following events
Switch boot and EFP
Build Fabric (BF)
Reconfigure Fabric (RCF)
OPT-2T01
9899_06_2004_X
Fabric Build Process
When the switch first boots up and the first E_Port of a switch becomes
operational, the switch starts 2xF_S_TOV timer and then sends out an
exchange fabric parameters (EFP) from that port containing its own
Destination ID (DoID) in the list trying to become Principle Switch (PS)
The switch receiving the Exchange Fabric Parameter (EFP) replies with either
ACCept or ReJecT after comparing the priority and WWN
0x11
Record Len
(0x10)
Payload Len
Reserved Priority
Principal Switch WWN (Word 0)
Principal Switch WWN (Word 1)
Domain_ID record 0
Domain_ID record M
A (0) (FF, Aa)
B (0) (FF, Bb)
C (0) (FF, Cc)
E
_
P
o
r
t
E
_
P
o
r
t
Domain_id
Priority
WWN
E
F
P
E
F
P
E
_
P
o
r
t
E
F
P
E
_
P
o
r
t
E
F
P
EFP
D (0) (FF, Dd)
EFP EFP EFP
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fabric Build Process
If the recei ved information has a lower value, the switch keeps the recei ved
information and considers sending switch as potential principal switch and
also consider that link to be potential upstream link
At that point switch generates another EFP for all other links with the updated
potential principal switch
When 2x F_S_TOV expired, all switches in the fabric consider the information
collected for the principal switch to be definitive; At that point the principal
switch is responsible for assigning the Domain_IDs
A (0) (FF, Aa)
B (0) (128, Aa)
C (0) (FF, Aa)
E
_
P
o
r
t
E
_
P
o
r
t
Domain_id
Priority
WWN
S
W
_
R
J
T
S
W
_
R
J
T
E
_
P
o
r
t
S
W
_
A
C
C
Potential Upstream port
S
W
_
A
C
C
E
_
P
o
r
t
EFP EFP EFP
Potential Upstream port
D (0) (FF, Bb)
OPT-2T01
9899_06_2004_X
Fabric Configuration Details
After the principal switch selection, the PS will change its
priority to 0x02 (PS Priority) and then assign itself a domain
ID and then the process of domain distribution starts
The principal switch will initialize the process by sending
Domain ID Assigned (DIA) SW_REQ out of all its E_Port
The intermediate switch is actively involved in this process
Each switch will reply back with Request Domain ID (RDI)
To allow each switch to request for one or more domain ID
The neighboring switch receiving RDI will be able to identify
its downstream principal ISL
Each switch can send many RDI but once the principal switch
has granted the domain ID to the switch, the following RDI
from the switch must request the same set of domain_id
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
A (1) (XX, Aa)
B (2) (FF, Aa)
C (4) (FF, Aa)
E
_
P
o
r
t
E
_
P
o
r
t
D
I
A
D
I
A
E
_
P
o
r
t
Upstream port
E
_
P
o
r
t
Upstream port
D (3) (FF, Aa)
A B D
DIA (SW_REQ)
SW_ACC
SW_RJT
RDI (SW_REQ)
SW_ACC
SW_RJT
DIA (SW_REQ)
EFP (SW_REQ)
Contains DoID list
SW_ACC
SW_RJT
RDI (SW_REQ)
RDI (SW_REQ)
SW_ACC
SW_RJT
SW_ACC
SW_RJT
EFP (SW_REQ)
Contains DoID list
EFP (SW_REQ)
Contains DoID list
Fabric Configuration Flows ID Assignment
SW_ACC
SW_ACC
SW_ACC
B
D
A
OPT-2T01
9899_06_2004_X
Fabric Configuration: The PS Battle
After the principal switch selection and domain id
assignment, all switches in the fabric will start two processes
FC_ID assignment
FSPF path selection
When a new switch is added to the fabric, it will send out an
EFP with its local value (I am PS); the fabric rejects that EFP
and replies with DIA telling the new switch to send RDI; the
RDI is then routed to the current PS
If the new switch is part of another fabric (it also has a PS)
then both fabric sends out an EFP and after comparing the
DoID list the fabric enters one of the following states
BF state: If the DoID list does not overlap
RCF State: If the DoID list overlap
Isolation: No auto-reconfigure state or RCF disabled
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fabric Configuration
Disruptive/Non-Disruptive
One of the following three conditions can trigger BF
(non-disruptive) or RCF (disruptive)
Two disjoints fabric are combined together
A principal ISL fails (upstream or downstream)
A switch with Domain_ID request for another Domain_ID
Whenever a switch receives a BF/RCF, the switch starts
F_S_TOV timer and enters the BF/RCF state; It forwards
BF/RCF out of all E_ports except the incoming port
(onl y once) and wait for the timer to expire
When the timer expires, BF/RCF propagation state
is left and principal switch selection begins
BF is not a disruptive process
RCF is a disruptive process
OPT-2T01
9899_06_2004_X
Propagation of BF or RCF Requests
Fabric Configuration Distribution
Switch Starts
the
Reconfig
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fabric Configuration: Reserve IDs
N_ports and E_ports get one port ID; F ports dont get any IDs; FL
ports in public AL gets 0x00 port ID
Fabric F_Port, Fabric Login database FE FF FF
N_Port of fabric controller FD FF FF
Well Known Address F0-FC,FF FF FF
Reserved 00-EF FF FF
Reserved 00-FF FD-FE FF
Reserved F0-FF FC FF
N_Port of domain controll er. Port ID is the domain ID 01-EF FC FF
Reserved 00 FC FF
Multicast & Broadcast 00-FF FB FF
Reserved 00-FF 00-FA FF
Reserved 00-FF 00-FF F0-FE
N_Port & E_Port. Port ID=00 for FL port for public devices 255 address 00-FF 00-FF 01-EF
Reserved 00-FF 01-FF 00
Reserved NonAL_PA 00 00
Private Loop NL_Port AL_PA 00 00
Used during FLOGI 00 00 00
Description Port_ID Area_ID Domain_ID
OPT-2T01
9899_06_2004_X
Fabric Configuration: FSPF
FSPF stands for fabric shortest path first
Based on link state protocol
Begins after domain ID assignment is completed
Conceptually based on open shortest path first
(OSPF) internet routing protocol
Currently a standard defined in FC-SW-2
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FSPF has four major components
Hello protocol
Replicated topology database
A path computation algorithm
Routing table update
FSPF discovers the paths to switches using
DomainIds
Each switch performs its own shortest path
calculations
OPT-2T01
9899_06_2004_X
For FSPF a domain ID identifies a single switch
This limits the max number of switches that can support
in the Fabric to 239 when FSPF is supported
FSPF performs hop-by-hop routing
FSPF supports hierarchical path selection
Provides the scalable routing tables in large topologies
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Everyone says HELLO to their neighbor, on all
initialized ISLs
The neighbors say HELLO back, unless they
are dead
When the HELLO packet is received with both
originator and recipient domain id, the two way
communication is done and:
The ISL is active
The ISL may be available as a two-way path for frames
OPT-2T01
9899_06_2004_X
Fabric Configuration: Hellos
Point to Point Only
Default Hello Int = 20 S
Default HelloDead Int = 80 S
Hello Protocol
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fabric Configuration Link
State Update and Ack
After a 2-way HELLO is
established on a Link,
each switch exchanges
its entire database with
its neighbor using the
LSU service
When the recipient of the
LSU has processed the
database, it sends back
the LSA service
B A
LSU(DB-A)
LSU(DB-B)
LSA(DB-B)
LSA(DB-A)
OPT-2T01
9899_06_2004_X
Link State Record
When the databases are
in sync, each switch
sends its LSR with the
new link included using
the LSU service
The LSU is flooded to the
entire fabric
Each Switch retransmits
the LSU by a mechanism
called reliable flooding
B A
LSU(LSR-A)
LSU(LSR-B)
LSA(LSR-B)
LSA(LSR-A)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Link cost is calculated based on baud rate of the
link, plus an administratively set factor
Link cost = S * (1.0625E12/baud rate)
S is administrative factor defaults to 1
Ex: Link cost of 1G port = 1000
Path cost is the sum of the traversed link costs
Lower metric more desirable
OPT-2T01
9899_06_2004_X
FSPF routes are
full y functional
Routes are calculated and set up
within each switch. Links are
maintained by sending HELLOs
every Hello_Interval. Link databases
are maintained by flooding link
updates as appropriate
Initial Database
Exchange has
been completed
Running State
Link State
Databases have
been exchanged
LSU SW_ILS frames are exchanged
containing the initial database
Two
communication
has been
established
Perform Initial
Database
Exchange
Two way
communication
has been
established
HLO SW_ILS frames are exchanged
on the link until each switch has
received a HELLO with a valid
neighbor Domain field
The switch
sending HELLO
has a valid
Domain_ID
Perform initial
HELLO
Exchange
Ending
Condition
Process Starting
Condition
Operation
Path Selection (FSPF) Operation Summary
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FSPF Characteristics
Uses FSPF as the routing algorithm
FSPF routes traffic based on destination domain ID
FSPF uses total cost as the metric to determine
most efficient path
Static routes can be applied
OPT-2T01
9899_06_2004_X
FSPF Characteristics
Paths:
Finds the shortest path to each domain, then programs the
hardware routing tables
Routes:
Dynamically
Round robin
Statically
Administrator can configure the route
Automatically re-routes upon ISL going away and static routing
will again take effect upon ISL return
Automatic failover
Fault detection 150 ms
Self heals in 500 ms
So, alternate route is live in 650 ms
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Routing Software Configurable Parameters
Link cost
Static routes
In Order Delivery (IOD)
Timers (be careful)
OPT-2T01
9899_06_2004_X
What Is a Route and Path?
A route is map between the input and output
E_port used to reach the next switch
A path is a map through the topology between a
source and destination
FC FC
FC FC FC
Route
Path ISL
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Selecting a Path
Each inter switch link has a cost metric
The cost of an ISL is related to the bandwidth
The total cost of a path between two switches is the sum
of the cost of all the traversed ISLs
The path to a destination switch is the one with the
minimum total cost
More than one path can be selected (with the same cost)
Cost 500
Cost 250
Cost 250
FC FC FC
FC FC
OPT-2T01
9899_06_2004_X
ISL Oversubscription
Oversubscription
occurs when more
nodes can contend
for the use of one ISL
Oversubscription
ratio is the number of
different ports that
contend for the use
of one ISL
This a 3:1 over
subscription
Multiple Nodes
Switch
Switch
ISL
1G
1G
1G
1G
Presentation_ID.scr
FC ERROR MANAGEMENT
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Timers
Four different timers used
Receiver-transmitter time-out (R_T_TOV)
Error detect time-out (E_D_TOV)
Resource allocation time-out (R_A_TOV)
Connection request time-out (C_R_TOV)
Used in Class 1 You will never see class one
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Timers: R_T_TOV
Receiver-Transmitter Time-out
Used to time events at the link level
Loss of synchronization
Times Responses for link reset protocol
Generally controlled in hardware for all link
configurations
Default value in FC Standard is 100ms
OPT-2T01
9899_06_2004_X
Timers: E_D_TOV
Error Detect Time-out
Timers for events and responses at the
sequence level
Missing ACK or R_RDY when buffer credit has
reached zero
Class 1 or 2 expects response from data frames
N_Port logout
Timer value is set at fabric login to accommodate
the network environment for better scaling
according to delivery time of frames
Default is 10 sec
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Timers R_A_TOV
Resource Allocation Time-out
Time-out value for how long to hold resources
associated with a failed operation
Needed to free shared resources for reuse
Value to determine how long a port needs to keep
responding to a link service request before an error
is detected
R_A_TOV is 2 times E_D_TOV
Default setting in Pt to Pt is 20 sec and fabric is 120
seconds
OPT-2T01
9899_06_2004_X
Timers: CR_TOV
Connection Request Time-out
Determines how long the fabric can hold
a class-1 request in the queue during
connection establishment
Allows for separation of the time in a stacked
queue from the E_D_TOV; This separates
queuing time from frame transit time
Helps in controlling F_BSY issues
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Recovery: Class 3
Errors in class 3 sequence can only be detected by
the Sequence recipient because there are no ACKs
or rejects in class 3
Class 3 SR will discard single or multiple frames
until the exchange is terminated
The upper level recovery may retransmit the entire
Sequence or at least the sequence following the
error detection
OPT-2T01
9899_06_2004_X
Recovery: Class 3
Errors a class 3 operation can detect:
Out of order delivery and potential missing frame based
on SEQ_CNT
If missing frame is not Rxed within E_D_TOV
Indication of a new sequence when last frame of previous
Sequence has not been received (in-order delivery set)
Relative offset not in order with an order delivery set
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Abort Sequence: ABTS
ABTS protocol
Used to terminate sequence or exchange
Transmitted by the sequence initiator
Can be requested by the sequence recipient by setting
bits within the F_CTL of the ACK frame
Same class of service delimiter as the sequence being
aborted is used for ABTS frame
OPT-2T01
9899_06_2004_X
Timers
Four different timers used
Receiver-transmitter time-out (R_T_TOV)
Error detect time-out (E_D_TOV)
Resource allocation time-out (R_A_TOV)
Connection request time-out (C_R_TOV)
Used in Class 1 You will never see class one
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Timers: R_T_TOV
Receiver-Transmitter Time-out
Used to time events at the link level
Loss of synchronization
Times Responses for link reset protocol
Generally controlled in hardware for all link
configurations
Default value in FC Standard is 100ms
OPT-2T01
9899_06_2004_X
Timers: E_D_TOV
Error Detect Time-out
Timers for events and responses at the
sequence level
Missing ACK or R_RDY when buffer credit has
reached zero
Class 1 or 2 expects response from data frames
N_Port logout
Timer value is set at fabric login to accommodate
the network environment for better scaling
according to delivery time of frames
Default is 10 sec
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Timers R_A_TOV
Resource Allocation Time-out
Time-out value for how long to hold resources
associated with a failed operation
Needed to free shared resources for reuse
Value to determine how long a port needs to keep
responding to a link service request before an error
is detected
R_A_TOV is 2 times E_D_TOV
Default setting in Pt to Pt is 20 sec and fabric is 120
seconds
OPT-2T01
9899_06_2004_X
Timers: CR_TOV
Connection Request Time-out
Determines how long the fabric can hold
a class-1 request in the queue during
connection establishment
Allows for separation of the time in a stacked
queue from the E_D_TOV; This separates
queuing time from frame transit time
Helps in controlling F_BSY issues
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Recovery: Class 3
Errors in class 3 sequence can only be detected by
the Sequence recipient because there are no ACKs
or rejects in class 3
Class 3 SR will discard single or multiple frames
until the exchange is terminated
The upper level recovery may retransmit the entire
Sequence or at least the sequence following the
error detection
OPT-2T01
9899_06_2004_X
Recovery: Class 3
Errors a class 3 operation can detect:
Out of order delivery and potential missing frame based
on SEQ_CNT
If missing frame is not Rxed within E_D_TOV
Indication of a new sequence when last frame of previous
Sequence has not been received (in-order delivery set)
Relative offset not in order with an order delivery set
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Abort Sequence: ABTS
ABTS can be sent under abnormal conditions
End-to-end credits not required
Sequence initiative not required
Open sequence not required
Maximum number of concurrent sequences allowed
Unidirectional for class 1 connection
The reply to an ABTS is a Basic_Accept
iSCSI
RFC 3720
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Session Modules
What is iSCSI and what is the big picture?
iSCSI protocol Introduction
The iSCSI connection
Security, data integrity and error recovery
iSCSI protocol details in-depth
Simple iSCSI connection flows
Service location protocol for IP storage
OPT-2T01
9899_06_2004_X
What Is iSCSI?
A SCSI transport protocol that operates on
top of TCP
Encapsulates SCSI-3 CDBs (Control Descriptor Blocks)
and Data into TCP/IP byte-streams (defined by SAM-2
SCSI Architecture Model 2)
Allows IP hosts to access IP or Fibre Channel-connected
SCSI targets
Allows Fibre Channel hosts to access IP SCSI targets
Standards status
RFC 3720 (assigned May 2004)
Major industry support (Cisco, IBM, EMC, HP, Microsoft)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Storage Technology
SCSI Domain
To be functional, a SCSI Domain needs to contain a
SCSI device that contains a target and a SCSI
device that contains an Initiator
SCSI Device
Initiator
Service Delivery
Subsystem
SCSI Device
Target
P
o
r
t
P
o
r
t
Device Service Request
Device Service Response
Task Request
Task Response
Task
Manager
Logical
Unit 1
Device
Server
Application
Client
Application
Client
OPT-2T01
9899_06_2004_X
SAN, NAS, iSCSI Comparison
Adopted from IBM Redbook IP Storage Networking:
IBM NAS & iSCSI Solutions
DAS
SCSI
Computer System
SCSI Bus Adapter
SCSI Device Driver
Volume Manager
File System
Application
SAN
SAN
FC
Fibre Channel HBA
SCSI Device Driver
Volume Manager
File System
Application
iSCSI
Appliance
IP
File System
Application
SCSI Device Driver
iSCSI Driver
TCP/IP stack
NIC
Volume Manager
NIC
TCP/IP stack
iSCSI layer
Bus Adapter
iSCSI
Gateway
IP
FC switch
File System
Application
SCSI Device Driver
iSCSI Driver
TCP/IP stack
NIC
Volume Manager
NIC
TCP/IP stack
iSCSI layer
Bus Adapter
NAS
IP
NIC
TCP/IP stack
I/O Redirector
File System
Application
NFS/CIFS
NIC
TCP/IP stack
File System
Device driver
File I/O
Block I/O
Block I/O
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
IP Storage Networking
IP storage networking provides solution to carry storage
traffic within IP
Uses TCP: A reliable transport for delivery
Can be used for local data center and long haul applications
Two primary protocols:
IP
TCP TCP
FCIP FCIP
FC FC SCSI
Data
IP
TCP TCP
iSCSI iSCSI SCSI Data
iSCSIIP-SCSIUsed to Transport SCSI CDBs and Data
within TCP/IP Connections
FCIPIP-SCSIUsed to Transport SCSI CDBs and Data
within TCP/IP Connections
OPT-2T01
9899_06_2004_X
Initiator and Target Model for iSCSI
InitiatorSCSI device which is capable of originating SCSI
commands and task management requests
TargetSCSI device which is capable of executing SCSI
commands and task management requests
iSCSI
Gateway
iSCSI
Initiator
iSCSI
Target
FC
Target
FC
Initiator
iSCSI
Gateway
FC
Initiator
FC
Target
iSCSI
Target
iSCSI
Initiator
iSCSI Target Mode iSCSI Initiator Mode
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Components
iSCSI is an end-to-end protocol
iSCSI has human readable SCSI device (node)
naming
iSCSI includes the following base components:
IPSEC connectivity security
Authentication for access configuration
Discovery of iSCSI nodes
Process for remote boot
iSCSI MIB standards
OPT-2T01
9899_06_2004_X
iSCSI: Internet SCSI PDU
The iSCSI layer encapsulates the SCSI CDB into a iSCSI
Protocol Data Unit (PDU) and forwards it to the Transmission
Control Protocol (TCP) layer
It also extracts the CDB from an iSCSI PDU received from the
TCP layer, and forwards the CDB to the SCSI layer
iSCSI mapping provides the SCSI-3 command layer with a
reliable transport
The communications between the Initiator and target will
occur over one or more TCP connections
The TCP connections form a session and will carry the iSCSI
PDUs; the sessions are given an ID called a connection ID
(CID); there are two parts of the ID, Initiator Session ID (ISID)
and Target ID (TSID) and together make up an I_T nexus
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Data Server Host
Initiator iSCSI Target
FC Storage Device
FC Target
Application
Client
requests
data from
LUN 1
Application
Client
requests
data from
LUN 1
P
o
r
t
P
o
r
t
P
o
r
t
Device Service Request
Device Service Response
Logical
Unit 1
Logical
Unit 1
Device
Server
Device
Server
Logical
Unit 2
Logical
Unit 2
Device
Server
g
e
2
g
e
2
f
c
1
f
c
1
Target
Mapping
LUN 1 = LUN 2 LUN 1 = LUN 2
iSCSI Model
SCSI CDBs Carried in iSCSI PDUs
SCSI CDBs carried by Fibre
Channel Exchange and Sequences
OPT-2T01
9899_06_2004_X
IP IP
iSCSI Stack
SCSI Applications (File Systems, Databases) SCSI Applications (File Systems, Databases)
SCSI Block Commands SCSI Block Commands SCSI Stream Commands SCSI Stream Commands Other SCSI Commands Other SCSI Commands
iSCSI
SCSI Over TCP/IP
iSCSI
SCSI Over TCP/IP
TCP TCP
Ethernet Ethernet Fibre Channel Fibre Channel Parallel SCSI Interfaces Parallel SCSI Interfaces
FCP SCSI Over FC FCP SCSI Over FC Parallel SCSI Transport Parallel SCSI Transport
SCSI Commands, Data, and Status SCSI Commands, Data, and Status
SCSI Device-Type
Commands
SCSI Generic
Commands
SCSI Transport
Protocols
Layer 3
Network Transport
Layer 2
Network
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Sourced Port Destination Port Destination Port
Sequence Number
Acknowledgment Number
Window
Checksum
Offset Reserved U A P R S F
Options and padding
Urgent Pointer
Destination
Address
Source
Address
Type IP TCP Data Data
461500 bytes
8 6 6 2
FCS
4 Octet
Preamble
TCP Header
iSCSI
encapsulated
iSCSI
encapsulated
Opcode Opcode Opcode Specific Fields Opcode Specific Fields
Length of Data (after 40Byte header) Length of Data (after 40Byte header)
LUN or Opcode-specific fields LUN or Opcode-specific fields
Initiator Task Tag Initiator Task Tag
Opcode Specific Fields Opcode Specific Fields
Data Field Data Field
Well-known Ports:
21 FTP
23 Telnet
25 SMTP
80 http
3260 iSCSI 3260 iSCSI
iSCSI
iSCSI Packet
OPT-2T01
9899_06_2004_X
iSCSI Naming and Discovery
RFC 3721
Initiator and target require iSCSI names
Name is location independent
iSCSI node name = SCSI device name of iSCSI device
Associated with iSCSI nodes, not adapters
Up to 255 byte displayable/human readable string
(UTF-8 encoding)
Use SLP, or iSNS, or query target for names (sendtargets)
Two iSCSI name types:
iqniSCSI qualified name
euiExtended Unique Identifier (IEEE EUI-64)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Name Structure
iSCSI Name Structure
Type . Unique String
iqn.1987-05.com.cisco.1234abcdef987601267da232.betty
iqn.2001-04.com.acme.storage.tape.sys1.xyz
Type Date
Organization
Naming Authority
Subgroup Naming Authority or
String Defined by Organization Naming Authority . . .
iqn
eui
Date = yyyy-mm when
Domain Acquired
Reversed Domain Name
Type EUI-64 Identifier (ASCII Encoded Decimal)
.
eui.02004567a425678d
Host Name
OPT-2T01
9899_06_2004_X
iSCSI Naming and Addressing Terms
iSCSI host name
Name of computer
iSCSI initiator name (iSCSI Node)
Name created at iSCSI driver load time on host system
InitiatorTarget Session ID (SSID)
One or more TCP connections between Initiator and target;
This session ID is derived from iSCSI host name, iSCSI
target name and TSID, ISID
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI initiator address
IP address on Initiator interface; Initiator can have
multiple addresses
Initiator portAlso known as network portal
IP address on initiator no port number assigned, again
Initiator can have several network portals
Target portAlso known as network portal
IP address + TCP port number on target interface
There can be more then one target interface
OPT-2T01
9899_06_2004_X
iSCSI target name
Used to identify multiple SCSI targets behind a single IP
address+port; This name is globally unique
Initiator session ID
This is an initiator-defined session identifier; It will be the
same for all connections within a session; An iSCSI
initiator port is uniquely identified by the value pair (iSCSI
Initiator Name, ISID)
Target session ID
Target assigned tag for a session with a specific named
initiator that, together with the ISID uniquely identifies a
session with that initiator
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI network entity
Client
It is a combination of
the following:
iSCSI initiator
iSCSI host
iSCSI initiator address
Initiator port (network
portal)
iSCSI network entity
Server
Is a combination of the
following:
iSCSI target name
Target port (network
portal)
Initiatortarget
session (SSID)
OPT-2T01
9899_06_2004_X
iSCSI Node
iSCSI Initiator or iSCSI Target; There can be one or more
iSCSI nodes in a network entity
iSCSI node will equal
iSCSI initiator name
iSCSI target name
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Portal Group
Groups multiple TCP connections across the same
session that is is sent across multiple portals
The portal groups are identified by a portal group tag
(1-65535)
One or more portal groups can provide a path to the
same iSCSI node (target node or initiator node)
SendTargets requires portal group tag
OPT-2T01
9899_06_2004_X
iSCSI Discovery Methods
Small networks
Static configuration, initiators and targets
SendTargets command makes configuration easier
Medium-sized networks
Service Location Protocol (SLP multicast discovery)
Large-sized networks
iSNS (internet storage name service)
Includes soft zone domains
Includes database for ongoing management
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Architecture
Network Portal
10.1.30.1
Network Portal
10.1.40.1
iSCSI Node
(Initiator)
Network Portal
10.1.30.2
Network Portal
10.1.40.2
iSCSI Node
(Target)
iSCSI Node
(Target)
Network Entity (iSCSI Client)
Network Entity (iSCSI Server)
OPT-2T01
9899_06_2004_X
iSCSI Architecture
Network Portal
10.1.30.1
Network Portal
10.1.40.1
iSCSI Session (Target Side)
iSCSI Name +TSID=2
iSCSI Target NodeNode
(within Network Entity)
Portal Group 1
Network Portal
10.1.50.1
Portal Group 2
iSCSI Session (Target Side)
iSCSI Name +TSID=1
IP Network
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Session Model
An iSCSI session exists between a single iSCSI initiator (host) and a
single iSCSI target (iSCSI router)
An iSCSI session consists of one or more iSCSI (TCP) connections
Login phase begins each connection
Deliver SCSI commands in order
iSCSI (TCP) Connection
iSCSI Session
iSCSI
Storage Router
iSCSI Routing Instance
TCP/3260
TCP/3260
TCP/3260
OPT-2T01
9899_06_2004_X
iSCSI Session Images
Across all connections within a session, an initiator
sees one target image
The target image would represent all identifying
elements such as LUNs
A target also sees one initiator image across all
connections within a session
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
3.3.3.3 4.4.4.4 5.5.5.5
Target-1 Target-2 Target-3
Target-5 Target-4
iSCSI Target name
configured on iSCSI
Device
iSCSI
Node
s
Target Port These
Network Portals listens
for iSCSI connections
on WKP 3260
iSCSI Network
Entity-Server
iSCSI Initiator
address
Initiator Port
iSCSI Host Name
iSCSI Initiator Name
IP
TCP
Connection
TSID
ISID
iSCSI
Driver,
Storag
e NIC
Storage Systems
Configuration will
make the
connections
between storage
and iSCSI Initiator
SSID
1.1.1.1 2.2.2.2
mike.cisco.com
disk.cisco.com.stor.1
23
Put It All Together for iSCSI
OPT-2T01
9899_06_2004_X
iSCSI Connections and SCSI Phases
A SCSI command and its associated data and status phase
exchanges must traverse the same TCP connection
Linked SCSI commands can traverse separate TCP
connections for scalability
iSCSI (TCP) Connection 1 iSCSI (TCP) Connection 1
iSCSI Session
iSCSI Storage Router
iSCSI Routing
Instance
SCSI Command (1) (Read)
SCSI Data (1)
SCSI Status (1)
iSCSI (TCP) Connection 2 iSCSI (TCP) Connection 2
Linked SCSI Commands
SCSI Command (1) (Write)
SCSI Data (1)
SCSI Status (1)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Connection Session
iSCSI Session has four phases
Initial login phase
Security authentication phase
Operational negotiation phase
Full featured phase
Session Can Process SCSI Commands
and Data after Login Is Complete
OPT-2T01
9899_06_2004_X
iSCSI Session Establishment
Login Begins with the First Connection
Initial login phase
Initiator sends login with text strings for InitiatorName, TargetName, and
authentication options (which are then selected by the target)
Security authentication phase
Authentication text exchanges (ID, password, certificates, etc)
Operational negotiation phase
Each side (initiator and target) negotiate the supported options using
Keyword=value, or Keyword=value,value,value
Amount of unsolicited buffer
Types of data delivery
Solicited, unsolicited, immediate, etc
Full featured phase
Can carry SCSI CDBs/data, task management, and responses
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Session Key Points
Sessions:
iSCSI Session = a group of TCP connections linking
an initiator with a target (i.e., can be one or more
connections)
NOTE: A TCP connection that is part of an iSCSI
session will only be used to carry iSCSI traffic
The iSCSI initiator and target use the session to
communicating iSCSI commands, control
messages, parameters, and data to each other
TCP connections can be added and removed from a
session using the iSCSI Login/Logout commands
OPT-2T01
9899_06_2004_X
iSCSI Sessions
During session establishment, the target identifies
the SCSI initiator port (the I in the I_T nexus )
through the value pair (InitiatorName, ISID)
Any persistent state (e.g., persistent reservations)
on the target associated with a SCSI initiator port is
identified based on this value pair
Any state associated with the SCSI target port (the
T in the I_T nexus ) is identified externally by
the TargetName and portal group tag and internally
in an implementation dependent way
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Connection Allegiance
For SCSI commands that require data transfer, the
data phase and status phase must be sent over the
same TCP connection used by the command phase
Consecutive commands that are part of a SCSI task
may use different connections within the session
(linked commands)
Connection allegiance is strictly per-command and
not per task
Multiple connections allow the iSCSI session to be
scaled across multiple links/devices
OPT-2T01
9899_06_2004_X
iSCSI Connection Termination
Session may end with logout or I/O error causing
dropped connection
TCP connections are closed through normal methods
i.e. TCP FINs
Graceful shutdowns can only occur when no
outstanding tasks are on the connection and not in
full-feature phase
Termination of connection abnormally may require a
recovery method by logout request for all connections;
This prevents stale iSCSI PDUs being received after
going down
Logout can also be issued by the target through
asynchronous message PDU
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Security
Two types of security
IPSec secures TCP/IP nodes; setup at TCP/IP startup
before iSCSI login
Session authentication via IKE (Internet Key Exchange)
Packet by packet authentication (also provides Integrity)
Privacy via encryption (also provides Integrity)
See SEC-IPS
iSCSI techniques (done/setup during iSCSI Login)
Authentication (ensures nodes are authorized to use the
iSCSI target node) may use SRP, Chap, or Kerberos
OPT-2T01
9899_06_2004_X
Challenge Handshake
Authentication Protocol
In-band initiator-target authentication
IP-SEC is not assumed
No clear text password accepted
Compliant iSCSI initiators and targets MUST
implement the CHAP (RFC1994)
Implementations MUST support use of up to
128 bit random CHAP secrets
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Security
Various levels of security can fit different
topologies
Examples:
Secure main floorNo security
Campus LANiSCSI authentication and CRC32c
(digests)
Remote private WANIPSec with session/packet
authentication
Remote internet WANIPSec with privacy encryption
OPT-2T01
9899_06_2004_X
iSCSI Data Integrity
Basic level of end-to-end data integrity can be
reasonably handled by TCP using the standard
checksum
iSCSI CRC32c digest checks for Integrity beyond
TCP/IP XOR checksum
a) Header digest
b) Data payload digest
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Digests (Checksums)
Optional header and data digests protect the integrity of
the header and data, respectively; The digests, if
present, are located, respectively, after the header and
PDU-specific data, and cover the proper data and the
padding bytes
The existence and type of digests are negotiated during
the login phase
The separation of the header and data digests is useful
in iSCSI routing applications, in which only the header
changes when a message is forwarded; In this case,
only the header digest should be recalculated
Digests are not included in data or header length fields
A zero-length data segment also implies a zero-length
data-digest
OPT-2T01
9899_06_2004_X
Error Recovery
Two Considerations for Errors
An iSCSI PDU may fail the digest check and be
dropped, despite being received by the TCP layer;
The iSCSI layer must optionally be allowed to
recover such dropped PDUs
A TCP connection may fail at any time during the
data transfer; All the active tasks must optionally
be allowed to be continued on a different TCP
connection within the same session
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Error Recover: iSCSI Initiator
A. NOP-OUT to probe sequence numbers of
the target
B. Command retry
C. Recovery R2T support
D. Requesting retransmission of status/data/R2T
using the SNACK facility
E. Acknowledging the receipt of the data
F. Reassigning the connection allegiance of a task
to a different TCP connection
G. Terminating the entire iSCSI session to start fresh
OPT-2T01
9899_06_2004_X
Error Recover: iSCSI Target
A. NOP-IN to probe sequence numbers of the
initiator
B. Requesting retransmission of data using the
recovery R2T feature
C. SNACK support
D. Requesting that parts of read data be
acknowledged
E. Allegiance reassignment support
F. Terminating the entire iSCSI session to force the
initiator to start over
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Error Recover Classes
Within a command
(i.e., without requiring command restart)
Within a connection (i.e., without requiring the
connection to be rebuilt, but perhaps requiring
command restart)
Connection recovery (i.e., perhaps requiring
connections to be rebuilt and commands to be
reissued)
Session recovery
OPT-2T01
9899_06_2004_X
Error Levels
Level determined during logon text negotiation
Error recovery level is proposed by an originator in a
text negotiation
Presentation_ID.scr
iSCSI PROTOCOL
DETAILS IN-DEPTH
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
iSCSI Key Points
Tasks:
A linked set of SCSI commands
One and only one SCSI command at a time can
be processed within any given iSCSI task
Initiator Task Tag (ITT) and Target Transfer Tag (TTT)
Initiator tags for all pending commands must be unique
initiator-wide
SCSI Data PDUs are matched to their corresponding
SCSI commands using tags specified in the protocol
ITT for unsolicited data
TTT for solicited data
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Key Points
Solicited or unsolicited messages:
Initiator to target
User data or command parameters will be sent as either solicited data or
unsolicited data
Solicited data is sent in response to ready to transfer (R2T) PDUs
Unsolicited data can be part of an iSCSI command PDU ( Immediate
data ) or an iSCSI data PDU
The maximum size of an indi vidual data PDU or the immediate part of
the initial unsolicited burst may be negotiated during login
Target to initiator
Ready to transfer (R2T) message to Initiator, requesting data for
a write command
Command responses
Asynchronous messages (SCSI and iSCSI) describing an unusual
or error event
OPT-2T01
9899_06_2004_X
iSCSI Numbering
iSCSI uses command and status numbering
Command numberingSession wide and is used for
ordered command delivery over multiple connections
within a session; It can also be used as a mechanism for
command flow control over a session
Status numberingper connection and is used to enable
recovery in case of connection failure
Fields in the iSCSI PDUs communicate the
reference numbers between the initiator and target
During periods when traffic on a connection is
unidirectional, iSCSI NOP PDUs may be issued to
synchronize the command and status ordering
counters of the initiator and target
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SCSI Command Numbering
and Acks within iSCSI
Initiator and target device have three sequence
number registers per session
CmdSNCurrent command sequence number; Sent
by initator
ExpCmdSNExpected command by the target; Sent to
the initiator by the target to acknowledge CmdSN; Can
be used to ACK several sequences
MaxCmdSNMaximum number target can receive in its
queue; Can be sent to Initiator from target to adjust
queue size
OPT-2T01
9899_06_2004_X
SCSI Command Numbering
iSCSI supports ordered command delivery within
the session
Command-Sequence-Number (CmdSN) is assigned
by initiator and carried in the iSCSI PDU
CmdSN starts at iSCSI login
CmdSN not assigned to data-out (DataSN used)
Immediate delivery does not advance CmdSN
iSCSI must deliver commands to target in order of
CmdSN and will not increment until executed state
by target
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SCSI Status Numbering
Status Sequence Number (StatSN) is used to
number responses to the Initiator from the target
ExpStatSN is sent by Initiator to acknowledge
status
Status numbering starts after Login; During login
there can be only one outstanding command per
connection
OPT-2T01
9899_06_2004_X
Initiator iSCSI OPcodes
0x00 NOP (No operation, used as ping to target gateway)
0x01 SCSI command (Indicates encapsulated iSCSI packet
has a SCSI CDB for target device)
0x02 SCSI task management command
0x03 iSCSI login
0x04 text command
0x05 SCSI data-out (Write data to target device)
0x06 iSCSI logout
0x10 SNACK (Request retransmission from target)
0x1c-0x1e Vendor specific codes
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Target iSCSI OPcodes
0x20 NOP-In (No operation in, used for ping response from target
0x21 SCSI response (Indicates encapsulated iSCSI packet has status
or from target device)
0x22 SCSI task management response
0x23 login response
0x24 text response
0x25 SCSI data-in (Read data from target)
0x26 logout response
0x31 Ready to transfer (Sent to initiator from target to indicate it is
ready to receive data)
0x32 async message (Message from target to indicate special conditions)
0x3c-0x3e vendor specific codes
0x3f reject
OPT-2T01
9899_06_2004_X
iSCSI PDUs
Several different types of iSCSI PDUs used, each of
the different iSCSI Operation Codes (Opcodes)
determine what iSCSI PDU to use; Some of the
more used PDUs are:
Login and logout PDU
Command and response PDU
Data-In and data-out PDU
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Login PDU
If Set to 1 = Recovery
from Failed Connection
If Set to 1 Indicates
Initiator Is Ready to
Transit to Next Stage
Initiatior May Provide
Initial Text Parameters
in This Area
Unique ID
for This
Connection
Current Stage/Next Stage
0 Security Negotiation
1 Login Operational Negotiation
- 3 Full Feature Phase
Initiator ID
for This
Connection
OPT-2T01
9899_06_2004_X
iSCSI Login
Login Phase used to:
Enable TCP connection (Target listens on well known port)
Authentication (CHAP)
Negotiate session parameters
Open security protocols
Mark the TCP connection as a iSCSI session
and assign IDs
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSCSI Text Mode
During Login Some Sessions or Connection
Parameters May Be Negotiated in a Text Format
list = values sent in order of preference
Example of values can be:
MaxConnections=<1-65535> T or I
Sendtargets=all I only
Targetname=<iSCSI-Name> T or I
SessionType=<Discovery|Normal> I only
Othersaddressed later in slides (see RFC)
OPT-2T01
9899_06_2004_X
iSCSI Full Feature Phase
iSCSI PDUs can be sent
PDUs must flow over same connection as login
Size of PDU is negotiated during login
A Connection Is in Full Feature Mode after a
Completed Login
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Data Sequencing within iSCSI
The iSCSI PDUs used for data input and output
are the 0x05 iSCSI command and the 0x25 iSCSI
command, along with R2T (0x31 ready to transfer)
DataSN is a number field and advances by 1 for each input
(read) and output (write)
Targets will operate in two modes, solicited (R2T) or
unsolicited (non-R2T)
Target operating in R2T mode can only receive
solicited data from the initiator
R2TSN advances by one for each received R2T
during the data transfer
The DataSN and R2TSN fields are for the initiator
to detect missing data
OPT-2T01
9899_06_2004_X
Data-Out PDU
Final Bit Say This
Is the Last PDU of
a Sequence
Data Segment
Length Based
on Capabilities
Exchange
LUN Number
for Data
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Data-In PDU
Final Bit say this
is the last read
of a sequence
Acknowledge Bit
used when error
recovery level is 1
or higher
Status bit tells that there
is meaningful data in the
StatSN, Status, and
Residual Count fields
Flags valid when
S bit is set tells
how to read
Residual Count
OPT-2T01
9899_06_2004_X
iSCSI Read Command Example
1. Initiator sends iSCSI command PDU (CDB=Read)
2. Target sends iSCSI data-in PDU(s)
3. Target sends iSCSI response PDU
Notes:
Solicited data via read command PDU (Initiator requests data
from the target)
Target may satisfy the single read command with multiple iSCSI
data read PDUs (PDUs can be out-of-order)
Command is not complete until all data and status is recei ved
by the initiator
Good status can be sent within the last iSCSI data-in PDU
All iSCSI data-in PDUs and the response PDU will be delivered
on the same TCP connection that the command was sent on
All data-in PDUs will carry the same value in the ITT field
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SCSI Command PDU
Lets Target Know if
More Data Is to
Foll ow along with
Expected Data
Transfer Length
R=1 If the
Command Is
Expected to
Input Data
W=1 If the
Command Is
Expected to Output
Data
Task Attributes See
RFC for Detailed
Meaning
16 bytes of
SCSI CDB,
Some SCSI Commands
Have Additional Data and
This Field Is Used for the
Accompanied Data
CRC If
Capabilities
Required This
OPT-2T01
9899_06_2004_X
SCSI Response PDU
Ox00 = Command Completed at Target
0x01 = Target Failure
0x08 0xff = Reserved for Vendor Response
SCSI Status
per SAM2
CRC Check
Sums
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SCSI Status and Response
Fields for iSCSI OpCode 0x21
The status field of the iSCSI PDU is used to report
status of the command back to the initiator
The specific status codes are documented in
the SCSI architectural model for the device
Response field contains the iSCSI codes that
are mapped to the SAM-2 response
OPT-2T01
9899_06_2004_X
Ready to Transfer PDU
When the initiator has sent a SCSI write command
to the target the target can specify the blocks be
delivered in a convenient order; This information
is passed to the initiator in the R2T PDU
Allowing an initiator to write data to a target
without a R2T is agreed upon during login
The target may send several R2T PDUs and
have several data transfers pending if allowed
by the initiator
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Task Management
Functions to provide the initiator a way to control
management of the target device
Abort the TASK
Clear allegiance
Logical reset
Target reset
Each of these and more are broken down in detail
in the iSCSI RFC
OPT-2T01
9899_06_2004_X
SACK, NOP-IN, NOP-OUT
SACK
Optional
Used to request retransmission of numbered responses, data
or R2T PDUs from the target
NOP-IN
Sent by a target as a response to a NOP-Out, as a
ping to an initiator
Or a means to carry a changed ExpCmdSN and/or MaxCmdSN
if there is no other PDU to carry them for a long time
NOP-OUT
Used by Initiator as a ping command , to verify that a
connection/session is still active and all its components
are operational
Used to confirm a changed ExpStatSN if there is no other
PDU to carry it for a long time
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Message Synchronization and Steering
Steering of iSCSI out of order TCP segments into
pre-allocated buffers instead of temporary buffers
To decrease reassembly time
Not needing to rely on message length information
Provides a synchronization method using fixed
interval markers telling where the start of the next
iSCSI PDU is in the buffer
Optional for iSCSI RFC
OPT-2T01
9899_06_2004_X
List of Negotiated Parameters Prior
to Going into Full Feature Mode
Header Digest
Data Digest
Max Connections
Send Targets
Target Name
Initiator Name
Target Alias
Initiator Alias
Target Address
Target Portal Group Tag
Initial Ready 2 Transfer
Immediate Data
Max Rec Data Segment Length
Max Burst Length
First Burst Length
Default Time 2 Wait
Default Time 2 Retain
Max Outstanding R2T
Data PDU In-order
Data Sequence In-order
Error Recovery Level
Session Type
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Standards: Where to Find Details
http://www.ietf.org/html.charters/ips-charter.html
T10 Technical committeewww.t10.org
Technical committee of the National Committee on
Information Technology Standards (NCITS), deals with
the storage devices
T11 Technical committeewww.t11.org
Technical committee of the NCITS, deals with the physical
interface and transport level
SIMPLE ISCSI
CONNECTION FLOWS
EXAMPLE OF DISCOVERY
SESSION WITH CHAP
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Establish Initial TCP Session Phase
0X03 CommandLogin
Key Values Are Sent, InitiatorName, InitiatorAlias,
SessionType=Discovery, AuthMethod=CHAP/none,
HeaderDigest, DataDigest
TCP Port
3260
TCP Port
3260
Target Initiator
TCP port 1026
(Random)
Discovery
Session
TCP port 1026
(Random)
Discovery
Session
iSCSI Driver
This
Device
Has
Already
Initialized
Onto the
Fibre
Channel
This
Device
Has
Already
Initialized
Onto the
Fibre
Channel
iSCSI Flows
0X23 Login Response
Status= Accept Logi n (0X0000), Keyvalues Are Sent,
AuthMethod=CHAP, HeaderDigest=none, DatDigest=none
0X03 CommandLogin
Key Values Sent, Initi atorName, InitiatorAli as,
SessionType=Discovery, CHAP_A=5 (CHAP with MD5)
0X23 Login Response
Status=Accept Logi n, KeyValues CHAP_A, CHAP_I & CHAP_C
OPT-2T01
9899_06_2004_X
0X23 Login Response
Final PDU in Sequence, Status= Accept login (0X0000)
0X23 Login Response
Status=Accept Logi n, Negotiate Session Parameters
0X03 CommandLogin
Key Values Are Sent, InitiatorName, InitiatorAlias,
SessionType=Discovery, CHAP_R, CHAP_N
iSCSI Flows
0X03 CommandLogin
Key Values Sent, Initi atorName, InitiatorAli as,
SessionType=Discovery, Negotiate Session Parameters
End of Authentication Phase
Start of Parameter Negotiation Phase
for Discovery Session
Target Initiator
TCP port 1026
(Random)
Discovery
Session
TCP port 1026
(Random)
Discovery
Session
iSCSI Driver
TCP Port
3260
TCP Port
3260
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
0X24 Login Response
Final PDU in Sequence, KeyValue=TargetName (iqn number
along with target name configured on iSCSI Target)
0X04 Text Command
SendTargets=all
Target Initiator
TCP port 1026
(Random)
Discovery
Session
TCP port 1026
(Random)
Discovery
Session
iSCSI Driver
TCP Port
3260
TCP Port
3260
iSCSI Flows
0X03 CommandLogin
Key Values sent, Initi atorName, InitiatorAli as,
SessionType=Normal, TargetName, AuthMethod=CHAP,none
0X23 Login Response
Status=Accept Logi n, AuthMethod=CHAP
Establish TCP connection for target
TCP port 1027
(random)
Target
Session #1
TCP port 1027
(random)
Target
Session #1
Note the
Addition of
Another TCP
Session
Start of Target Session Authentication and
Target Session Parameter Negotiation
OPT-2T01
9899_06_2004_X
0X23 Login Response
Status=Accept Logi n, KeyValues CHAP_A, CHAP_I & CHAP_C
0X03 CommandLogin
Key Values are sent, InitiatorName, InitiatorAlias,
SessionType=Normal, CHAP_R, CHAP_N
0X03 CommandLogin
Key Values are sent, InitiatorName, InitiatorAlias,
SessionType=Normal, TargetName, CHAP_A=5
Target Initiator
iSCSI Driver
TCP Port
3260
TCP Port
3260
iSCSI Flows
0X23 Login Response
Status=Accept Logi n
TCP Port 1027
(Random)
Target
Session #1
TCP Port 1027
(Random)
Target
Session #1
0X03 CommandLogin
Key Values sent, Initi atorName, InitiatorAli as,
SessionType=Normal, TargetName, Negotiate session
Parameters
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Target Initiator
iSCSI Driver
TCP Port
3260
TCP Port
3260
TCP Port 1027
(Random)
Target
Session #1
TCP Port 1027
(Random)
Target
Session #1
0X01 iSCSI Command
SCSI Inquiry CDB 0X12
iSCSI Flows
0X23 Login Response
Status=Accept Logi n, Negotiate session Parameters
0X25 iSCSI Data-in (read)
FCIP CONCEPTS
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Agenda
What FCIP Is About
The Standards
Fibre Channel T11 Standards
IETF IPS Working Group Drafts
Understanding FCIP Protocol
Relationships to Other SCSI Transport
Technologies
OPT-2T01
9899_06_2004_X
FCIP: Fibre Channel over IP
FCIP provides a standard way of encapsulating FC
frames within TCP/IP, allowing islands of FC SANs
to be interconnected over an IP-based network
TCP/IP is used as the underlying transport to
provide congestion control and in-order delivery
of error-free data
FC frames are treated the same as datagrams
It is not iFCP, mFCP, IPFC, iSCSI transports
or extended FC fabric
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FCIP Design
IP Network
Tunnel Session Tunnel Session
IP Services
Available at Aggregated
FC SAN Level
FC Tape
Library
FC Tape
Library
FC Server
FC
Server
FC
Server
FC
JBOD
FC
JBOD
FC Switch
FC Switch
FC Switch
FC Switch
FC Switch
FC Switch
FC Switch
FC Switch
FSPF Routing
Backbone
FSPF Routing
Backbone
FSPF Routing
Backbone
FSPF Routing
Backbone
Fiber
Channel
SAN
Fiber
Channel
SAN
FCIP
Tunnel
FCIP
Tunnel
FC
Server
Tunnel Session Tunnel Session
OPT-2T01
9899_06_2004_X
Four (4) Specifications Define Basic FCIP
ANSI: http://www.t11.org/index.htm
FC-SW-2 describes the operation and interaction of Fibre
Channel switches, including E_Port, B_Port and fabric
operation
FC-BB-2 is a mapping that pertains to the extension of
Fibre channel switched networks across a TCP/IP network
backbone and defines reference models that support
E_Port and B_Port
IETF IPS working group:
Fibre channel over TCP/IP covers the TCP/IP requirements
for transporting Fibre Channel frames over an IP network
FC frame encapsulation defines the common Fibre Channel
encapsulation format
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
ANSI: FC-SW-2 Standard
E_Ports are used at both ends of an Inter
Switch Link (ISL)
E_Ports forward user traffic (storage data) and
control information (class F SW_ILS frames
containing FSPF, zone exchanges, etc.)
FC-SW-2 defines fabric merge procedures
(Domain_ID assignment, zone transfers, etc.)
FC-SW-2 also defines FSPF
OPT-2T01
9899_06_2004_X
ANSI: FC-SW-2 Essentials (Recap)
E_Ports provide switch-to-switch connectivity
E_Ports negotiate parameters such as:
ELPExchange Link Parameters
ESCExchange Switch Capabilities
FSPF is enabled over E_Ports only
Separate fabrics can be merged over E_Ports
Zoning information is exchanged over E_Ports
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
IETF FCIP: Fibre Channel Over IP
Each interconnection is called a FCIP link and can
contain one (1) or more TCP connection(s)
Each end of a FCIP link is associated to a virtual
ISL link (VE_Port or B_Access Portal)
VE_Ports communicate between themselves just
like normally interconnected E_Ports by using
SW_ILS: ELP, EFP, ESC, LKA, BF, RCF, FSPF, etc.
B_Access portals communicate between
themselves by using SW_ILS: EBP, LKA
The result (when all goes well ) is a fully merged
Fibre Channel fabric between FC switch SANs
OPT-2T01
9899_06_2004_X
IETF FCIP
IETF draft standard that allows IP connectivity
to link Fibre Channel storage area networks
across WANs
Two methods can be used
1) Similar to Cisco STUNNailed up tunnel
2) Similar to DLSWDynamic peering method
We will visit the details of each in later slides
draft-ietf-ips-fcovertcpip
Draft 12 is current, will RFC Jan/Feb 2003
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FCIP Architecture Model
FCIP FCIP
FC-2 FC-2
FC-1
FC-0
FC-1
FC-0
TCP TCP
IP IP
LINK LINK
PHY PHY
TCP/IP
Network
SAN SAN
FCIP Link
Key:
FC-0Fibre Channel Physical
Media Layer
FC-1Fibre Channel Encode
and Decode Layer
FC-2Fibre Channel Framing
and Flow Control Layer
TCPTransmission Control
Protocol
IPInternet Protocol
LINKIP Li nk Layer
PHYIP Physical Layer
OPT-2T01
9899_06_2004_X
FCIP
End-station addressing, address resolution,
message routing, and other fundamental elements
of the network architecture remain unchanged
from the Fibre Channel model, with IP introduced
exclusively as a transport protocol for an
inter-network bridging function
IP is unaware of the Fibre Channel payload and
the fibre channel fabric is unaware of IP
Ethernet
Header
Ethernet
Header
//
CRC CRC IP
Checksum
IP
Checksum
TCP FCIP FCIP SCSI Data FCP FCP
//
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FCIP
FCIP only supports class 2, class 3, class 4,
and class F frames
No FC primitive signals or primitive
sequences supported
Physical signal sets used by FC ports to indicate
events, i.e. NOS, OLS, LR
IP transport is transparent to Fibre
Channel topology
OPT-2T01
9899_06_2004_X
Understanding FCIP Terms
FC end nodeA Fibre Channel device that uses the connection services
provided by the FC fabric
FC entityThe Fibre Channel specific functional component that
combines with an FCIP entity to form an interface between an FC
fabric and an IP network
FC fabricAn entity that interconnects various Nx_Ports attached to it,
and is capable of routing FC frames using only the destination ID information
in a FC frame header
FC fabric entityA Fibre Channel specific element containing one or more
Interconnect_Ports (see FC-SW-2) and one or more FC/FCIP entity pairs
FC frameThe basic unit of Fibre Channel data transfer
FC frame recei ver portalThe access point through which an FC frame and
time stamp enters an FCIP data engine from the FC entity
FC frame transmitter portalThe access point through which a reconstituted
FC frame and time stamp leaves an FCIP data engine
to the FC entity
FC/FCIP entity pairThe combination of one FC entity and one FCIP entity
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Understanding FCIP Terms (Cont.)
FCIP data engine (FCIP_DE)The component of an FCIP entity that handles
FC frame encapsulation, de-encapsulation, and transmission FCIP frames
through a single TCP connection
FCIP entityThe entity responsible for the FCIP protocol exchanges on the
IP network and which encompasses FCIP_LEP(s) and FCIP control and
services module
FCIP frameAn FC frame plus the FC frame encapsulation header, encoded
SOF and encoded EOF that contains the FC frame
FCIP linkOne or more TCP connections that connect one FCIP_LEP to
another
FCIP link endpoint (FCIP_LEP)The component of an FCIP entity that that
handles a single FCIP link and contains one or more FCIP_DEs
Encapsulated frame recei ver portal The TCP access point through which an
FCIP frame is received from the IP network by an FCIP data engine
Encapsulated frame transmitter portal The TCP access point through which
an FCIP frame is transmitted to the IP network by an FCIP data engine
FCIP special frame (FSF)A speciall y formatted FC frame containing
information used by the FCIP protocol
OPT-2T01
9899_06_2004_X
FCIP Diagram
FC Entity
TCP
Ports
WKP = 3225
VE_Port
FCIP_LEP
DE DE
FCIP Entity
TCP
Ports
WKP = 3225
VE_Port
FCIP_LEP
FCIP Physical Link
TCP Connection
FC Frames in TCP/IP
Class 3 and Class F Can Be
on Separate Ports or Connections
IP Address = 172.16.0.5
Ethernet Gigabit/WAN Interface
FCIP Link
Virtual ISL
IP Address = 192.168.1.10
Ethernet Gigabit/WAN Interface
FCIP Link End Point
FCIP Data Engine
(Detail)
FCIP Frame TX RX Portal
FC Frame Receiver Portal
FC/FCIP Enti ty Pair
TX RX
More than One TCP
Connection Is Allowed
Dynamic
CONNECTION
PORT for FCIP
Connections
Non Dynamic
Connections
FCIP Data Engine
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
ANSI Meets IETF E-Port
FC-BB-2
FCIP
OPT-2T01
9899_06_2004_X
ANSI Meets IETF B-Port
FC-BB-2
FCIP
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FCIP Standards Stack Details
This Will Be the ISL Connection
Either a Bridged connection or
E_Port; Depending on FCIP
Implementation selected by Vendor
OPT-2T01
9899_06_2004_X
Additional IETF Drafts
SLP: Service Location Protocol
draft-ietf-ips-fcip-slp
Used for dynamic discovery of FCIP ports
IPSec for storage
draft-ietf-ips-security
More details later on this requirement for FCIP
MIBs
draft-ietf-ips-scsi-mib
draft-ietf-ips-fcmgmt-mib
draft-ietf-ips-fcip-mib
FC-BB
Published ANSI project being superseded by BB-2
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
ANSI: FC-BB-2 Essentials
(FCIP E-Port)
Defines a slightly
complex model;
FC-BB-2 covers the
FC portion of this
model (FC entity
and some of above)
Ciscos FCIP E_Port
implementations
will closely follow
this model
OPT-2T01
9899_06_2004_X
IETF: FCIP Essentials
(FCIP E-Port)
FCIP follows the
model proposed
in FC-BB-2;
FCIP covers the
lower portion of this
model (FCIP entity
and below)
Ciscos FCIP E_Port
implementation will
follow this model
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
ANSI: FCIP Essentials
(FCIP B-Port)
Again the FC side
of the this model
follows SW-BB-2
standards
With B_Port there
is no FC switching
element so the
B_Port device will
not be seen as a
switch in the
fabric but as a
passive device
OPT-2T01
9899_06_2004_X
IETF: FCIP Essentials
(FCIP B-Port)
The FCIP part
of the B-Port
operation is the
same as FCIP
for the E_Port
Note in this
diagram that
implementations
of this standard
can be any
number of ports
from 1 to n
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
About FCIP Links
The FCIP interface
represents both the VE_Port
and the FCIP link
An FCIP link is defined
as one or more TCP
connections
FCIP link endpoint (LEP)
terminates FCIP links
FCIP data engine: One per
TCP connection
Entity 1
TCP
Ports
WKP = 3225
VE_Port
FCIP_LEP
DE DE
FCIP Link
Class F
Class 3
IP Address = 192.168.1.10
TCP/IP Network Interface
OPT-2T01
9899_06_2004_X
About the FC Entity
FC entity interfaces
(internally) with FCIP
entity
FC entity components:
Control and Services
Module
Provides FC frame and
timestamp along with
synchronization with
FCIP entity
Correct order delivery of
FC frames
Works with FCIP entity for
flow control
Computes end-to-end
transit time
Throws away expired frames
Answers to authentication of
TCP connection request
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
About the FCIP Entity
FCIP entity interfaces
(internally) with
FC entity
FCIP entity
components:
Provide FC frame and
timestamp to FC entity
Tells FC entity about
discarded bytes
Tells FC entity about new
and lost TCP connections
and reason codes
Monitors special frame
changes
Makes request to FC entity
for authentication
OPT-2T01
9899_06_2004_X
FCIP Link Endpoint: Details
FCIP_LEP is the translation point between
an FC entity and an IP network
LEP coordinates between FC and TCP flow
control mechanisms
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Error Detection and Recovery
Data engine uses various methods to detect errors
but does not correct errors
Rather, it inserts EOFa (abort) frame delimiters
when possible
Requests sent up to FC entity to handle recovery
OPT-2T01
9899_06_2004_X
IETF: Fibre Channel
Frame Encapsulation Header
Defines the encapsulation header for Fibre Channel frames
Not specific to FCIP
Includes timestamp, CRC and provision for special frames
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Initialization of Port
B_Port
Link initialization
Exchange link
parameters
Link reset
E_Port
Link initialization
Exchange link parameters
Reset link
Exchange switch
capabilities
Exchange fabric
parameters
Assign domain IDs
Establish routes
Merge zones if required
OPT-2T01
9899_06_2004_X
Link Initialization Flow
NOS = Not Operational Sequence
OLS = Offline Sequence
LR = Link Reset
LRR =Link Reset Response
AC = Acti vity State
OL = Offline State
LF
LF
LR
OL
LR
AC
AC
NOS
OLS
LR
Idle
LRR
Idle
E_Port on
Switch
B_Port or E_Port
on FCIP Device
These Are All Special Ordered
Sets of 8B/10B Coding
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Link Capture
E_Port on
Switch
B_Port or E_Port
on FCIP Device
LR & LRR to
Initialize Flow
Control
Parameters Per
FC-PH
NOS
LR
IDLE
R-RDY
IDLE
IDLE
IDLE
R-RDY
IDLE
LRR
At this Point B_Port Device Is Up and
E_Port to E_Port Exchange Continues
OPT-2T01
9899_06_2004_X
ISL E_Port
If It Is an E_Port FCIP Device or If the B_Port Is Now
up the Switch to Switch Exchange Continues
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
ELP Data Bit 15 of flag
will be a 1 for
B_Port
RA_TOV is fabric
wide timer, ED_TOV
is per Link
Credit value is one to
start to allow only one
out standing frame
during link start-up
Class 2 & 3
supported
PWWN & WWN,
Vendor ID also
OPT-2T01
9899_06_2004_X
E_Port and B_Port Summary
FC SAN
FC SAN
FC Switch
FC Switch
FCIP E-Port
FCIP E-Port
IP
Network
VE - Port
Exchange Fabric
Parameters
Exchange Fabric Parameters
Exchange Fabric
Parameters
Exchange Link Parameters
Exchange FCIP-
Link Parameters
ESC
ESC
ESC
FC SAN
FC SAN
FC Switch
FC Switch
FCIP E-Port FCIP E-Port
IP
Network
VB - Port
Exchange FCIP-Link
Parameters
Exchange Fabric Parameters
ESC (Exchange Switch Capabilities) if required
B Port Operation
E Port Operation
FCIP B-Port FCIP B-Port
7200 w/ PA-FC-1G 7200 w/ PA-FC-1G
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FCIP: ISL Connection
The E-Port or B-Port FCIP Connection Will Provide:
Simple name service across the IP tunnel
FC discovery between SAN islands
FSPF routing services between fabric switches
Management server information
Buffer credits
OPT-2T01
9899_06_2004_X
Comparisons
B-Port and E-
Port Differences
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FCIP Connection Establishment
Non-dynamic
TCP connection to a specific IP address
Dynamic
Discovery of FCIP entities using SLPv2
Use of FCIP special frame
Use of options
OPT-2T01
9899_06_2004_X
Non-Dynamic TCP Connections
The FCIP entity is informed of a TCP connection
is needed (Most likely done by configuration
parameters in the device)
IP address and security features are established
(Configured)
Destination WWN is determined (Configured)
TCP/IP parameters are set (Configured)
Quality of service is determined (Configured)
Connection request is made to Port 3225
or configured port
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Dynamic TCP Connections SLPv2
IP security for SLP determined
Enter FCIP discovery domain process
Advertise availability to SLP discovery domain
service agent
Locate FCIP entities in the discovery domain as
a SLP user agent
For each discovered entity follow same process
as non-dynamic method to establish connection
OPT-2T01
9899_06_2004_X
FCIP Special Frame
TCP Connection Is Established
Sending Side
First frame sent after TCP connection is established
Sending side waits for FSF echo (90 seconds)
Echo is match or non-match (Non-match terminates
TCP connection)
Creation of FCIP_LEP and FCIP_DE
Inform FC Entity of connection and usage flags
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FCIP Special Frame
TCP Connection Is Established
Receiving Side (Listening)
Listen for connections on WKN port 3225
or configured port
Checks database to allow connection
Checks security features
Wait for FSF frame (90 seconds)
Inspect FSF contents and send echo frame
Connection nonce
Destination FC fabric entity world wide name
Connection usage flags
Connection usage code
OPT-2T01
9899_06_2004_X
FCIP Special Frame Details
Used to exchange WWNs, entity pair identifiers,
TCP connection identifiers and to except or reject
connection
Identify what kind of traffic (SOFi3, SOFn3, EOF)
is intended; Not enforced
In conjunction with connection usage flags,
connection usage code help FCIP entity apply
proper QoS parameters for the connection
Adjustments to FSF with use of change bits can
be made when frame is echoed back
If two entities are trying to send FSF connection
frames simultaneously first to Rx echo wins
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
The first frame transmitted in each direction
is a special frame used to identify the peers
FCIP entities and to synchronize
Connection Nonce (random number)
proto
0x01
version
0x01
~proto
0xFE
~version
0xFE
Reserved
0x0000
Reserved
0xFFFF
Conn
Usage
flags
0x00
Connection
usage code
proto
0x01
version
0x01
~proto
0xFE
~version
0xFE
Flags
0x00
Frame
Len 0x12
~Flags
0x3F
~Frame
Len 0x3ED
CRC (Reserved in FCIP)
0x00-00-00-00
Source FC Fabric Entity WWN
(identify the fabric)
Source FC/FCIP Entity Identifier
Destination FC Fabric Entity WWN
Reserved
0x0000
Reserved
0xFFFF
0
1
2
3
4/5
6
7
8/9
10/11
12/13
14
15/16
17
Timestamp integer/fraction
~pFlags pFlags 0x00 0x00
1) Special Frame Sent
3) FCIP Tunnel Setup Complete
2)Special Frame echoed
Fibre Channel
IP WAN
Fibre Channel
FC
FC
I Am WWN1, This
Is my FC/FCIP
Identifier
Are You Fabric
WWN2?
Ok WWN1, I
Am WWN2
Lets Setup the
Connection
FCIP: Tunnel Setup as
Proposed in FCIP Draft
FCIP Device
OPT-2T01
9899_06_2004_X
pFlag Breakdown
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FCIP Header Format
FCIP header
used after FSF
exchange is
completed
Ones Compliment
for Synchronization
and Error Checking
OPT-2T01
9899_06_2004_X
Connection Options
TCP selective acknowledgement (SACK)
Per RFC 2883
TCP window scale option
Protection from sequence number wrap (PAWS)
TCP keepalives (KAD)
Flow control mapping between TCP and
Fibre Channel
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
FCIP Security Requirements (Per Draft)
To Support IP Network Security FCIP Entities MUST:
Implement cryptographically protected
authentication and cryptographic data integrity
keyed to the authentication process, and Implement
data confidentiality security features
FCIP utilizes the IPSec protocol suite to provide
data confidentiality and authentication services,
and IKE as the key management protocol
FCIP Security compliant implementations MUST
implement ESP and the IPsec protocol suite based
cryptographic authentication and data integrity [11],
as well as confidentiality using algorithms and
transforms as described in this section
OPT-2T01
9899_06_2004_X
FCIP Security Requirements
(Per Draft) (Cont.)
FCIP implementations MUST meet the secure key
management requirements of IPsec protocol suite
FCIP entities MUST implement replay protection
against ESP sequence number wrap
FCIP entities MUST use the results of IKE phase 1
negotiation for initiating an IKE phase 2 quick
mode exchange and establish new SAs
Note: An External Device May Be Used in
Conjunction with the FCIP Implementation to
Meet the Must Implement ESP Requirement
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Important FC and FCIP Timers
Resource Allocation Timeout Value (R_A_TOV)
Timeout value that determines how long a FC frame can
be in transit on the Fibre Channel network
This is a fabric wide value with a default value usually
at 120 sec on switch networks
Error Detect Timeout Value (E_D_TOV)
A value that times events and responses at the link level;
Errors at the link level will cause delays of these events
This value is defaulted to 10 sec and should be lower
then R_A_TOV; Again this is a fabric wide setting
Keep Alive Timer K_A_TOV
A value that is applied to TCP connection and is used
when no data is present
OPT-2T01
9899_06_2004_X
Time Stamps and Synchronization
Clock synchronization is required if timestamps
are used
Synchronized to FC services
Synchronized to IP NTP
Transit time through IP network is applied via a
timestamp Integer
If no timestamp value is available zero will be used
Fibre channel time values still apply across the ISL
link and are timed-out via lack of RDY coming back
End system devices such as HBA attached hosts
still require normal responses to timers end-to-end
(no spoofing)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Timestamps
TS are the responsibility of the FC entity
This allows transit through the FCIP entity to be
included in the measurement
This transit time should be well below R_A_TOV
OPT-2T01
9899_06_2004_X
Buffer Credits
Fibre channel buffer credit methods do not change
R_Rdys will be used to control flow coming from
FC switch on a per link basis
Buffer credit establishment is determined at FLOGI
Mechanisms to control flow of R_Rdys to
FC switch based on TCP/IP congestion is
per FCIP solution
FC switches do not require extended credit
methods
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Error Recovery
Errors on FC side of local B_Port are not
forwarded over the IP network; Issues such as
loss of sync or a FC encapsulation error will not
be set to the FC entity
Errors on IP side are handled by TCP and frame is
dropped if checksum is in error
OPT-2T01
9899_06_2004_X
Summary
FCIP is the standards approach to connect Fibre
Channel ISLs over TCP/IP LAN/WAN connections
State of draft wording will most likely stay as it is
worded today
Security, network delay and error recovery will be
biggest concerns
No shipping product today conforms to the
proposed FCIP draft
Cisco will have several platforms supporting
FCIP solutions
Presentation_ID.scr
INTERNET FIBRE
CHANNEL PROTOCOL
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
iFCP
FC-1
FC-0
TCP
IP
LINK
Gateway Region
PHY
iFCP
FC-1
FC-0
TCP
IP
LINK
Gateway Region
PHY
IP Network
FC-4
iFCP Protocol Model
iFCP replaces the transport layer of Fibre Channel (FC-2) with an IP
network but keeps the FC-4 mapping the existing Fibre Channel transport
services on TCP/IP
iFCP processes differentl y FC-4 frame images (applications), FC-2 frame
images (link service request), FC broadcast and iFCP control frames
Topology within the gateway regions are opaque to the IP network and other
gateway regions (they appear just like collection of N_Ports)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
IFCP
Gateway
Gateway Region
IFCP
Gateway
Gateway Region
iSNS
N_port-to-N_port session
N_port-to-N_port session
IP Network
iSNS
Queries
iSNS
Queries
IFCP
Gateway
Gateway Region
IFCP
Gateway
Gateway Region
iFCP Network Model: iSNS Role
An iFCP gateway cannot
operate without access to
an iSNS server
Client-Server architecture
iSNS functions:
Device Discovery and
fabric management
Emulation of the services
provided by the FC name
server and RSCN
Definition and management
of discovery domains
Definition and management
of logical fabrics
OPT-2T01
9899_06_2004_X
iFCP Protocol Description:
N_Ports Addresses Allocation
Two different schemes:
Address transparent mode (optional): The N_Port FC_IDs
are unique across the whole logical fabric
Address translation mode (mandatory): The N_Port
FC_IDs are unique only inside the gateway region the
N_Port belongs to
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Address Transparent Mode
All the gateways belonging to the same logical
fabric cooperate to assign addresses that are
unique across the gateway regions that form the
logical fabric
No need for address translation
Not scalable (max 239 gateways)
OPT-2T01
9899_06_2004_X
Address Translation Mode
iFCP gateways use aliases to map the local
representation of addresses of external
gateway regions to the real addresses outside
the gateway region (comparable to IP NAT)
Requires a rewrite of the FC_IDs in the FC
frame header and in the FC payload for some
ELS (i.e. ADISC)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
TCP/IP
IFCP
Gateway
IFCP
Gateway
F
C
N
S

R
e
q
u
e
s
t
iSNS
Give Me the Remote
Gateway
IP Address, N_Port ID,
N_Port WWN
2
)

i
S
N
S

q
u
e
r
y
/
r
e
p
l
y
1) The N_Port
Issues a
NS Query
Remote GW IP
Dest N_Port ID (y.y.y)
3) The Requesting GW Fills Up the
Address Translation Table
Local N_Port alias (z.z.z)
Dest N_Port WWN
FC_ID = x.x.x
FC_ID = y.y.y
F
C
N
S

R
e
p
l
y
4) The Gateway
Sends Back to the
N_Port the NS
Repl y (for FC_ID z.z.z)
Address Translation Mechanism
OPT-2T01
9899_06_2004_X
TCP/IP
IFCP
Gateway
IFCP
Gateway
P
l
o
g
iI d
id

z
.z
.z
s
i
d
x
.x
.x
1) The N_Port
I a
PLOGI to
D_ID z.z.z
Remote GW IP
Dest N_Port ID (y.y.y)
4) The Receiving GW
Rewrites the S_ID of
the Incoming Request
Local N_Port alias (z.z.z)
Dest N_Port WWN
FC_ID = x.x.x
FC_ID = y.y.y
2) The GW Makes a Table
Lookup Gets the Remote
GW IP Address (to Set Up
the iFCP Session) and the
Actual Dest N_Port ID ( to
Rewrite the D_ID) PLOGI did y.y.y
sid x.x.x
Remote GW IP
Dest N_Port ID (x.x.x)
3) The receiving GW Fills Up
Its Own Translation Table
Local N_Port alias (w.w.w)
Dest N_Port WWN
PLOGI did y.y.y
sid w.w.w
Address Translation Mechanism (Cont.)
In case of fabric reconfiguration all the address
translation tables need to be recalculated with a
consequent loss of every active login session
Presentation_ID.scr
ISNS AND SLP DISCOVERY
PROTOCOLS FOR THE IP-SAN
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Discovery Approach
Deploy and Interoperate in Three Stages:
1. Naming and static configuration
Configure both targets and initiators
Use SendTargets to reduce initiator config
2. SLPv2 for multicast and simple discovery
Configure targets
3. iSNS for centralized management
Configure central iSNS server
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Service Location Protocol (SLP)
Based on service location protocol v2 (RFC 2608)
Allows hosts to search for instances of a network
service they are interested in:
Example: printers
OPT-2T01
9899_06_2004_X
Basic SLP Discovery Requirements
Find targets by initiators worldwide unique
identifier
Tell me which targets you have that I should see
Find targets by targets worldwide unique identifier
Where is target iscsi.com.acme.foo?
Propagate attributes needed before connecting
Boot information, authentication information
Scaling requirements
Zero-configuration, no servers in small environments
Reduce or eliminate multicast in medium environments
Interoperate with LDAP/iSNS in large environments
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Service Location Protocol (SLP)
SAService Agent; Services register with SA
UAUser Agent; Queries SA or DA for registered
services
DADirectory Agent; Proxies for a set of SAs
SA
DA
UA
register query/
response
query/response
services
services
services
register
Three Components, Two of Which Run
in Our Storage Router
OPT-2T01
9899_06_2004_X
Service Location Protocol for IP Storage
Service Agent (SA)
Advertises services
Services have attributes
User Agent (UA)
Finds services
Zero configuration
Directory Agent (DA)
Optional
Propagate service adverts
SLP Protocol
UDP or TCP
Minimize multicast
device
host
iSCSI
Target
SLP
UA
SLP
DA
iSCSI
Initiator
IP
TCP/IP
Management Code
SLP
SA
TCP/IP
Management Code
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Implementing SLP for iSCSI
Targets implement a service agent
Answer multicast requests or register with DA
Initiators implement a user agent
Use multicast or DA to locate targets
Devices containing targets register:
The canonical target or individual targets
Attributes of targets
Register target at each of its addresses
OPT-2T01
9899_06_2004_X
SLP Summary
Serverless discovery of targets
Optional, generic DA to scale services
Zero-configuration of hosts
SLP makes careful use of multicast
Access list and attribute propagation
Optional message authentication
Available open source implementations
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
What Is iSNS
iSNS Facilitates Scalable Configuration and
Management of iSCSI, iFCP and Fibre Channel (FCP)
Storage Devices in an IP Network, By Providing a Set
of Services comparable to that Available in Fibre
Channel Networks
http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-22.txt
OPT-2T01
9899_06_2004_X
iSNS Functions
There Are Four Main Functions of the iSNS:
1. A name server providing storage resource
discovery
2. Discovery Domain (DD) and login control service
3. State change notification service
4. Open Mapping of Fibre Channel and iSCSI devices
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Basic: How iSNS Works
IP Network
Fibre Channel SAN
iSCSI & iSNS Clients
iSCSI Clients Register with iSNS Server,
Done By Adding iSNS IP Address to
iSCSI Application Driver
1
1
1
1
iSCSI Targets Register with iSNS Server
2
2
iSNS Clients Query iSNS Server for
Storage Location and Name
3
3
iSCSI Client then Selects and
Logs into iSCSI Target Using
Information from iSNS Server
4
FC FC
FC FC
OPT-2T01
9899_06_2004_X
Internet Storage Name Service (iSNS)
iSNS server functions:
Allows an iSNS client to register/deregister/query with the
iSNS server
Provides centralized management for enforcing access
control of targets from specific initiators
Provides a state-change notification mechanism for
registered iSNS clients on the change of status of other
iSNS clients
Similar to the functionally provided by the FC name
Server, Zone Server and the RSCN mechanism
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSNS Components
iSNS protocol (iSNSP)
A flexible and lightweight protocol that specifies how iSNS
clients and servers communicate
Discovery Domain (DD)
A grouping of storage devices much like a zone in the FCP;
discovery domains help in control and manage logins and
services available to the clients in the domain; Based on
the FC-GS standard for fiber channel; Items like default
domain are used
Discovery Domain Set (DDS)
A group of one or more discovery domains; A method to
store sets of domains within the iSNS database; Multiple
DDSs can be active at one time, unlike zonesets in FCP
where only one can be active at a time
OPT-2T01
9899_06_2004_X
iSNS Components
iSNS client
The iSNS client is located within storage system and talk to the
iSNS server using the iSNSP within its configured device domain;
client can belong to one or more DDs; iSNS client registers its
attributes with the iSNS server and recei ves notices of changes
within the domain
iSNS database
The iSNS database is the information repository for the iSNS
server; it maintains information about iSNS clients attributes; a
directory-enabled implementation of iSNS may store client
attributes in an LDAP directory infrastructure
iSNS server
iSNS servers respond to iSNS protocol queries and requests,
and initiate iSNS protocol state change notifications; properly
authenticated information submitted by a registration request is
stored in an iSNS database; listens on port 3205
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSNS SCN (State Change Notifications)
iSNS clients who wish to receive SCN have to
explicitly register with iSNS server the events
in order to receive the notifications
Initiator/target/object with add/remove event or
to/from discovery domain are the events that
can be registered
iSNS servers generate SCN when either the state
of any target device changes or when the target
device itself requests an SCN to be generated
using SCN event message; iSNS listens to FCNS
to registration/deregistration
OPT-2T01
9899_06_2004_X
SCN Types
Regular registrations
This type of SCN is used within a DD; The discovery
domain will control where the SCN message will go
Management registrations
Used by control nodes and can travel outside the DD
from which they came
Can be TCP or UDP messaging
(Most implementations only using TCP for now)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Services Provided by the Discovery Domain
Login control
Authorization and control policies for storage targets can
be maintained by iSNS servers only allowing authorized
devices to access the targets
Control of what target portals are accessible within the
discovery domain
Fibre Channel to iSCSI device mapping
iSNS database learns and stores naming and discovery
information about FC storage devices discovery on the
iSCSI Gateway and iSCSI devices in the IP network; This
database can then be available by FC and IP iSNS clients
OPT-2T01
9899_06_2004_X
High Availability of iSNS Servers
Can use SLP to discovery other iSNS servers
Database transfers between servers using
iSNSP or SNMP
Heartbeat mechanism used between active
and backup iSNS servers
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Internet Storage Name
Service (iSNS) for iSCSI
The iSNS protocol (iSNSP) provides:
A mechanism for iSCSI clients to discover other iSCSI
targets/initiators
Enforce access control
Notifications from an iSNS server on changes to the
status of a logged in iSCSI device
Provide ability to discovery iSCSI target on different
IP network
iSCSI target discovery can happen through:
Static configuration of initiator
iSCSI sendTargets command
Name server/directory server (via iSNS)
OPT-2T01
9899_06_2004_X
iSNSP Header
iSNSP VersionC the Current Version is 0x0001; All Other Values Are RESERVED
iSNSP Function IDDefines the Type of iSNS Message and the Operation to Be Executed; iSNSP
PDU LengthSpecifies the Length of the PDU PAYLOAD Field in bytes; The PDU Payload Contains
Attributes for the Operation
iSNSP FlagsIndicates Additional Information About the Message and the Type of Network Entity
That Generated the Message
iSNSP Transaction IDMUST Be Set to a Unique Value for Each Concurrentl y Outstanding
Request Message; Replies MUST Use the same TRANSACTION ID Value as the Associated iSNS
Request Message
iSNSP Sequence IDThe SEQUENCE ID Has a Unique Value for Each PDU Within a Single Transaction
iSNSP PDU PayloadThe iSNSP PDU PAYLOAD Is Variable Length and Contains Attributes Used for
Registration and Query Operations
Authentication BlockFor iSNS Multicast and Broadcast Messages, the iSNSP Provides
Authentication Capability; The iSNS Authentication Block Is Identical in Format to the SLP
Authentication Block
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSNSP Commands for iSCSI
The Following Are iSNSP Commands Messages Used in
Support of iSCSI:
OPT-2T01
9899_06_2004_X
iSNSP Responses for iSCSI
The Following Are iSNSP Response messages Used in
Support of iSCSI:
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSNS Queries for iSCSI
iSNS clients can perform two types of queries:
Device attribute query: iSNS server responds with requested
attributes of one or more iSNS clients
The iSNS server converts the recei ved query to a FC name
server query in the SAN
FC name server will ensure that the resultant set is filtered
based on zones
The iSNS server translates each entry returned by the FC
name server to the corresponding iSNS clients
Apply filters based on iSCSI access control by removing all
statically configured virtual targets the query initiator is not
allowed to access
Device get next query: Allows an iterative query of the iSNS
servers iSNS client database
OPT-2T01
9899_06_2004_X
Return Information from iSNS iSCSI Query
iSCSi Name
IP Address of
Portal to Log to
and Ask for This
target
Name of Port on
the IP Gateway
Entity
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSNS for iFCP
Will work much the same manor as iSCSI
just will require other related attributes to
be registered and queried
Is required for iFCP
Functions much like domain name server
and domain ID manager
Needs to be highly available service for
FC devices
OPT-2T01
9899_06_2004_X
iSNSP Commands for iFCP
The Following Are iSNSP Commands Messages Used
in Support of iFCP:
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
iSNSP Responses for iFCP
The Following Are iSNSP Response Messages Used
in Support of iFCP:
OPT-2T01
9899_06_2004_X
SLP and iSNS
SLP used for target discovery
No configuration required for the simplest networks
Small footprint; no servers required
Just enough discovery for small-to-medium networks
Device-centric access control model
iSNS adds storage management capabilities
Active monitoring of initiators and targets
Event propagation
Public key distribution
Centralized access control model
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Using Both SLP and iSNS
Initiators can use both SLP and iSNS to
discover targets
Targets should use SLP only if not
configured for iSNS
Gateways or proxies may provide local SLP
discovery of remote iSNS devices
TECHNICAL TOOLS AND SKILLS
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Storage Networking Toolbox
Test tools for Fibre Channel and IP
Host based tools
Network component serviceability tools
Software debug tools
Knowledge
OPT-2T01
9899_06_2004_X
Fibre Channel Analyzers
Most units are based on dedicated hardware, and might be
supplied with software tools for performance base lining
Very expensi ve
Oriented to protocol conformance testing
Requires 2 GBICs interfaces to be implemented
Monitoring units might have a retiming mode, to cleanup
some of the timing problems on a link, and to separate
them from the real problem at layer 1
Statistical software can run on these type units
Collecting statistics on the status of the line, or other parameters
(number of bits, exchanges)
Sharing is still a dream in most cases, it is complex to
share in the field, so in most cases the portable versions
are the most suitable
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Fibre Channel Analyzers
Snooping GBICs or fiber taps; allow to monitor
without service interruption; very important for
Fibre Channel work in the field
Traffic probes; used to remotely monitor the state
of a network without service interruption
Trace viewers (free from the vendor websites)
Each vendor has its own PC viewer and must be used
with each capture tool; these can be found at each of
their websites
OPT-2T01
9899_06_2004_X
FC Test Vendors
Leaders in dedicated hardware tools:
Finisar (www.finisar.com)
Xyratex (www.xyratex.com)
Aglient (www.agilent.com)
I-Tech (www.I-tech.com)
Ancot (http://www.ancot.com/)
Spirent/Netcom systems
(www.netcomsystems.com)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
SCSI Host-Based Testing
I/O meter
http://developer.intel.com/design/servers/devtools/iometer/
I/O zone
http://www.iozone.org/
SCSI tools
http://scsitools.com/
Xyratex disk basher
http://www.xyratex.com/
Freeshare or software tools for SCSI
and I/O analysis, tools for disk manufacturing
www.ethereal.com
www.wildpackets.com
OPT-2T01
9899_06_2004_X
Windows Tools
iSCSI Driver debug helpers
Windows debug utilities
http://www.osr.com/resources_downloads.shtml
http://www.sysinternals.com/
Detail uses of O/S disk administrator to verify and
check health of target devices
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
IP: GiGE
GiGE testers $$$
Agilent
Sniffer
Fluke
Finisar/Shomiti
iSCSI decodes just becoming available on most tools
All your IP tools
IP Ping, trace, etc.
Fibre Channel ping available at
http://www.teracloud.com/utilities.html
OPT-2T01
9899_06_2004_X
iSCSI Decoding
Software only analyzers like Ethereal
(www.ethereal.com)
Hardware analyzers
Can use monitor command on Cisco switches to
span the iSCSI GiGE port to a 10/100
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Available Certifications
SNIA (Storage Networking Industry Association)
Level 1Fibre Channel storage networking professional
Level 2Fibre Channel storage networking practitioner
iSCSI training available at many education sources
Infinity I/O, medusa, solution technology, others
Other certifications that are vendor specific
ARCHITECTURAL DESIGN OF
STORAGE AREA NETWORKS
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Section Agenda
Introduction
Hierarchy
Modularity
Architecture Examples
INTRODUCTION
488 488 2004 Cisco Systems, Inc. All rights reserved.
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Hierarchy, Modularity and
Limited Failure Domains
Why Do This? (Benefits Summary):
Scalable architecture
Improved performance
Manage change
Improve service
Improved security
Simplified management and troubleshooting
Reduced cost of ownership
OPT-2T01
9899_06_2004_X
What Problem Are We Solving?
Have consistently high performance
Are reliable, scaleable, and manageable
Are secure and cost-efficient
Are service and solution enabling
Adapt to changing requirements
Applications Must Be Available and Perform Well
Networks that Deliver on this Requirement:
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Network Design Goals
Architecture Provides:
Performance
Reliability, availability,
and scalabilityRAS
Cost efficiencies
Security
A base to enable services
and solutions
To Meet Mission-Critical
Business Objectives,
Applications Need to
Be Consistently Up,
Available, and
High-Performing
OPT-2T01
9899_06_2004_X
Architecture:
Hierarchy, Modularity, and Domains
Functionally Divides the Problem
Create Manageable Building Blocks
Limits Scope of Potential Failures
Hierarchy
Modularity
Domains
Fundamentally, We Break the Network Design
Process into Manageable Blocks so that the
Network will Function within the Performance
and Scale Limits of Applications, Protocols
and Network Services
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Focus of This
Discussion
What Does This Mean?
We Build
Networks that
Have Structure:
Distribution
Backbone
WAN Internet PSTN
Application
Servers
Building Blocks
Core
Enterprise
Storage
Access
OPT-2T01
9899_06_2004_X
Applying Design Principles to Storage
Hierarchy
Predictable performance
Scaleable design
Fault isolation
Modularity
Cost-effective
Repeatable
Domain
Reliability
Security
Shared Storage
Unified
Storage
Mgmt
Core Core
Presentation_ID.scr
HIERARCHY
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Hierarchy: Physical and Logical
Physical hierarchy
Predictable
performance
Scaleable design
Fault isolation
High availability
Logical hierarchy
Virtual SANs
Zoning
Enhances physical
hierarchy
Physical Architecture
Logical Architecture Logical Architecture
Virtual SAN A Virtual SAN B
H1 H2
D1 D2 H3
Zone 1 Zone 2
Zone 3
H1 H2
D1 D7 H7
Zone 1 Zone 2
Zone 4
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Hierarchy: Physical
Consolidated Storage Network
Cost-effective solution
Benefits of consolidation
Limited scalability
Small to medium business
Expansion can be
disruptive
Single fault redundancy
Double fault would likely
result in isolation
iSCSI iSCSI
OPT-2T01
9899_06_2004_X
Hierarchy: Physical
Collapsed Core Architecture
Collapsed core
High performance
Multiple unequal paths
Better scalability
Medium to large
enterprise
ISLs can limit scalability
Redundant
Mesh topology
Network survives some
double faults
iSCSI iSCSI
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Hierarchy: Physical
Core Edge Architecture
CoreEdge
High performance
Load balancing
Consistent hop count
Good scalability
Large to very large enterprise
Non-disruptive expansion
Better fault tolerance
Improved fault isolation
Single fault within
layer okay
iSCSI iSCSI
OPT-2T01
9899_06_2004_X
Hierarchy: Physical
Oversubscription
To be expected in storage networks
Typically lower factors than we see in LANs
Architecture should be flexible to accommodate
differing requirements for various hosts and
storage subsystems
Bandwidth can be modified non-disruptively by
using port channels between switches
Take into account any inherent over subscription
in networking hardware
Use actual anticipated throughput rather than link
speed for calculating bandwidth requirements
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Hierarchy: Physical
Inter-Switch Links
Inter-Switch LinkISL
Physical FC link between
two fabric switches
forming a trunk
Utilized for FC services
and data traffic
Port Channel
Multiple FC ISLs combined to
form a single aggregated trunk
All links in a Port Channel
must be directly connected
to the same two switches
Individual link state
changes do not cause ISL
trunk state changes
ISL
Port
Channel
iSCSI iSCSI
OPT-2T01
9899_06_2004_X
Hierarchy: Physical
Scalability
Oversubscription
Higher OS acceptable
for some hosts
Lower OS for High
performance hosts
and storage devices
Consider impact of multi-
path load balancing
Determine acceptable
worst case in various
failure scenarios
Can be non-disruptively
changed by adding/
removing links to port
channels
8:1
OS
1:1
OS
Core
3:1
OS
4x2Gb
ISL
8x2Gb
ISL
15:1
OS
iSCSI iSCSI
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Hierarchy: Logical
Virtual SANs
VSANs provide a means to build a logical structure
on top of a physical SAN
Similar to how VLANs are used to scale ethernet
networks VSANs help scale Fibre Channel networks
Topology changes are isolated within the VSAN
therefore adds, moves, and changes are not
disruptive to other VSANs
VSANs can be utilized to establish administrative
domains
Zoning provides an additional access control
mechanism within each VSAN
OPT-2T01
9899_06_2004_X
Hierarchy: Logical
Logical Architecture
Virtual SANs
Similar to Ethernet VLANs
except no inter-VSAN flows
Enhanced ISL provides
VSAN trunking (EISL)
Complimentary to port
channel
Services scalability
Independent Fibre Channel
services for each VSAN
Zoning is per VSAN
Failure domain
Faults contained within
VSAN
iSCSI iSCSI
EISL
Port
Channel
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Hierarchy: Logical
Maximizing VSAN Architecture
Isolate multiple paths
into separate VSANs
Independent FC services
per VSAN
Provides complete traffic
isolation between
redundant paths
Each VSAN converges
independently for faster
recovery and improved
fault isolation
iSCSI iSCSI
OPT-2T01
9899_06_2004_X
Hierarchy: Combining Physical and Logical
iSCSI iSCSI
Fabric A provides one
set of links and Fibre
Channel services
A
Fabric B provides an
independent set of
links and services
B
Presentation_ID.scr
MODULARITY
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Modularity: Key Elements
The ability to scale the network while maintaining
consistent performance
Building block approach breaks network into
smaller chunks that are easier to understand,
replicate, and deploy
Changes and additions can be made
non-disruptively
Provides consistent and limited failure domains
Modularity can also define administrative
boundaries
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Modularity: Building Blocks
iSCSI iSCSI
Fiber
Channel
Core
Functional Building Blocks
Provide Scalability with
Deterministic Performance
Storage
Modules
Application
Modules
OPT-2T01
9899_06_2004_X
Modularity: Utilizing VSANs
Adds, moves, and changes contained within a
VSAN are non-disruptive to other VSANs
Using VSANs facilitates application modeling
and testing
Per VSAN statistics
Per VSAN traffic engineering
Per VSAN administration (if desired)
Eliminates costs associated with separate
physical fabrics
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Modularity: Benefits of VSANs
Overlay isolated virtual fabrics on
same physical infrastructure
Each VSAN contains zones and
separate (replicated) fabric services
VSAN membership determined by port
VSANs for availability
Isolate virtual fabrics from fabric-wide
faults/reconfigurations
Security
Complete hardware isolation
Scalability
Replicated fabric services
Thousands of VSANs per storage
network
Management
Roll Based Access ControlRBAC
Provides administrati ve boundaries
Department/
Customer A
Shared Storage
Department/
Customer B
VSAN-Enabled
Fabric
Mgmt
VSAN
VSAN
Trunks
OPT-2T01
9899_06_2004_X
Modularity:
Storage Intelligence and VSANs
Dept 3
VSAN
Dept 2
VSAN
Dept 1
VSAN
VSANs created to provide
isolation of fabric-wide
services.
Virtualization allows
physical storage to be in
its own VSANs, separate
from the host VSANs.
VSANs provide
Secure isolation of physical storage
Easier configuration
Dynamic configuration of fabrics
Role-based access control
Data Center VSANs
Virtualization
Presentation_ID.scr
ARCHITECTURE EXAMPLES
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Architecture: iSCSI
Scalability
Less expensive
alternati ve for host not
requiring 2Gbps
Recommend separate NIC
TCP Offload EngineTOE
Consider actual
throughput requirements
for scalability
Host Services
Appears as normal HBA
Compatible with host
based storage utilities
multi-path, load balance,
mapping, etc.
TOE
File System File System
Adapter
Driver
Adapter
Driver
TCP/IP
Stack
TCP/IP
Stack
NIC
Driver
NIC
Driver
Block Device Block Device
SCSI Generic SCSI Generic
iSCSI
Driver
iSCSI
Driver
Applications
Network File
System
Network File
System
NIC
Driver
NIC
Driver
TCP/IP
Stack
TCP/IP
Stack
iSCSI
Driver
iSCSI
Driver
TCP/IP
Stack
TCP/IP
Stack
NIC
Driver
NIC
Driver
SCSI Adapter SCSI Adapter
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Architecture: iSCSI High-Availability
Redundant connections to hosts or servers
High-availability iSCSI services
Redundant paths to backend FC SAN
Redundant iSCSI to
Fibre Channel Connections
and Services
Storage Array with
Redundant Controller
Ports
Multiple
Ethernet
Switches
Host with Multiple(iSCSI)
NICs and Multipathing
Software Installed
iSCSI Driver
Multipathing
Application Application
OPT-2T01
9899_06_2004_X
Architecture: iSCSI Authentication
SCSI routing service
passes username
and MD5-hashed
password from
initiators to AAA server
AAA authentication list
used to determine
which service(s) to use
for authentication
iSCSI
Hosts
(Initiators)
RADIUS TACACS+
RADIUS
Server
Local
User1/pwd1
User2/pwd2
/
CHAP
iSCSI Services
TACACS
+ Server
User1/pwd1
User2/pwd2
/
User1/pwd1
User2/pwd2
/
Authentication Services List
AAA Authentication Services AAA Authentication Services
IP Network FC Fabric
User1/pwd1 User1/pwd1
SCSI Routing Instance
iSCSI
Storage
(Targets)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Architecture: iSCSI Topology
iSCSI Best Practices
Isolate IP storage network
behind application hosts
with VLANs
Minimized potential for
bandwidth contention
Map VLANs to VSANs
for manageability
Dedicated ethernet
interfaces on host for
attachment to storage
network
iSCSI-enabled
Hosts
Storage
Pool
iSCSI
Services
Ethernet
Switches
Clients
Front-Side IP
Network
Front-Side IP
Network
FC Attached
Hosts with
HBAs
iSCSI iSCSI iSCSI iSCSI
IP
Storage
Network
FC
Fabric
OPT-2T01
9899_06_2004_X
Architecture: SAN Extension Technology
IP
WAN
Technology Choice Requires Matching Storage Application
Requirements with Service Availability, Cost, Throughput, and Latency
FCIP
FCIP
CWDM
DWDM
SONET/SDH
FC
FC
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Both fabrics remain connected if one of the paths fails
Use of portchannel prevents state change on link failure
Architecture:
High Availability for SAN Extension: FC
Utilize disparate paths and portchannel for high availability
Utilize VSANs to limit the failure domain in the event of lost
connectivity
Fabric
A
Fabric
A
Fabric
B
Fabric
B
PortChannel
FC FC
CWDM
DWDM
SONET/SDH
OPT-2T01
9899_06_2004_X
Architecture:
High Availability for SAN Extension: FCIP
Utilize disparate paths and portchannel for high availability
Utilize VSANs to limit the failure domain in the event of lost
connectivity
Recommend not using etherchannels
Fabric
A
Fabric
A
Fabric
B
Fabric
B
PortChannel
PortChannel
IP
WAN
IP
WAN
FCIP FCIP
Both fabrics remain connected if one of the paths fails
Use of portchannel prevents state change on link failure
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Architecture:
Legacy Storage Implementation
Storage is captive
behind applications
Inefficient
allocation of
storage resources
Multiple
administrative
domains
Backbone
Application
Servers
Captive Storage
Blocks
LAN Core
Campus Clients Remote Clients Internet Clients
SAN
Islands
OPT-2T01
9899_06_2004_X
Architecture:
Factors for Determining Architecture
Current size and anticipated growth for both
application servers and storage elements
Baseline performance requirements for servers
and storage
Business continuance requirementsSAN
extension
Administrative domains
Migration plans
Interoperability considerations
Costs
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Architecture: Collapsed Core Architecture
Servers and storage
elements connected
to collapsed core
Some scalability
especially with iSCSI
Redundant paths
Achieves
economical
storage consolidation
VSANs can add
scalability and
management benefits
Application
Servers
iSCSI
Shared
Storage
Unified
Storage
Mgmt
OPT-2T01
9899_06_2004_X
Architecture: Large Scale Architecture
Application servers
connect to edge
switches
Storage devices
connect to edge
switches
Highly scalable
Highly redundant
Highly modular
Multiple equal paths
VSANs limit the size
of any one SAN
Application
Servers
Shared
Storage
Unified
Storage
Mgmt
iSCSI
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Network Design Goals
Performance
Planned hierarchy, managed oversubscription, and modular design
Reliability, Availability, and ScalabilityRAS
Limited failure domains, leveraged VSANs, and modular design
Cost efficiencies
Consolidated storage, central management, and leveraged resources
Security
Limited domains, RBAC management, and consistent architecture
A base to enable services and solutions
Business continuance and disaster recovery
Management of heterogeneous storage elements
Ubiquitous access to storage from anywhere
Infrastructure for storage virtualization
Architecture Summary:
OPT-2T01
9899_06_2004_X
Architecture: End-to-End SAN Architecture
Intelligent Workgroup
Storage Networks
Highl y Scalable
Storage Networks
Optical Network
Resilient Optical
Transport Networks
Synchronous ReplicationOptical (FCIP/FC)
SONET Network
Multiprotocol/Multiservice
SONET Network
Asynchronous ReplicationFCIP over SONET
FCIP
Remote
Storage
Access
FC FC
FC FC FC
FC
FC
FC
FC FC
FC FC FC
FC FC
FC FC FC
iSCSI-
Enabled
Storage
Network
Ethernet
Switches
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
Presentation_ID.scr
Q & A
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Complete Your Online Session Evaluation!
WHAT: Complete an online session evaluation
and your name will be entered into a
daily drawing
WHY: Win fabulous prizes! Give us your feedback!
WHERE: Go to the Internet stations located
throughout the Convention Center
HOW: Winners will be posted on the onsite
Networkers Website; four winners per day
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
EXTRAS
OPT-2T01
9899_06_2004_X
Presentation_ID.scr
FC LOOP OPERATIONS
OPT-2T01
9899_06_2004_X
OPT-2T01
9899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
IDLE
IDLE
IDLE
IDLE
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. The Loop is initially filled with IDLES
2. Each port is in the monitoring state
3. Because of no activity CFW = Idle
4. Rx IDLES are replaced with CFW
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(01)
IDLE
IDLE
IDLE
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. Port_01 begins to arbitrate for access
to the Loop
2. Port_01 changes its CFW from IDLE to
ARB(01)
3. Port_01 transmits ARB(01) when a fill
word is required
OPT-2T01
9899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(01)
ARB(01)
IDLE
ARB(01)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. ARB(01) is Rx by the next port and
updates its CFW to ARB(01)
When a Port Discards Rx Fill Words and Transmits the CFW this Allows the Port to
Compensate for Clock Differences Between Rx Data Stream and Tx Data Stream
2. Whenever a fill word is required
ARB(01) is used; With no other acti vity
on the loop ARB(01) is sent
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(F0)
OPN
ARB(01)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. When Port_01 receives its own
ARB(01) it wins arbitration
2. Port_01 sends on OPN to open a loop
circuit and changes its CFW to ARB(F0)
3. Port_01 discards any Rxed ARB(x)
OPT-2T01
9899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
IDLE
ARB(F0)
ARB(F0)
ARB(F0)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. As each port Rxs the ARB(F0) it updates
its CFW to ARB(F0)
2. Assuming that no other port is arbitrating,
ARB(F0) travel the complete loop
3. When ARB(F0) is Rxed by Port_01 the CFW
in Port_01 is changed to IDLE
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
IDLE
IDLE
ARB(F0)
IDLE
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. Each port receives the IDLE and updates
its CFW to IDLE
Discarding the Receiving Arb(x) Prevents Any
Other Port from Winning Arbitration
2. Assuming the no other port is arbitrating
and the IDLES travel the complete loop
3. As long as Port_01 owns the loop it
discards any Rxed IDLE or ARB(x) and
continues to send its CFW when
necessary
OPT-2T01
9899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(01)
IDLE
IDLE
ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. Port_01 begins arbitrating for
access to the loop; Done by
replacing IDLE and ARB(x) with
ARB(01)
2. Port_B2 also begins arbitrating for
the loop; It replaces Idle and ARB(x)
with ARB(B2)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(01)
ARB(01)
ARB(B2)
ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. The ARB(01) gets to Port_2A which
updates its CFW with ARB(01) and
transmits this when the CFW is
needed
2. The ARB(B2) also travels to Port_EF
which updates its CFW with ARB(B2)
OPT-2T01
9899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(01)
ARB(01)
ARB(B2)
ARB(01)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. When Port_B2 recei ves ARB(01) it
changes its CFW to ARB(01)
because of 01 has higher
priority(Lower AL_PA wins)
Because Port_B2s ARB(B2) Is Replaced with
ARB(01) It Will Not Win Arbitration at this Time
2. When Port_01 recei ves ARB(B2) it
is replaced with ARB(01)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(F0)
and OPN
ARB(01)
ARB(01)
ARB(01)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. ARB(01) is Rx by Port_01 and wins
arbitration
2. Port_01 then opens the loop circuit
and updates its CFW with ARB(F0)
when a fill word is required
3. Port_B2 is still arbitrating but is lower
priority
OPT-2T01
9899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(F0)
ARB(F0)
ARB(01)
ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. Port_2A receives ARB(F0) and
updates the CFW to F0
2. Port_B2 replaces the lower-priority
ARB(F0) and transmits ARB(B2)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(F0)
ARB(F0)
ARB(B2)
ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. Port_EF updates its CFW to ARB(B2) and
transmits on to Port_01
2. Port_01 Txs ARB(F0)
When Port_01 Relinquishes Control of the Loop It
Changes Its CFW to ARB(B2) Allowing Port_B2 to Win
3. Port_B2 continues to replace F0 with B2;
Port_01 discards all Rxed ARB(x)
ordered sets
OPT-2T01
9899_06_2004_X
Lower Priority Port ARB
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
IDLE
IDLE
IDLE
ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. Port_B2 begins to arbitrate for the
loop by changing CFW to B2
2. Each Rxed IDLE and lower-priority
ARB(x) is discarded by Port_B2 and
the ARB(B2) is substituted in its
place
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
IDLE
IDLE
ARB(B2)
ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. ARB(B2) propagates around the
loop to Port_EF
2. Port_EF changes its CFW to ARB(B2)
and Txs the ARB(B2) whenever a fill
word is needed
OPT-2T01
9899_06_2004_X
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(B2)
IDLE
ARB(B2)
ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. The ARB(B2) propagates around the
loop to Port_01
2. Port_01 changes its CFW to ARB(B2)
and Txs ARB(B2) whenever a fill
word is needed
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(01)
ARB(B2)
ARB(B2)
ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. Port_01 begins arbitrating after a single
ARB(B2) has passed
3. The single ARB(B2) travels around the loop to
Port_2A. Port_2A passes the ARB(B2)
2. Port_01 has higher priority than Port_B2 and
discards ARB(B2) and replaces it with ARB(01)
4. When ARB(01) is Rxed at Port_2A its CFW is
changed from B2 to 01
ARB(01)
XX
OPT-2T01
9899_06_2004_X
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(01)
ARB(B2) ARB(F0)
and OPN
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. The single ARB(B2) is Rxed by
Port_B2 which wins arbitration and
begins to discard any Rxd ARB(x)
2. Port_B2 changes its CFW to ARB(F0)
ARB(01)
ARB(B2)
Presentation_ID.scr
OPT-2T01
9899_06_2004_X
RX TX
Port
RX TX
Port
R
X
T
X
P
o
r
t
R
X
P
o
r
t
T
X
ARB(01)
ARB(01)
ARB(F0)
ARB(F0)
AL_PA 2A
AL_PA EF
AL_PA B2
AL_PA 01
1. Port_EF changes its CFW to ARB(F0) and
sends it on to Port_01
2. Port_01 substitutes ARB(01 for every ARB(F0) it
receives
3. Port_B2 discards the ARB(01) and sends ARB(F0)
as its fill word
4. When Port_B2 relinquishes the loop, it will
change its CFW to ARB(01) and allow Port_01 to
win the loop

Storage Area Network Basics

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Storage Area Network Basics

Uploaded by

Copyright:

Available Formats

2004 Cisco Systems, Inc. All rights reserved. Printed in USA.

48 48 48 2004 Cisco Systems, Inc. All rights reserved.

You might also like