Brocade Product Training: CFP264 Brocade 4 Gbit/sec Accelerated BCFP Instructor-Led Module 8 Trunking and Distance

2006 Brocade Communications Systems, Incorporated.
Revision CFP264 ILT 0806

Page 8-1
1
Brocade Education Services
Brocade
Product Training
CFP264 ILT 0806
CFP264
Brocade 4 Gbit/sec Accelerated BCFP
Instructor-Led Module 8
Trunking and Distance
Page 8-2
2
CFP264 ILT 0806
Following this module and associated lab, an attendee should
be able to
Describe, create, validate, and differentiate behavior in 2
and 4 Gbit/sec Brocade ISL Trunking
Describe and differentiate between available distance
extension options
Please review supplementary appendix material
Objectives
Page 8-3
3
CFP264 ILT 0806
What is Brocade ISL Trunking?
Trunking combines two or more physical ISLs into a single logical link
Trunking Goals
Reduces individual ISL congestion
Forms a fault-tolerant high bandwidth logical ISL (called a trunk or trunk
group) that withstands the failure of individual ISLs
Trunk group characteristics:
Frames are multiplexed across ISLs in the trunk group
One port in the trunk group represents the link in the routing data base
ASICs preserve in-order delivery
Bandwidth of individual links is aggregated
One logical link consisting of 2 or more ISLs
.
.
.
The routing data base determines how frames are routed from input port to
output port when going to the next destination. Fabric Shortest Path First
(FSPF) puts available equal cost routes in the routing data base. One
output port in the trunk group is put into the routing data base. When a
communication between two end devices in a fabric are assigned a route
through a trunk the ASIC of the assigned trunk group port will be the same
ASIC as all ports in the trunk group. This ASIC will multiplex frames across
ISLs in the trunk group and maintain in-order delivery. The ASIC will send a
frame down each link to determine the links latency, these individual link
latency calculations will be used to maintain in-order delivery.
Page 8-4
4
CFP264 ILT 0806
Trunking Requirements
Trunking license required for all switches participating in trunking
Available when the license is installed and ports are reinitialized
(switchdisable/enable or portdisable/enable on all ports)
Trunking is enabled by default; if previously disabled, it must be re-
enabled (portcfgtrunkport) on the trunk ports
Trunk ports must operate at a common speed (2 or 4 Gbit/sec)
Trunk ports must originate and end in a port group on the same ASIC
When trunking criteria is met the trunk forms automatically
Example trunk: 1 logical link consisting of 3 ISLs
SilkWorm 4100
SilkWorm 4100
To use trunking, you must first install the Brocade trunking license. Trunking is enabled
automatically when the trunking license is activated and ports are reinitialized (after
installing the license, you enter the switchdisable and switchenable commands).
Trunks are easily managed using either Fabric OS CLI commands or Web Tools.
Switches are shipped with trunking enabled.
Trunk port-groups are ASIC specific and will be discussed in more detail in subsequent
slides
Bloom port groups have historically been called quads. Bloom port groups include:
ports 0-3; 4-7; 8-11; and so on. Bloom ASICs include:
The Bloom I ASIC is the foundation for the following Brocade SilkWorm models:
SilkWorm 3000, 3200, 3800, 3900, and 12000.
The Bloom II ASIC is the foundation for the following Brocade SilkWorm switches:
Silkworm 3250, 3850, and 24000.
Condor port groups include: ports 0-7; 8-15; and so on
The Condor ASIC is the foundation for the following Brocade SilkWorm models:
Silkworm 4100, 4900, 7500, 48000, and FR4-18i.
GoldenEye port groups include: ports 0-3; 4-7; 8-11; and so on
The GoldenEye ASIC is the foundation for the following Brocade SilkWorm switch:
SilkWorm 200E.
Additional ASIC specific and advanced trunking criteria includes:
Interoperability is not supported. Use the interopmode 0 command to disable
interoperability capability with other vendor switches.
The port ISL mode must be disabled (use the portcfgislmode command). This
parameter is related to using 3
rd
party gateways to extend fabric distances. This will
be further discussed later in this course.
See the Brocade Fabric OS Administrators guide for additional information.
Page 8-5
5
CFP264 ILT 0806
When trunking criteria
1
is met, Bloom trunking implementation:
Automatically aggregates 2-4 ISLs when switches are connected
Supports multiple 2 Gbit/sec aggregated physical links between switches
Trunks formed between 2 Gbit/sec Bloom switches can provide
aggregate speeds up to 8 Gbit/sec
Bloom trunking port groups include ports 0-3; 4-7; 8-11; and so on
2 Gbit/sec
1 Gbit/sec
1.5 Gbit/sec
0.5 Gbit/sec
1 Gbit/sec
2 Gbit/sec
2 Gbit/sec
1 Gbit/sec
1.5 Gbit/sec
0.5 Gbit/sec
1 Gbit/sec
2 Gbit/sec
8 Gbit/sec ISL Trunk maximum:
All streams share bandwidth
The Bloom ASIC preserves in-order frame delivery
Bloom ASIC aggregates up to 4 ISLs
Bloom - 2 Gbit/sec Trunking Overview
Footnote 1: Automatically creates 2-4 ISL trunks when switches are connected if
requirements are met (trunking license, 2 Gbit/sec ports, same port group; and
portcfgtrunkport on)
Fibre Channel links of between 2 and 4 ISLs are connected to form one logical link.
Bloom logical links can provide aggregate speeds up to 8 Gbit/sec.
Page 8-6
6
CFP264 ILT 0806
Bloom - 2 Gbit/sec Without Trunking:
Unpredictable Performance
ISL utilization in topologies before trunking:
HBA Traffic Round-robin across links
In this example, the 5
th
HBA shares the 1
st
ISL with the 1
st
HBA
Could have single ISL bandwidth congestion (hot spot)
1
1
1
2
1
2
3
2 2
2G 1.5G 1G .5G
1G 1.5G 1G .5G 1G
Frame Traffic on ISLs before Trunking
Four ISLs in
use in this
example
3
4
3
4
1
7Gb IN
5Gb OUT
2G
Congestion
When the allocation to ISL becomes saturated, congestion may occur on an independent
ISL creating a hot spot.
Page 8-7
7
CFP264 ILT 0806
Bloom - 2 Gbit/sec With Trunking:
Maximum Efficiency due to Frame Striping in Hardware
Trunking Improves performance:
Multiplexes frames across links
Maintains in-order delivery
Prevents single ISL bandwidth congestion (hot spot)
1
1
2
1
4
3
3
2
4
1
2
2G 1.5G 1G .5G 2G
2G 1.5G 1G .5G 2G
Frames on the trunk
Frames
can use
any link
Frames
arrive
in-order
Bloom trunks
can be
1, 2, 3 or 4
links wide
3
2
1
7Gb IN 7Gb OUT
Additional
Bandwidth
With trunking now in effect, the 4 ISLs are treated as a single logical ISL, capable of
an 8 Gbit/sec aggregate bandwidth. The sequences of frames are now evenly
multiplexed on the frame level across all four physical paths. Since the source and
destination port groups use the same buffer pool, frames received on the ports will
be reassembled in the same sequence preserving in-order delivery.
Page 8-8
8
CFP264 ILT 0806
Bloom - 2 Gbit/sec Trunk Masters
The first Bloom ISL to connect becomes the trunk master
The trunk master represents the trunk group for routing
FSPF only sees the trunk master
The Bloom ASIC associated with the trunk master will multiplex frames
across ISLs in the trunk group and maintain in-order delivery
Loss of an ISL in Bloom trunking
When a non-master port goes offline no FSPF rerouting occurs. There is
only a loss of bandwidth.
When a trunk master goes offline, loss of bandwidth occurs and rerouting
is done by FSPF
The trunk master defines the trunking group in the routing database and is added as
a route to FSPF. There can be several trunks, with several trunk master ports,
between two adjacent switches; however, there is only one designated principal ISL
between the switches. After a trunking group is fully established, all data intended
for transmission across the trunk are dynamically distributed at the frame level
across the ISLs in the trunking group, while preserving in-order delivery.
If a Master ISL fails or is removed, there is a pause in the I/O associated with that
trunk as a new Master ISL is designated and traffic is redistributed. If data frames
were in flight during the failure of the Master ISL, some frames may be lost,
depending on the Fibre Channel class; Class 2 frames are resent and Class 3
frames are dropped. The impact of any dropped frame on the I/O depends on the
host, application, storage, and host bus adapter (HBA) in use.
When the trunk master port goes offline, its associated ASIC will disable trunking for
all its member ports as well and the FSPF will be deleted. It will send a NOS (Not
Operational Signal) to all its member ports and force a fabric reconfiguration. The
remaining ports will go through the trunking process of selecting a new trunk master
and reform the trunk group. During this reconfiguration of reforming the trunk group,
frames will still be routed (IOD is off) while the fabric is reconfiguring. When the
trunk group re-builds and the trunk group will start processing frames after a FSPF
entry has been made . If a slave port goes offline, the trunk master port updates its
set of trunking group information and FSPF is not informed.
Page 8-9
9
CFP264 ILT 0806
Condors trunking behavior is similar to Blooms
1
Supports multiple 2 Gbit/sec trunks and 4 Gbit/sec trunks between switches
Trunks formed between 4 Gbit/sec Condor switches can provide a
maximum aggregate bandwidth of up to 32 Gbit/sec
Condor trunking port groups include: ports 0-7, 8-15, and so on
4 Gbit/sec
2 Gbit/sec
1.5 Gbit/sec
0.5 Gbit/sec
1 Gbit/sec
2 Gbit/sec
4 Gbit/sec
2 Gbit/sec
1.5 Gbit/sec
0.5 Gbit/sec
1 Gbit/sec
2 Gbit/sec
The Condor ASIC preserves in-order frame delivery
Condor ASICs join up to 8 ISLs
Condor 4 Gbit/sec Trunking Overview
The SilkWorm 4100, 4900, 7500, and 48000 use Brocade Condor ASICs.
Footnote 1: Automatically creates 2-8 ISL trunks when switches are connected if
requirements are met (trunking license, 2-4 Gbit/sec ports, and same port group;
portcfgtrunkport on)
Fibre Channel links of between 2 and 8 ISLs are connected to form 1 logical link.
Condor logical links can provide aggregate speeds up to 32 Gbit/sec.
Page 8-10
10
CFP264 ILT 0806
GoldenEye 4 Gbit/sec Trunking Overview
The GoldenEye ASIC trunking behavior is the same as Condors with
some implementation differences:
Trunks formed between 4 Gbit/sec GoldenEye switches can provide
maximum aggregate bandwidths up to 16 Gbit/sec
GoldenEye trunk port groups include: ports 0-3, 4-7, 8-11, and 12-15
4 Gbit/sec
2 Gbit/sec
1.5 Gbit/sec
0.5 Gbit/sec
1 Gbit/sec
4 Gbit/sec
2 Gbit/sec
1.5 Gbit/sec
0.5 Gbit/sec
1 Gbit/sec
The GoldenEye ASIC preserves in-order frame delivery
GoldenEye ASICs join up to 4 ISLs
The SilkWorm 200E use Brocade GoldenEye ASICs.
Page 8-11
11
CFP264 ILT 0806
Port-based Routing over Trunks
Port based routing is supported on 2 and 4 Gbit/sec switches
A trunk group is considered one logical ISL identified by the trunk master
The load assigned to a trunk group is primarily based on the ISL link cost and
secondarily on total bandwidth metrics of all the ISLs in the group
PT
GP
PT
GP
1 logical ISL (3 ISL Trunk Group)
(Metric Cost 500, Bandwidth = 6 Gbit/sec)
1 Regular ISL
10 Paths Established
Through the Fabric
PT
GP
PT
GP
4
(1
G
b
p
s
)
1
(1
G
b
p
s
)
2
(.5
G
b
p
s
)
3
(1
.5
G
b
p
s
)
1
0
(1G
bp
s)
8
(.5G
b
ps)
7
(.5G
b
p
s)
6 (.5Gbps)
5 (.5Gbps)
9
(2G
bps)
Legend
Trunked ISL
Regular ISL
FSPF considers both the cost and the bandwidth metric when allocating paths in the fabric. In this
diagram we have a trunk group made up of three ISLs (2 Gbit/sec each) providing an aggregate
bandwidth metric of 6 Gbit/sec. In addition to the ISL trunk group we have a single ISL with a
bandwidth metric of 2 or 4 Gbit/sec. Since all physical links are capable of moving data at 2 or 4
Gbit/sec the cost metric assigned for both ISL will be 500. Since both ISLs have a cost metric of 500,
FSPF will look at the bandwidth metric to decide how to allocate paths. Since the ISL trunk group has
6 Gbit/sec and the single ISL has 2 Gbit/sec, this is a 3:1 allocation ratio. As paths are established
though this Fabric the ISL trunk group will receive three times as many paths as the single ISL. In this
example, the 13 paths have been allocated across the ISL trunk group followed by only one path
allocation across the single ISL. The next three paths will be allocated across the ISL trunk group and
then another single path is established across the single ISL. When the allocation is complete, the
ISL trunk group will have eight paths established and the single ISL will have two paths established.
Since real-time bandwidth utilization is volatile, the FSPF does not attempt to consider this in the path
allocation algorithm.
Note - Allocation of paths is dependent on which individual ISL or trunk group becomes ready first.
Because of this, the number of paths allocated to each ISL/trunk group may differ. In this example,
the trunk group ISL came ready first, and therefore received eight paths; whereas the single ISL
received two paths. If the single ISL would have become ready first, it would have received three
paths and the trunk group ISL would have received seven paths.
In this example the ISL trunk group has 10 devices communicating with an accumulated bandwidth
load of 9 Gbit/sec. Since the ISL trunk group has a bandwidth capability of 6 Gbit/sec, some store
and forward will take place in the switch under a full utilization condition. Frame load sharing across
all trunked ISLs will assist in reducing the store-and-forward load in the switch.
Page 8-12
12
CFP264 ILT 0806
Exchange-based Routing over Trunks
Exchange based routing is only supported on 4 Gbit/sec switch trunks
The exchanges assigned to a trunk group/ISL are based primarily on the link
cost & secondarily on total bandwidth metrics of all the ISLs
1 logical ISL (Metric Cost 500,
Bandwidth = 12 Gbit/sec)
1 Regular ISL
Different Exchanges
through the Fabric
Legend
4 Gbit/sec Trunk
4 Gbit/sec ISL
Exchange 1
Exchange 2
Exchange 3
Exchange 4
PT
GP
PT
GP
PT
GP
PT
GP
Dynamic Path Selection (DPS) is exchange base routing where exchanges or
communications between end devices in a fabric are assigned to egress ports in
ratios proportional to the potential bandwidth of the ISL or trunk group.
When there are multiple paths to a destination, the input traffic will be distributed
across the different paths in proportion to the bandwidth available on each of the
paths. This improves utilization of the available paths, thus reducing possible
congestion on the paths. Every time there is a change in the network (which
changes the available paths), the input traffic can be redistributed across the
available paths. This is a very easy and non-disruptive process when the 4 Gbit/sec
Exchange based routing policy is engaged.
Exchanges in the example depicted on this slide are allocated based on the primary
criteria: link cost and 2ndary criteria: potential bandwidth. The potential bandwidth
allocation depicted in this example yields flow allocations of 6:4.
Page 8-13
13
CFP264 ILT 0806
2 & 4 Gbit/sec Trunking Differences
Four 2 Gbit/sec Bloom ports form trunks to 4-ports within an 8-port Condor or
4-port GoldenEye group
1. Four 2 Gbit/sec Bloom ports form trunks to 4-ports within an 8-port Condor or
4-port GoldenEye group
There can simultaneously be a 2 Gbit/sec and a 4 Gbit/sec trunk in the
same Condor or GoldenEye port group
Ports participating in the same trunk group need to have the same speeds
2. Frame allocations across the trunks
2 Gbit/sec switches evenly multiplex frames across the trunk
4 Gbit/sec switches use predetermined port selection
3. Master ISL trunk behavior
2 Gbit/sec switches assign a master ISL
1
used in routing database, when
the Master ISL goes down the trunk has to be rebuilt causing the fabric to
reroute
4 Gbit/sec ASICs assign a psuedo-master port which is used in routing
database
When this psuedo-master port goes offline another port in the trunk
group becomes the new psuedo-master with no disruption
2
Footnote 1: Bloom ASICs assign the 1
st
ISL that connects to be the trunk master.
Footnote 2: When 4 Gbit/sec ASICs initially build a trunk they use a pre-determined
port to become what is called the pseudo-master. If the original pseudo-master goes
down the ISL a second pre-determined port will become the new pseudo-master.
This very predictable behavior allows loss of master ISLs without fabric disruption.
Page 8-14
14
CFP264 ILT 0806
1
st
major difference: One port group can simultaneously have both 2
Gbit/sec and 4 Gbit/sec trunks
SilkWorm4900:admin> switchshow
switchName: SilkWorm4900
switchType: 44.0
switchState: Online
<output truncated>
Area Port Media Speed State
==============================
0 0 id N2 Online E-Port (Trunk port, master is Port 4 )
4 4 id N2 Online E-Port 10:00:00:05:1e:34:01:e6 "toist04b41"
(upstream)(Trunk master)
7 7 id N4 Online E-Port 10:00:00:05:1e:34:3b:8b "toist03b41"
<output truncated>
A six-ISL
2 Gbit/sec
trunk
group
A two-ISL
4 Gbit/sec
trunk group
One Port Group with Multiple ISL Trunks
The SilkWorm 200E GoldenEye ASIC can also form two trunks in the same port
group to another 4 Gbit/sec switch port group.
Page 8-15
15
CFP264 ILT 0806
2
nd
major difference: 4 Gbit/sec and Bloom ASICs allocate trunk frames differently
4 Gbit/sec ASICs evenly distribute frames when bandwidth of trunk is fully utilized
Frames at low bandwidth will not appear to be evenly distributed
Frames use predetermined ports which results in no fabric configurations when
pseudo-master ports go offline
Frames on the trunk
Frames
prefer pre-
determined
ports
Trunks can be
2 8
links wide
Frames
arrive
in-order Additional
Bandwidth
Exchanges
1
1
1
1
1
1
4
4
4
4
4
2
2
2
2
2 2
7
8
6 6
5
5
5
5 3
3
3
3
3
3
Exchanges
Trunking Frame Allocation
Recall, Bloom trunks stripe frames across participating links regardless of total
traffic load, no ISL in trunk group has priority over another. With Bloom trunking in
effect, the 4 ISLs are treated as a single logical ISL capable of 8 Gbit/set aggregate
bandwidth. The sequences of frames are distributed evenly on the frame level
across all four physical paths. Since the source and destination quads use the same
buffer pool, frames received on the ports will be reassembled in the same sequence
preserving in-order delivery.
With 4 Gbit/sec ASIC trunking in effect, 8 ISLs are aggregated into a single logical
ISL capable of 32 Gbit/set aggregate bandwidth. Frames entering the logical pipe
will be allocated to a predetermined port which just happens to be the lowest back
ASIC port in the port group, if that port is busy, they will then be allocated another
predetermined port. When the bandwidth of the pipe is fully utilized the frames are
evenly distributed. When the aggregate bandwidth of the logical pipe is not fully
utilized and the 1st predetermined port is free, the frames do not need to be evenly
distributed. Why calculated time differentials needed to maintain in-order delivery of
frames if not required!
If traffic patterns involve large sequences of frames and high bandwidth utilization
then the frame allocation across the ISLs in the trunk will begin to evenly distribute
frames across the ISLs.
The predictable way frames are allocated to 4 Gbit/sec ASIC trunk groups enables
non-disruptive loss of trunk master behavior which is also referred to as pseudo-
master trunking. This is sometimes called Masterless Trunking.
Page 8-16
16
CFP264 ILT 0806
3
rd
major difference no fabric disruption when trunk master goes
offline!
4 Gbit/sec ASICs changed the role of the Master ISL to a role referred
to as Pseudo-Master
The trunkshow output still depicts the port as MASTER
rsl1_st11_b41_1:admin> trunkshow
1: 4 -> 8 10:00:00:05:1e:02:12:b1 deskew 15 MASTER
0 -> 9 10:00:00:05:1e:02:12:b1 deskew 15
1 -> 10 10:00:00:05:1e:02:12:b1 deskew 16
5 -> 11 10:00:00:05:1e:02:12:b1 deskew 16
. . .<truncated output> . . .
When the MASTER is disabled, trunkshow immediately depicts the
new MASTER NO fabric disruption
rsl1_st11_b41_1:admin> trunkshow
1: 1 -> 10 10:00:00:05:1e:02:12:b1 deskew 16 MASTER
5 -> 11 10:00:00:05:1e:02:12:b1 deskew 16
0 -> 9 10:00:00:05:1e:02:12:b1 deskew 15
4 Gbit/sec ASIC Trunk Pseudo-Master
When the Trunk Master is disabled another pre-determined port takes over the role
without fabric disruption.
Page 8-17
17
CFP264 ILT 0806
Trunks are formed between 2-4 ISLs at 2 Gbit/sec
Four Bloom ports form trunks to 4-ports within an 8-port Condor or 4-port
GoldenEye group
Bloom to 4 Gbit/sec ASIC frames are evenly distributed across the trunk
4 Gbit/sec to Bloom ASIC frames are not evenly distributed at low loads
4 Gbit/sec 2 Gbit/sec ASIC Trunks
1 1
2
1
2 1
Frames on the trunk
Frames
evenly distributed
across all links
Trunks can be
2, 3 or 4
links wide
2
2G 1.5G 1G .5G 2G
1
1 1
1
2
1
2
1
2 Gbit/sec
4 Gbit/sec
2G 1.5G 1G .5G 2G
2
Frames at low
bandwidth are not evenly
distributed across all links
Bloom to 4 Gbit/sec trunk ISLs can only operate at 2 Gbit/sec because Blooms do
not support 4 Gbit/sec. The same trunk masters information is reflected at each
end of the link. If a master ISL is lost in a Bloom to 4 Gbit/sec ASIC configuration
the fabric will reconfigure as the Bloom ASIC selects a new trunk master.
Page 8-18
18
CFP264 ILT 0806
Deskew values are related to distance and link quality
Deskew units represent the time difference for traffic to travel over each
ISL as compared to the shortest ISL in the group
The system automatically sets the minimum deskew value of the shortest
ISL to 15 deskew units
Cable length between shortest ISL and longest ISL in a trunk
Max cable difference between ISLs is 400 m remember to consider this
when creating long distance trunks over WDMs!
Differences > than 30 m could introduce performance degradation
A 30 m difference = 15 deskew unit difference
Since shortest ISL is set to a deskew of 15, an ISL with a difference of 30 m
will have a deskew of 30
An ISL with a deskew above 30 indicates possible degradation
Deskew values are displayed in the trunkshow command output
The Deskew Counter
Light in a vacuum travels much faster, but in optical cable the rate is about
5 ns/meter. 5ns/meter times 30 meters is equal to 150 ns. The difference
in cable lengths between the ISLs in a trunk determines the deskew value.
This is needed for timing purposes so in-order delivery of frames across
the trunk can be ensured. The shortest ISL is selected as the base and is
assigned a deskew value of 150 nsec. The deskew values are expressed
(shown in all command displays) by dividing the time value by 10.
Example: A deskew value of 150 nanoseconds is shown as 15 (150/10).
The first ISL in the Bloom trunk to initialize is selected as the trunk master.
The ISL attached to the lowest backport will become the 4 Gbit/sec ASIC
switch trunk master. The length of the cable is not a consideration when
selecting the master. The deskew values for the other ISLs in the trunk will
be calculated from the base ISL and will have a higher value. Each switch
connected by the ISL will have a deskew value since each has a separate
transmit line to the other. Due to the signal quality/optical media, cables
that are identified as the same length may have a different deskew value.
For example, one cable may have a deskew value of 16 and a cable of the
same length may calculate to be 17. This is not a problem since deskew is
a true measurement of its transmission capabilities.
Page 8-19
19
CFP264 ILT 0806
Commands that enable you to view trunk information
trunkshow
switchshow
islshow
portcfgshow
Commands that allow you to configure trunk parameters
portcfgtrunkport
switchcfgtrunk
Command that allows you to test trunking
trunkdebug <start port>, <end port>
Trunking Related Commands Overview
trunkshow
Displays each trunk group
Displays which local port is connected to which remote port
Displays WWN of the other switch
Displays deskew values and identifies the trunk master port
switchshow
Displays the master port (trunk master) and each non-master port
Displays the WWN of connected switch to trunked ports
islshow
Displays bandwidth information associated with each trunk group
switchcfgtrunk
Used to configure trunking to be ON or OFF on all ports on switch
portcfgtrunkport
Used to configure trunking per port as either ON or OFF, default is ON
portcfgshow
Displays port configuration information including trunk ON/OFF status
trunkdebug <start port>, <end port> tests specified ports in trunk group and reports trunking
status. Possible trunkdebug outputs include: Switch does not support trunking; Trunking license
required; Trunking not supported in switch interop mode; port<port_id> is not E_Port; port<port_id>
trunking is disabled; port<port_id> speed is not 2 Gbit/sec; port<port_id> and port <port_id> are not on
the same quad; port<port_id> and port <port_id> are connected to different switches; port<port_id> is
not a trunk port due to E_port being disabled, or trunking may be disabled at remote port; port<port_id>
and port<port_id> cannot trunk. Check the link length to verify that the difference is less than 400 m.
Page 8-20
20
CFP264 ILT 0806
trunkshow Command
Silkworm4100:admin> trunkshow
1: 8 -> 11 10:00:00:05:1e:34:55:fa deskew 15 MASTER
9 -> 8 10:00:00:05:1e:34:55:fa deskew 16
10 -> 9 10:00:00:05:1e:34:55:fa deskew 16
11 -> 10 10:00:00:05:1e:34:55:fa deskew 17
2:13 -> 13 10:00:00:05:1e:34:55:fa deskew 16 MASTER
12 -> 12 10:00:00:05:1e:34:55:fa deskew 15
3:15 -> 15 10:00:00:05:1e:34:55:fa deskew 16 MASTER
14 -> 14 10:00:00:05:1e:34:55:fa deskew 15
SilkWorm 200E
SilkWorm 4100
Trunk
groups
Source port
Destination port on
attached switch
Destination switch WWN
Trunk master selection is not related to deskew.
Bloom trunk master selection is based on the 1
st
ISL to connect; it is not predictable . It is
dependent on distance and quality of the link and its associated connection points.
With no Bloom ASIC involved, 4 Gbit/sec ASIC trunk master selection is predictable , it is based on
a set of predetermined ports in the trunk group.
The switches in this example have multiple trunk groups between them. Notice that the trunk master
is not always the lowest port number in the group nor is it related to the deskew value.
Note: The trunk group ports 12-15 formed two trunk groups because there are two speeds
(one trunk group has 4 Gbit/sec SFPs and the other has 2 Gbit/sec SFPs).
Page 8-21
21
CFP264 ILT 0806
switchshow/islshow
SilkWorm4100:admin> switchshow
switchName: SilkWorm4100
switchType: 32.0
switchState: Online
switchMode: Native
switchRole: Principal
switchDomain: 1
switchId: fffc01
switchWwn: 10:00:00:05:1e:34:55:fa
zoning: ON (No Access)
switchBeacon: OFF
[* Output Truncated *]
Port Media Speed State
=========================
*** <Truncated Output> ***
8 8 id N4 Online E-Port 10:00:00:05:1e:34:55:fa "SilkWorm200E (downstream)
(Trunk master)
13 13 id N4 Online E-Port 10:00:00:05:1e:34:be:5a "SilkWorm200E" (Trunk master)
15 15 id 2G Online E-Port 10:00:00:05:1e:34:be:5a "SilkWorm200E (Trunk master)
*** <Truncated Output> ***
SilkWorm4100:admin> islshow
1: 8-> 8 10:00:00:05:1e:34:be:5a 2 SilkWorm200E sp: 4.000G bw: 16.000G TRUNK
The switchshow commands shows trunk groups associated with ports 12-15 with the master port of the
individual trunk groups noted. The remaining ports in the trunk groups are the non-master ports. It should also
be noted that only one trunk group represents the principal ISL path.
islshow displays the speeds for each ISL in the trunk groups, the aggregate bandwidth for trunk groups, and the
WWN of the other switch (in this example we are connected to one other switch with three trunk groups. Each
trunk group is defined by the trunk master of each trunk. Port 8 of trunk group 1 points to point 8 if a trunk group
on the attached switch; port 8 is the trunk master of this trunk group on both of these switches.
Here are the attached SilkWorm 200E switchshow and islshow truncated command outputs:
SilkWorm200E:admin> switchshow
switchName: SilkWorm4100 *truncated output*
Area Port Media Speed State
==============================
*truncated output*
8 8 id N4 Online E-Port 10:00:00:05:1e:34:55:fa "SilkWorm4100" (Trunk master)
13 13 id N4 Online E-Port 10:00:00:05:1e:34:55:fa "SilkWorm4100" (Trunk master)
14 14 id 2G Online E-Port (Trunk port, master is Port 15 )
15 15 id 2G Online E-Port 10:00:00:05:1e:34:55:fa "SilkWorm4100"
SilkWorm200E:admin> islshow
1: 8-> 8 10:00:00:05:1e:34:55:fa SilkWorm4100 sp: 4G bw: 16G TRUNK
Page 8-22
22
CFP264 ILT 0806
Trunking turned off for port 8
SilkWorm200E:admin> portcfgtrunkport 8,0
SilkWorm200E:admin> portcfgshow
Ports of Slot 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
-----------------+--+--+--+--+----+--+--+--+----+--+--+--+----+--+--+--
Speed AN AN AN AN AN AN AN AN AN AN AN AN AN AN 2G 2G
Trunk Port ON ON ON ON ON ON ON ON .. ON ON ON ON ON ON ON
Long Distance .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
VC Link Init .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked L_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked G_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Disabled E_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
CreditShare .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
ISL R_RDY Mode .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
RSCN Suppressed .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Persistent Disable.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
where AN:AutoNegotiate, ..:OFF, ??:INVALID.
LM:L0.5
portcfgtrunkport/portcfgshow
portcfgtrunkport is used to turn on or off trunking on one port, specified by port
number. The last parameter indicates on or off (1 for on and 0 for off).
Usage: portCfgTrunkPort [SlotNumber/]PortNumber Mode
Mode: 1 - Configure port to be Trunking capable
0 - Configure port to be Trunking incapable
switchcfgtrunk is used to turn on or off trunking capability for the whole switch
Usage: switchCfgTrunk Mode
Mode: 0 - Configure ports to be Trunking incapable
1 - Configure ports to be Trunking capable
Page 8-23
23
CFP264 ILT 0806
SilkWorm4100:admin:> trunkdebug <start port #>, <end port #>
Used to debug a trunk link failure, operands required
The command reports one of the following:
Switch does not support trunking
Trunking license required
Trunking not supported in switch interop mode
port<port_id> is not E_Port
port<port_id> trunking is disabled
port<port_id> speed is not 2 Gbit/sec or 4 Gbit/sec
port<port_id> and port <port_id> are not on the same port group
port<port_id> and port <port_id> are connected to different
switches
port<port_id> is not a trunk port due to E_port being disabled, or
trunking may be disabled at remote port
port<port_id> and port<port_id> cannot trunk. Check the
link length to verify that the difference is less than 400 m.
SilkWorm4100:admin> trunkdebug 11, 12
port 11 and 12 are not within the same port group
trunkdebug Command
Synopsis trunkdebug port1, port2
port1 - Specify the first port number where you want to debug a trunk link error. This operand is required.
port2 - Specify the last port number where you want to debug a trunk link error. This operand is required.
Example 1: To debug a trunk connection for ports 1 through 4 enter
sw2:admin> trunkdebug 1, 4
port 4 is not E port
Example 2: To debug a trunk for ports 8 11 when only 8 and 11 are trunked
sw2:admin> trunkdebug 8, 11
ports 8 and 11 are trunked together
Director command syntax:
Director:admin> trunkdebug
trunkdebug: area_number1 area_number2
Note: spinfab was not designed to do performance testing over trunks. Trunk performance testing
will have to be done without spinfab.
spinfab was designed to run a functional test of individual switch-to-switch ISL cabling
Optionally use Fabric OS 3.1/4.1+ porttest with a loopback plug at the other end of the link
What is porttest?
Test online ports (F/FL/E or loopback ports); looks for port type and runs appropriate test - can
optionally select port types
Runs in the background so does not affect traffic
Use porttestshow to retrieve information from porttest
Use stopporttest to stop porttest
porttestshow will return porttest data when successful or failed reason and error code if not
successful.
Page 8-24
24
CFP264 ILT 0806
Long Distance Fabric Options Overview
Techniques used to connect SilkWorm switches over long distances:
Use native Fibre Channel and Extended Fabric licenses per ASIC:
Bloom-based switches: Up to 200 km at 1 Gbit/sec, 100 km at 2
Gbit/sec *
Condor-based switches: Up to 500 km at 1 Gbit/sec, 250 km at 2
Gbit/sec, and 100 km at 4 Gbit/sec.
GoldenEye-based switches: Up to 293 km at 1 Gbit/sec, 146 km
at 2 Gbit/sec, and 72 km at 4 Gbit/sec
Select the appropriate cables and SFPs based on the actual
distance
2 and 4 Gbit/sec ASIC switches support long distance trunks
Connect fabrics over non-native FC protocols with ISL R_RDY at
distances 10 km apart over a WAN using 3
rd
party gateways
* Brocade Bloom ASICs include both Bloom I and Bloom II switches
Brocade Bloom I ASIC switches include the SilkWorm 3200, 3800,
3900, and 12000. Bloom I based switches have a 63 KM limit at full 2
Gbit/sec performance.
Brocade Bloom II ASIC switches include the SilkWorm 3250, 3850,
and 24000
Condor-based switches include: SilkWorm 4100, 4900, 7500 and 48000
GoldenEye-based switches include: SilkWorm 200E, and some
embedded products
Page 8-25
25
CFP264 ILT 0806
Extended Fabric Connections
Extended Fabric licensed switches that connect the two remote ends of the
fabrics over a long distance (greater than 10 km) are called Bookend Switches
Connecting ports on bookend switches need to be set to the same long
distance parameters
Use Web Tools Switch Admin Extended Fabric View panel
Use the portcfglongdistance command to specify an Extended
Fabric Distance Level
Components can include: dark fiber, ELW SFPs, WDMs, and FC extenders
Long Distance Link
Native Fibre Channel
Data traffic Virtual Channels (VCs) are collapsed to optimize performance over long
distances using the portcfglongdistance command
SFP options include:
4 Gbit/sec short-wavelength laser (SWL) SFPs can transmit data up to 300
meters on 50 optical cables, and up to 150 meters on 62.5 optical cables.
4 Gbit/sec long-wavelength laser (LWL) SFPs can transmit data up to 10
kilometers on 9 optical cables.
Extended long-wavelength laser (ELWL) SFPs can transmit data up to 80
kilometers. ELWL SFPs may not be supported on all Brocade switches by
our partners investigate this carefully.
VC 2 only
VC2
VC3
VC4
VC5
VC2
VC3
VC4
VC5
Page 8-26
26
CFP264 ILT 0806
Extended Fabric Distance Levels
Bloom ASICs 5 to 10 km at 2 Gbit/sec Level E static mode LE
100 km at 1, 2, or 4 Gbit/sec Level 2 static mode L2
SilkWorm Condor ASIC:
- 500 km at 1 Gbit/sec; 250 km at 2
Gbit/sec; and 100 km at 4 Gbit/sec
SilkWorm GoldenEye ASIC:
- 293 km at 1 Gbit/sec; 146 km at 2
Gbit/sec; and 72 km at 4 Gbit/sec
Bloom ASICs:
3
- 100 or 200 km at 1 Gbit/sec
- 100 km at 2 Gbit/sec
Dynamic mode uses
automatic distance
detection and a user-
defined distance
LD
Based on user specified distance
Distance limitations same as LD
Static mode that allows a
user-defined distance
2
LS
50 km at 1, 2, or 4 Gbit/sec Level 1 static mode L1
25 km at 1, 2, or 4 Gbit/sec Level 0.5 static mode L0.5
Maximum ISL Distance (km) Description Level
1
Footnote 1: Level is also referred to as distance mode.
Footnote 2: LS mode is for Fabric OS v5.1 and above ONLY.
Footnote 3: The maximum Extended Fabric distance depends on the version of
SilkWorm switch ASIC installed in the switch. Bloom ASICs can allocate enough
buffers to keep a 100 km link full at 1 Gbit/sec and ~2 Gbit/sec. Bloom II ASICs
with Fabric OS v4.4 and above can allocate enough buffers to keep a 200 km link
full at 1 Gbit/sec. Bloom II ASICs with Fabric OS v4.4 and above can now also
allocate enough buffer credits to keep a 100 km link full at 2 Gbit/sec.
Extended Fabric distance levels (L0, LE, L0.5, L1, L2, LD) persist across switch
reboots and power cycles
Can not be set or removed by configure or configdefault
Can be cleared by portcfgdefault
Saved in a switch configuration file (configupload ) as portcfg parameter
Page 8-27
27
CFP264 ILT 0806
Extended Fabric Configuration
Use the portcfglongdistance command to configure bookend
switch ports used to connect the fabrics
Distance level = L0.5, L1, L2, LS, and LD must be the same at each end
of the link
Desired Distance represents the user-estimated (LD) maximum
distance in km or a user-specified (LS) distance in km
Example setting port 3 to a user defined 75 km LS distance level:
SilkWorm4100:admin> portcfglongdistance 3 LS 1 75
The 1 is associated with a VC Link Init setting that needs to be manually
configured from CLI when an LD or LS desired distance parameter is
supplied (example: 75)
The VC Link Init value is automatically set to 1 for all other Lx
configurations engaged with the portcfglongdistance command
Syntax portcfglongdistance [Slot/]<port_number> , distance
level, [vc_translative_init ] , <desired_distance>
The vc_translative_init value of 1 in the example on this slide is needed as
a syntax position holder. If a desired_distance is not specified then
vc_translative_init is enabled without explicitly calling it out. Example setting
a L1 connection on port 4 (vc_translative_init will automatically be enabled if
command is issued from Fabric OS v4.4 or higher switch):
portcfglongdistance 4 L1
Best Practice When VC Translation Link Initialization is enabled (set to ON), the
port uses an extended link initialization sequence, which is an enhanced link reset
protocol designed to avoid excessive resetting of ports used on a long distance
connection. As a best practice, Brocade recommends this feature be enabled. It is
enabled by default on Fabric OS v4.4 and higher switches. Use the portcfgshow
command to verify that vc_translaltion_link_init is enabled at both ends
of the link
Page 8-28
28
CFP264 ILT 0806
Extended Distance Trunking
Trunking over extended fabrics is supported on switches running Fabric
OS v3.2.0 (or later) and v4.4.0 (or later)
SilkWorm 200E switches support extended trunks in Fabric OS v5.1 and
later
Criteria for trunking an Extended Fabric
All trunking criteria must be met
Extended Fabric license on supported switches at both sides of the ISL
All trunk ports in the same trunk group must be configured to the same long
distance mode
Example: LD LD, L0.5 L0.5, L1 L1, L2 L2
LD modes must specify the same maximum distance parameter
LS modes must specify the same user defined distance parameter
Long distance ISLs within trunk groups are subject to distance limitations
1
In Fabric OS versions earlier than Fabric OS v5.1, the only long-distance modes supported by the
SilkWorm 200E were modes L0 and LE.
Trunking criteria review
Trunking license on both sides of the ISL.
Trunks must originate and end in a port group.
All ports in the same trunk group must operate at the same speed (2 or 4 Gbit/sec) and
must be within recommended deskew limits. 400 m max distance between ISLs correlates to a
max deskew difference = 200 (2000 nanoseconds); 30 m or less distance between ISLs for
optimal performance correlates to a deskew difference = 15 (150 nanoseconds).
Ports in trunk group must have trunking turned on.
Footnote 1: Distance limitations are based on available buffer credits. 108 buffers are allocated per
Bloom I and Bloom II 4-port group. Bloom IIs allow all 108 of these credits to be allocated to one
port in the quad. Bloom I is allows a maximum of 63 buffer credits to be allocated to any one port
in a quad. Condors can allocate up to 255 buffers to each of one, two, or three ports in a buffer
allocation group, and up to 1000 buffers for all 32 ports. GoldenEyes allocated up to 288 buffers
for all 16 ports, with a maximum of 128 buffers to any one port.
For dynamic long distance links, you can approximate the number of buffer credits using the following
formula: Buffer credits = [(distance in km) * (data rate) * 1000] / 2112 The data rate is 1.0625 for 1
Gbit/sec, 2.125 for 2 Gbit/sec, and 4.25 for 4 Gbit/sec and Fibre Channel.
Note: VC Translation Link Init on required for long distance trunks. While this is a
recommended enabled parameter for all extended distance ISL connections, extended trunk
connections require that this parameter be enabled. Recall, VC Link Init uses an extended link
initialization sequence, which is an enhanced link reset protocol designed to avoid excessive
resetting of ports used on a long distance connections. Fabric OS v3.2/4.4 turns this parameter on
by default when the portcfglongdistance command is engaged.
Page 8-29
29
CFP264 ILT 0806
Extended Trunk Configurations
A Fabric OS v3.2/4.4 Extended Fabric can be built over a trunk up to
supported distances:
Bloom-based switches support trunks up to 50 km (2 ISLs)
4 Gbit/sec switches support one 3-port trunk up to 250 km at 2 Gbit/sec
and 100 km at 4 Gbit/sec
Extended distance trunks can be formed between any combination of
2 and 4 Gbit/sec switches
When a 4 Gbit/sec ASIC switch forms an extended distance trunk with
a Bloom-based switch
The 4 Gbit/sec ASIC switch is limited to the supported speed and buffer
availability of the connecting Bloom-based switch
See Fabric OS Administrators Guide and tables at the end of this module for additional
information.
spinfab was not designed to do performance testing over trunks, extended trunks, or extended
distance ISLs. Trunk, extended trunk, and extended distance ISL performance testing will have to
be done without spinfab
spinfab was designed to run a functional test of individual switch-to-switch ISL cabling under
normal operation (L0 mode) only
spinfab utilizes VCs so is not available for long distance links, including long distance trunks
spinfab assumes a standard LO VC connection which is why it will not work on long distance
connections that collapse VCs 2-5 to VC 2. The spinfab command sends frames out an ISLs
VCs 2-5 with an OX_ID and an S_ID of the original switch that causes the E_Port at the other
end of the link to spin them back. When they are returned (spun back) the originating E_Port
verifies that the frames are still in order and there are no 8b/10b encoder errors.
Optionally use Fabric OS 3.1/4.1+ porttest with a loopback plug at the other end of the link
What is porttest?
Test online ports (F/FL/E or loopback ports
)
1
; looks for port type and runs appropriate test -
can optionally select port types
Runs in the background so does not affect traffic
Use porttestshow to retrieve information from porttest
2
Use stopporttest to stop porttest
porttestshow will return porttest data when successful or failed reason and error code if
not successful.
Page 8-30
30
CFP264 ILT 0806
ISL R_RDY
For distances over 100 km, connect fabrics through a WAN via 3
rd
party Gateway Switches
On the bookend switches use the native (non-licensed) ISL
R_RDY Mode
Switch B
Gateway A Gateway B
ATM
FCIP
SONET
Fibre
Channel
Fibre
Channel
Switch A
In the example above, two switches are connected via two gateway switches. These gateway
switches convert the Fibre Channel data transmitted by the SilkWorm switch port into ATM, FCIP, or
SONET-formatted data for transmission over greater-than-Fibre Channel distances. There can be
multiple SilkWorm switches and directors connected to either gateway switch, and all switches and
directors are merged into a single Fibre Channel fabric.
In pre-Fabric OS v3.1 and v4.1 use a Remote Switch license to make long-distance connections
through a WAN gateway.
Remote Switch license required on each switch attached to the gateway; Fabric OS configures
port(s) as needed.
Gateway must support a special Brocade mode - With a Remote Switch license, SilkWorm
switch ports initialize using the same mode as an E_Port (ELP Mode 1). This mode is not what
is expected by a WAN gateway; thus, the WAN gateway must support this special mode for
Brocade switches.
Fabric OS v3.1, v4.1 and above (including Fabric OS v5.x) simplifies gateway connections with the
new ISL R_RDY Mode.
Part of Fabric OS no separate license required; user identifies the port(s) as needed.
Uses a standard gateway mode no Brocade-only support needed - When a port is configured
to use ISL R_RDY mode, SilkWorm switch ports initialize using a different mode (ELP Mode 2,
based on R_RDYs) than that used with E_Port (ELP Mode 1). This mode is expected by a WAN
gateway; thus, the WAN gateway does not need to support a special mode for Brocade switches.
Page 8-31
31
CFP264 ILT 0806
Summary
Four 2 Gbit/sec Bloom ports form trunks to 4-ports within an 8-port Condor
or 4-port GoldenEye group
Trunking commands include: trunkshow, switchshow, islshow,
portcfgshow, portcfgtrunkport, switchcfgtrunk, and
trunkdebug
Criteria for trunking an Extended Fabric
All trunking criteria must be met (trunking license, common speed, and
ports originate and end in a port group)
Extended Fabric license on supported switches at both sides of the ISL
All trunk ports in the same trunk group must be configured to the same
long distance mode
Long distance ISLs within trunk groups are subject to distance
limitations
Techniques used to connect SilkWorm switches over long distances:
Using native Fibre Channel and Extended Fabric licenses
Connecting fabrics over non-native FC protocols with ISL R_RDY at
distances 10 km apart over a WAN using 3
rd
party gateways
Please review supplemental appendix material at your leisure
Appendix material includes a long distance summary and additional Extended
Fabric information including a overview of the portbuffershow command.
Page 8-32
32
CFP264 ILT 0806
Review Questions
1. List two trunking requirements
2. List two Extended Fabric trunking requirements
3. What ports will form a trunk if two SilkWorm 200E switches have ISLs
between ports 0-8 and ports 6 and 7 have 2 Gbit/sec SFPs and the
rest have 4 Gbit/sec ISLs?
4. What distance levels (Lx) could be used to configure a 50 km trunk?
5. What are two possible ways to create long distance fabrics?
1 . T r u n k i n g r e q u i r e m e n t s i n c l u d e :
a . T r u n k i n g l i c e n s e r e q u i r e d a l l p a r t i c i p a t i n g s w i t c h e s ; a u t o m a t i c a l l y e n a b l e d w h e n t h e
l i c e n s e i s i n s t a l l e d a n d p o r t s a r e r e i n i t i a l i z e d ( s w i t c h d i s a b l e / e n a b l e o r
p o r t d i s a b l e / e n a b l e o n a l l p o r t s )
b . T r u n k i n g m u s t b e e n a b l e d ( p o r t c f g t r u n k p o r t ) o n t h e t r u n k p o r t s
c . T r u n k p o r t s m u s t o p e r a t e a t a c o m m o n s p e e d ( 2 o r 4 G b i t / s e c )
d . T r u n k p o r t s m u s t o r i g i n a t e a n d e n d i n a t r u n k i n g p o r t g r o u p o n t h e s a m e A S I C
2 . E x t e n d e d f a b r i c t r u n k i n g r e q u i r e m e n t s i n c l u d e :
a . A l l t r u n k i n g c r i t e r i a m u s t b e m e t
b . E x t e n d e d F a b r i c l i c e n s e o n s u p p o r t e d s w i t c h e s a t b o t h s i d e s o f t h e I S L
c . A l l t r u n k p o r t s i n t h e s a m e t r u n k g r o u p m u s t b e c o n f i g u r e d t o t h e s a m e l o n g d i s t a n c e
m o d e a n d d i s t a n c e s p e c i f i c a t i o n ( L D a n d L S m o d e s )
E x a m p l e : L D L D , L 0 . 5 L 0 . 5 , L 1 L 1 , L 2 L 2
L D m o d e s s h o u l d s p e c i f y t h e s a m e m a x i m u m d i s t a n c e p a r a m e t e r
L S m o d e s s h o u l d s p e c i f y t h e s a m e u s e r d e f i n e d d i s t a n c e p a r a m e t e r
d . L o n g d i s t a n c e I S L s w i t h i n t r u n k g r o u p s a r e s u b j e c t e d t o a c r e d i t l i m i t
e . V C T r a n s l a t i o n L i n k I n i t o n r e q u i r e d f o r l o n g d i s t a n c e t r u n k s
3 . P o r t s 0 - 3 w i l l f o r m a 4 - p o r t 4 G b i t / s e c t r u n k ; p o r t s 4 a n d 5 w i l l f o r m a 2 - p o r t 4 G b i t / s e c t r u n k , a n d
p o r t s 6 a n d 7 w i l l f o r m a 2 - p o r t 2 G b i t / s e c t r u n k
4 . D i s t a n c e l e v e l L 1 , L D , o r L S c o u l d b e u s e d
5 . E x t e n d e d F a b r i c l i c e n s e o r I S L R _ R D Y a n d a W A N g a t e w a y .
Page 8-33
33
Brocade Education Services
Brocade
Product Training
CFP264 ILT 0806
CFP264
Brocade 4 Gbit/sec Accelerated BCFP
End of Instructor-Led Module 8
Trunking and Distance
Page 8-34
34
CFP264 ILT 0806
Long Distance Storage Area Networks
FC-to-TCP/IP Gateway ELWL,
SWL
or LWL
FCIP license and
Multiprotocol Router,
SilkWorm 7500, or FR4-18i
blade in a SilkWorm 48000
TCP/IP Distance
and Speeds
Fabric OS
ISL R_RDY mode
portcfgislmode
command
>10 km &
Fabric OS v3.1/4.1
CWDM/DWDM
Repeater
Link extender
Dark fiber
ELWL,
SWL
or
LWL
Extended Fabric
License
>10 km & 100 km
FC-to-ATM Gateway
FC-to-SONET Gateway
Dark fiber
LWL
Remote Switch
License
>10 km &
Fabric OS older
than v3.1/v4.1
none LWL Standard Fabric OS >150 m & 10 km
External Device(s) SFP Product Distance
Note: SWL is used for attaching to an external device less than 150 m (4 Gbit/sec) and LWL
is used for attaching to an external device greater than 10 km.
As SANs have evolved, enterprises increasingly need to interconnect SAN islands in
different locations as they attempt to support distributed facilities and address business
continuance. These high availability systems are designed to continue to work in light of
hardware or system failures. When distances greater than 10 km are desired between two or
more fabrics that a standard long wave SFP cannot reach, either an extended long wave
SFP (ELWL) or a third party fiber channel extender is required.
FC signals can be multiplexed using multiplexers: Coarse Wave Division Multiplexer
(CWDM) or Dense Wave Division Multiplexer (DWDM). Multiplexing signals over long
distance can be very costly. Considerations for pricing are protocol and distance.
Advantages include up to 32 signals over a single optical strand, mix and match nearly any
protocol (Fibre Channel, ESCON, FICON and ATM).
The portcfgislmode command is used when a distance is > 10 km is configured using
3
rd
party gateways. In Fabric OS v3.1/v4.1, no special license is needed. Pre-Fabric OS
v3.1/4.1 switches needed a Remote Switch license to make a long distance connection
using 3
rd
party gateways.
Page 8-35
35
CFP264 ILT 0806
Bloom ASIC Long Distance Port Matrix
4-port group
Configuration
Options
for Fabric OS
v3.1/4.1 and
higher
2 Gbit/sec
Bloom2-based
ASIC switches
ports operating
at full 2 Gbit/sec
E/LE/Fx E/LE/Fx E/LE/Fx E/LE/Fx
LE/Fx E/LE/Fx E/LE/Fx L0.5
Disabled LE/Fx L0.5 L1
LE/Fx LE/Fx L0.5 L0.5
Disabled E/L0.5/LE/Fx E/L0.5/LE/Fx L0.5
Fx LE/Fx LE/Fx L1
Disabled E E L1
Disabled Disabled L1 L1
Disabled Disabled Disabled L2
Port D Port C Port B Port A
See the Brocade Fabric OS Administrators Guide for tables that include the
number of ports that can be configured at specified distance levels per switch
model.
Note: A port connected to a device that is in loopback mode might become
disabled for lack of buffers if another port in that group is set to L2 mode.
The available distance configurations per Bloom four port group are based on
available buffer credits:
Bloom I and Bloom II ASICs have 112 buffers available per quad, but only 108
of them are available for data frames. The remaining four buffers are reserved
for link control, multicast, and broadcast traffic.
Bloom I switches (SilkWorm 3200, 3800, 3900, and 12000) can allocate a
maximum of 63 buffer credits to any one port.
Bloom II switches (SilkWorm 3250, 3850 and 24000s) had the same limitation
in pre-Fabric OS v3.2/4.4. Bloom II switches with Fabric OS v3.2/4.4 and
higher can allocate all 108 buffers to any one port.
Page 8-36
36
CFP264 ILT 0806
4 Gbit/sec Switch Buffer Allocation
4 Gbit/sec switches reserve buffers for all the ports associated with the ASIC
The embedded port has 24 buffer credits reserved
Each external port has 8 buffer credits reserved
Each internal port has 5 buffer credits reserved (used by switches with
more than 32 ports and 4 Gbit/sec blades)
Condor ASICs have a total of 1000 buffer credits available for ports
GoldenEye ASICs have a total of 264 buffer credits available for ports
Each external port distance level has associated buffer credits
20 max/port 10 max/port LD/LS
206 106 L2
106 56 L1
56 31 L0.5
26 16 LE
26 26 (17*) L0
4 Gbit/sec 2 Gbit/sec Distance Level
* The GoldenEye ASIC allocates 17 buffers in L0 mode (level)
Page 8-37
37
CFP264 ILT 0806
portbuffershow
Display buffer allocation information with the portbuffershow command
SilkWorm4100:admin> portbuffershow
User Port Lx Max/Resv Buffer Needed Link Remaining
Port Type Mode Buffers Usage Buffers Distance Buffers
---- ---- ---- ------- ------ ------- --------- ----------
0 L - 8 8 - -
... <truncated output> ...
3 E - 8 26 26 5km
4 E - 8 26 26 2km
5 E - 8 26 26 2km
6 - 8 0 - -
7 - 8 0 - -
8 E LM 56 31 31 25km
9 E LM 56 31 31 25km
10 E - 8 26 26 2km
11 E - 8 26 26 2km
12 E - 8 26 26 2km
13 E - 8 26 26 2km
14 E - 8 26 26 2km
15 E - 8 26 26 2km
16 - 8 0 - -
... <truncated output, all truncated ports the same as port 16>...
31 - 8 0 - - 486
Remaining
Bufferswill be
calculated by
subtracting the greater
value: Buffer
Usage or Max/Resv
Buffers from the
total available in buffer
allocation group
Remaining
Buffers
from 1000
available =
486
For each user port, the following information is displayed:
Possible Port Type values are E (E_Port), F (F_Port), G (G_Port), L (L_Port), or U (U_Port).
Possible Lx Mode (level) values are (Mode L0), LE (mode LE), LM (mode L0.5), L1 (mode L1), L2 (mode
L2), LD (mode LD) and LS (mode LS).
Max/Resv Buffers displays the maximum number of buffers that will be allocated to the port, based on the
maximum possible speed for the port and the specified LX mode. In the example on this slide, port 8 is
configured as mode L0.5. To support the maximum possible speed of 4 Gbit/sec, the Condor reserves 56
buffers for port 8.
Buffer Usage displays the actual number of buffers the system was able to allocate to the port. In the
example above, port 8 is configured as mode L0.5 but is operating at 2 Gbit/sec. To support the actual
speed, the Condor allocates 31 buffers (at 2 Gbit/sec) for port 8 but reserves 56 (for 4 Gbit/sec
potential) which will not be available in the Remaining Buffers. If port speed is locked to 2 Gbit/sec
the Max/Resv Buffers will equal the Buffer Usage.
Needed Buffers displays the estimated number of buffers that are needed for the port to operate at full
bandwidth (depending on the port configuration). In the example above, 31 buffers are needed for port 8, to
support a full-bandwidth, 2 Gbit/sec, mode L0.5 port.
Link Distance displays (in kilometers) either the physical distance of an LD/LS link, or the maximum
distance for a non-LD/LS port. In the example above, the link distance value for port 8 is 25 km, which
matches the 2 Gbit/sec, mode L0.5 configuration of the port.
On a Condor/GoldenEye-based switches, the Remaining Buffers value is determined by subtracting the
larger of the per-port Max/Resv Buffers or Buffer Usage values from the total buffers available. Since
the example above is taken from a SilkWorm 4100 with a Condor ASIC there are a total of 1000 buffers
available to the ports (there are no internal buffers). These values are:
There are 21 ports with 8 buffers (ports 0-2, 6,7, and 16-31): 8 x 21 = 168
There are 9 ports with 26 buffers (ports 3-5, and 10-15): 26 x 9 = 234
There are 2 ports with 56 buffers (ports 8 and 9): 56 x 2 = 112
Remaining Buffers is 1000 168 234 112 = 486 which is the value shown on the last line in
the slide above.
Page 8-38
Extended ISL Trunking
Trunking support for Bloom-based ASIC switches:
Trunking support for 4 Gbit/sec Condor ASIC switches:
Trunking support for 4 Gbit/sec GoldenEye ASIC switches:
1 (one 2-port trunk) 50 km L1
0 100 km L2
3 (one 3-port trunk) 25 km L0.5
4 (one 4-port trunk) 10 km LE
Number of 2 Gbit/sec ports Distance Mode
0 0 500 km LD/LS
0 3 (one 3-port trunks) 250 km LD/LS
0 3 (one 3-port trunks) 200 km LD/LS
7 (one 7-port trunks)
32 (four 8-port trunks)
Number of 2 Gbit/sec ports
7 (one 7-port trunks) 50 km L1
3 (one 3-port trunks) 100 km L2
15 (one 8-port trunks) 25 km L0.5
32 (four 8-port trunks) 10 km LE
1 (0 trunks)
Number of 2 Gbit/sec ports
1 (0 trunks) 50 km L1
0 100 km L2
3 (one 3-port trunks) 25 km L0.5
16 (four 4-port trunks) 10 km LE

Brocade Product Training: CFP264 Brocade 4 Gbit/sec Accelerated BCFP Instructor-Led Module 8 Trunking and Distance

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brocade Product Training: CFP264 Brocade 4 Gbit/sec Accelerated BCFP Instructor-Led Module 8 Trunking and Distance

Uploaded by

Copyright:

Available Formats

2006 Brocade Communications Systems, Incorporated.

Revision CFP264 ILT 0806

You might also like