Cisco - VPC Concepts

vPC Best Practices with Nexus
SAVBU TME Team
2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
Nexus 7000
5.0(3) 5.1(2) 5.1(3) 5.2
Complete Sync
Partial Sync
Nexus 5000
5.0(3)N1
5.0(3)N2 5.1(3)N1 5.2N1 Nexus 7000
E-Rocks Nexus 3000

5.0(3)U1
5.0(3)U2
Andaman
5.1(3)U1
Complete sync done at major releases

Architectural changes Major enhancements Major new features
Partial sync done at minor releases

Critical flaws/bugs Minor new features Minor enhancements
Cisco Confidential 2
vPC basic components Hardware Specific Considerations vPC enhancements L3 and vPC Adding FEX Summary designs
Cisco Confidential
vPC is a Port-channeling concept extending link
aggregation to two separate physical switches

Allows the creation of resilient L2 topologies based
on Link Aggregation.
Eliminates the need for STP in the access-distribution
Provides increased bandwidth

All links are actively forwarding
Physical Topology
Logical Topology
Virtual Port Channel

vPC maintains independent control planes vPC switches are joined together to form a domain
Si
L2
Si
vPC domain
Non-vPC vPC Increased BW with vPC
2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
vPC peer keepalive link
vPC peer link
vPC peer a vPC switch, one of a pair vPC member port one of a set of ports (port
channels) that form a vPC

Primary Secondary
vPC the combined port channel between the
vPC peers and the downstream device vPC peer

vPC peer link Link used to synchronize state
vPC
between vPC peer devices, must be 10GbE. Also carry multicast/broadcast/flooding traffic and data traffic in case of vpc member port failure
vPC member port
vPC peer keepalive link the peer keepalive
link between vPC peer switches. It is used to carry heartbeat packets

CFS Cisco Fabric Services protocol, used
for state synchronization and configuration validation between vPC peer devices
Orphan portNon-vPC member port
Orphan Port
Cisco Confidential
Graceful consistency check:

On the N7k: NXOS 5.2 On the N5k: NXOS 5.0(2)N2(1)
IGMP bulk sync: On N7k: to be verified On N5k: starting from NXOS 5.0(3)N1(1a)
Per VLAN consistency check:

On the N7k: NXOS 5.2 On the N5k: 5.0(2)N2(1)
Multicast Optimization on Peer-link: On N7k: hidden comand as of NXOS 5.1(3) (but not supported) On N5k: starting from NXOS 5.0(3)N1(1a)
Autorecovery:
On the N7k: NXOS 5.2 On the N5k: NXOS 5.0(2)N2(1)
ARP synchronization: On N7k: NX-OS 4.2(6) and 5.0(2) (Bogota), fixed in 5.1(1) (Cairo) On N5k: under investigation for Goldcoast
vPC peer-switch: On N7k: 4.2(6), 5.x On N5k: under investigation for Goldcoast
Config-sync:
On the N7k: Freetown On the N5k: NXOS 5.0(2)N2(1)
FEX preprovisioning: On N7k: Freetown On N5k: NXOS 5.0(2)N1(1)
vPC on FEX
On the N5k: NXOS 4.2(1)N1(1) On the N7k: NXOS 5.2
Dual Layer vPC: On N7k: TBD On N5k: Fairhaven
Orphan Ports shutdown:

On N7k: NXOS 5.2 On N5k: E-Rocks+
Cisco Confidential
vPC allows a single device to use a port
channel across two neighbor switches (vPC peers)

Eliminate STP blocked ports
Layer 2 port channel only Provide fast convergence upon
vPC Peers
Portchannel vPC Peers
link/device failure
Port channel
Cisco Confidential
Peer Link carries both vPC data and control
traffic between peer switches

Carries any flooded and/or orphan port traffic Carries STP BPDUs, IGMP updates, etc. Carries Cisco Fabric Services messages (vPC
vPC Peer Link 5k01 5k02
control traffic)
Carries multicast traffic (more details follow)
Minimum 2 x 10GbE ports ALL VLANS used on vPC PORTS MUST BE
PRESENT ON THE PEER-LINK

5020 5020 5020 5020 5020 (config)# interface port-channel 10 (config-if)# switchport mode trunk (config-if)# switchport trunk allowed <BETTER TO ALLOW ALL VLANS> (config-if)# vpc peer-link (config-if)# spanning-tree port type network
Cisco Confidential
Peer Keep-alive provides and out of band heartbeat
between vPC peers
Purpose is to detect and resolve roles if a Split Brain
(Dual Active) occurs timeout
Messages sent on 1 second interval with 5 second

3 second hold timeout on peer-link loss before
Peer Keepalive can be carried over the OOB management network int mgmt 0
triggering recovery
Should not be carried over the Peer-Link Use the mgmt0 interface in the management VRF Can optionally be a dedicated link, 1Gb is adequate
(first 16 ports on 5020 are 1/10GE ports)
3rd option, use a routed inband connection over L3
infrastructure (using SVIs in the default VRF)
dc11-5020-1(config)# vpc domain 20 dc11-5020-1(config-vpc-domain)# peer-keepalive destination 172.26.161.201 source 172.26.161.200 vrf management Note: --------:: Management VRF will be used as the default VRF ::--------
Cisco Confidential
10
Peer keep-alive is a routable protocol (both N5K
Standby Management Interface Active Management Interface
and N7K)
Primary design requirement is to have a physically
different path than all other vPC traffic

In all cases do not carry the peer-keepalive
communication over the vPC peer-link On Nexus 7000 when possible use dedicated VRF and front panel ports for peer-keepalive link (1G is more than adequate). 2nd best is to use the management interfaces 3rd option is to use an upstream L3 network for peer-keepalive
If using mgmt 0 interfaces do not connect the
supervisor management interfaces back to back
In a dual supervisor configuration only one management port will be active at a given point in time!
Connect both mgmt 0 ports to the OOB network
vPC basic components Hardware Specific Considerations
vPC forwarding rules

vPC enhancements L3 and vPC
Adding FEX
Summary designs
Cisco Confidential
12
Cisco Nexus 5000 Series

Peer keepalive:
1st option management port. 2nd option dedicated front panel port in dedicated VLAN. 3rd option upstream L3 network
Cisco Nexus 7000 Series vPC works on all existing I/O modules Peer keepalive:
1st option dedicated front panel port in dedicated VRF.
2nd option is management interface.

3rd option upstream L3 network
M1/F1 cards can be used for vPC Peer-link requires 10 GigE cards Peer-link should not span M1 and F1,
peer-link should be made on either all F1 cards or all M1 cards
Cisco Confidential
13
NEXUS 7000 I/O modules

Part number N7K-M132XP-12 N7K-M132XP-12L Model vPC Peer-link (10 GE Only) VPC Member Port
N7K-M148GT-11 N7K-M148GT-11L N7K-M148GS-11 N7K-M148GS-11L
N7K-M108X2-12L
N7K-F132XP-15
M-Series Mode
M
Mixed Chassis Mode

M
F-Series Mode vPC Peer-link on F-Series Modules
Mixed Chassis Mode vPC Peer-link on F-Series Modules (*)
vPC Peer-link on M-Series Modules
vPC Peer-link on M-Series Modules
Recommendation : for mixed chassis mode (F1/M1) with vPC peer-link on F1 ports, use at least 2 M1 LC. This will provide resiliency for L3 features (FHRP, SVI). (*) : command peer-gateway exclude-vlan <vlan list> needed for backup routing path over vPC peer-link
NX-OS 5.1.3 introduces new behavior for handling vPC peer-gateway in mixed chassis
mode (M1/F1) : Topology with M1 peer-link : IP/ARP packets destined to the remote Active IP/MAC get routed locally Topology with F1 peer-link : IP/ARP packets destined to the remote Active IP/MAC use the tunneling mechanism
M-Series Mode
Mixed Chassis Mode
F-Series Mode
Mixed Chassis Mode
Knob Not Required Classic behavior of peer-gateway
Knob not Required Peer Gateway not required
Knob Required for transit path/VLAN IP/ARP Tunneling over Peer link
Cisco Confidential
16
vPC Primary S1
F1
vPC Secondary S2
F1
vPC Primary
S1
M1
vPC Secondary S2
M1
vPC Peer-link
vPC Peer-link
vPC Primary S1
M1
vPC Secondary S2
F1
vPC Peer-link
Cisco Confidential
17

Adding FEX
Summary designs
Cisco Confidential
18
With dual-active scenarios
5k01
5k02
MAC address synchronization is interrupted

IGMP synchronization is interrupted
There is a 50% likelihood that
3 IGMP report for G1
unicast traffic is flooded and that multicast traffic is dropped
4 igmp sync lost
2 - Host subscribes to G1
Cisco Confidential
19
There will be 2 primary switches sending independent BPDUs
VPC Port-channels on upstream/downstream switches will be error-disabled by EtherChannel Misconfiguration Guard after ~90 seconds
http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a008009448d.shtml
If Nexus 7000/5000 is on the other end of VPC no action from
STP as 7000/5000 do not support EtherChannel Guard
Cisco Confidential
20
When the peer-link is
disconnected
vPC secondary detects primary
5k01 vPC Primary Po10
5k02
switch is alive through peer keepalive link

The secondary vpc peer switch
vPC Secondary
suspends all its vpc member ports in order to avoid traffic drop
KEEP PEER KEEPALIVE
AND PEER-LINKS SEPARATE
Cisco Confidential
21
dca-n7k2-vdc2
vPC supports standard 802.3ad port channels from upstream and or downstream devices Recommended to enable LACP channel-group 201 mode active
dca-n7k2-vdc2# sh run interface port-channel 201 version 4.1(5) interface port-channel201 switchport mode trunk switchport trunk allowed vlan 100-105
dc11-5020-1
dc11-5020-2
dc11-5020-1# show running int port-channel 201 version 4.1(3)N1(1) interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201
Cisco Confidential
22
vPC forwards only on locally connected
dca-n7k2-vdc2
members of the port channel if any exist (same principle as VSS)

Multiple topology choices Square Full Mesh
dca-n7k2-vdc2# sh run interface port-channel 201 version 4.1(5) interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201
dc11-5020-1
dc11-5020-2
Cisco Confidential
23
MAC_C
vPC maintains layer 2 topology synchronization
5
CFS
via CFS
Copies of flooded frames are sent across the
vPC-Link in case any single homed devices are attached
56 3
N5K-1 N5K-2
Frames received on the vPC-Link are not forwarded out vPC ports
1. 2. 3. Host MAC_A send packet to MAC_C FEX runs hash algorithm to select one fabric uplink N5K-1 learns MAC_A and flood packets to all ports (in that VLAN). A copy of the packet is sent across the peer link N5K-2 floods the packet to any port in the VLAN except the vPC member ports to prevent duplicated packets N7K-1 and N7K-2 repeat the same forwarding logic N5K-1 updates the the MAC address learned on the vPC port on N5K-2 via CFS
CFS
4.
1
MAC_A
5. 6.
Cisco Confidential
24
MAC_C
Traffic is forwarded if destination address is
known (both switches MAC address tables populated)

Always forward via a locally attached member of
a vPC if it exists 1. Host MAC_C send packet to MAC_A 2. N7K-2 forwards frame based on learned MAC address 3. N5K-2 forwards frame based on learned MAC address
N5K-1# sh mac-address-table vlan 101 VLAN MAC Address Type Age Port ---------+-----------------+-------+---------+----101 001b.0cdd.387f dynamic 0 Po30 101 0023.ac64.dda5 dynamic 30 Po201 Total MAC Addresses: 4
2
N5K-1 N5K-2
3
MAC_A
Cisco Confidential
25
MAC_C
On loss of all of the locally attached
members of the vPC MAC address table is updated to forward frames for the vPC across the vPC Peer Link
Note: Po20 is the vpc peer-link
2
N5K-1 N5K-2
3
MAC_A
Cisco Confidential
26

Both switches in the vPC Domain maintain
distinct control planes

CFS provides for protocol state synchronization
between both peers (MAC Address table, IGMP state, )

System configuration must also be kept in sync
Currently there are 2 options to keep
configuration consistent: a manual process with an automated consistency check to ensure correct network behavior config-sync Two types of interface consistency checks
Type 1 Will put interfaces into suspend state to prevent invalid forwarding of packets
Type 2 Error messages to indicate potential for undesired forwarding behavior
Cisco Confidential
28
Type 1 Consistency Checks are intended to
prevent network failures

Incorrectly forwarding of traffic Physical network incompatibilities vPC will be suspended
dc11-5020-1# sh run int po 201 interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201 dc11-5020-2# sh run int po 201 interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201 spanning-tree guard root
dc11-5020-2# show vpc brief Legend: (*) - local vPC is down, forwarding via vPC peer-link <snip> vPC status ---------------------------------------------------------------------------id Port Status Consistency Reason Active vlans ------ ----------- ------ ----------- -------------------------- ----------201 Po201 down failed vPC type-1 configuration incompatible - STP interface port guard Root or loop guard 2010 Cisco and/or its affiliates. All rights reserved. inconsistent
Cisco Confidential
29
Type 2 Consistency Checks are intended to
prevent undesired forwarding

vPC will be modified in certain cases (e.g.
VLAN mismatch)
dc11-5020-1# sh run int po 201 version 4.1(3)N1(1) interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201 dc11-5020-2# sh run int po 201 version 4.1(3)N1(1) interface port-channel201 switchport mode trunk switchport trunk native vlan 105 switchport trunk allowed vlan 100-104 vpc 201
dc11-5020-1# show vpc brief vpc 201 vPC status ---------------------------------------------------------------------------id Port Status Consistency Reason Active vlans ------ ----------- ------ ----------- -------------------------- ----------201 Po201 up success success 100-104 2009 May 17 21:56:28 dc11-5020-1 %ETHPORT-5-IF_ERROR_VLANS_SUSPENDED: VLANs 105 on Interface portchannel201 are being suspended. (Reason: Vlan is not configured on remote vPC interface)
Cisco Confidential
30
c-nexus5010-1# show vpc consistency-parameters global Legend: Type 1 : vPC will be suspended in case of mismatch Name ------------QoS Network QoS (MTU) Network Qos (Pause) Type ---2 2 2 Local Value Peer Value
Global Parameters
---------------------- ----------------------([], [3], [], [], [], (1538, 2240, 0, 0, 0, (F, T, F, F, F, F) ([], [3], [], [], [], (1538, 2240, 0, 0, 0, (F, T, F, F, F, F)
STP Mode
STP Disabled STP MST Region Name STP MST Region Revision STP MST Region Instance to VLAN Mapping STP Loopguard
1
1 1 1 1
Rapid-PVST
None "" 0
Rapid-PVST
None "" 0
Global QoS Parameters need to be consistent
Disabled
Disabled
STP Bridge Assurance

STP Port Type, Edge Allowed VLANs Local suspended VLANs
1
1 -
Enabled
Normal, Disabled, 1,50 50
Enabled
Normal, Disabled, 1 -
Global Spanning Tree Parameters need to be consistent
Cisco Confidential
31
Dont forget to keep global configuration in
sync
Any configuration that could cause an error in
forwarding (e.g. loop) will disable all affected interfaces

As an example if you make a change to an
MST region you must make it on both peers

Solution: define MST region mappings from
mst region vlans 1-5, 12
the very beginning of the deployment, for ALL VLANs, the ones that exist as well as the ones that have not yet been created
Defining a region mapping is orthogonal to
vPC vPC vPC
creating a VLAN
Cisco Confidential
33

Adding FEX
Summary designs
Cisco Confidential
34
Inconsistency VLAN to MST Region mapping mismatch STP global settings (BA, Loop Guard, Root Guard)
Type
Impact
Recommendation Pre-provision and MAP all VLANs on the MST region Perform STP operations per port Operate change during maintenance window Leverage graceful conflict resolution Operate change during maintenance window and/or leverage graceful conflict resolution
New Enhancements
Global
1
Spanning-tree per interface settings, switchport type (trunk/versus access) Port-channel mode
Config Sync (5.0(2)N1(1) on N5K, Freetown for N7K) & Graceful Conflict Resolution (CSCtf84865,N7K 4.2(8)& 5.2, N5K 5.0(2)N2(1))
Per-vPC
Quality of Service Configuration 2 VLANs configured on vPC
Global
Minimum disruption Per-vPC
Cisco Confidential
35
tc-nexus5010-1# show vpc consistency-parameters global Name Type Local Value Peer Value
------------QoS
---2
---------------------- ----------------------([], [3], [], [], [], []) ([], [3], [], [], [], []) (1538, 2240, 0, 0, 0, 0) (F, T, F, F, F, F) (50, 50, 0, 0, 0, 0) (F, F, F, F, F, F)
Network QoS (MTU)
(1538, 2240, 0, 0, 0, 0)
Network Qos (Pause) Input Queuing (Bandwidth) Input Queuing (Absolute Priority) Output Queuing (Bandwidth) Output Queuing (Absolute
2 2 2
(F, T, F, F, F, F) (50, 50, 0, 0, 0, 0) (F, F, F, F, F, F)
2 2
(50, 50, 0, 0, 0, 0) (F, F, F, F, F, F)
(50, 50, 0, 0, 0, 0) (F, F, F, F, F, F)
Cisco Confidential
36
With Graceful Resolution only ports on the
vPC secondary are suspended if a Type-1 global inconsistency occurs

This limits the impact of configuration
changes.
switch(config)# vpc domain 10 switch(config-vpc-domain)# [no] graceful
mst region vlans 1-5, 12 mst region vlans 1-5, 10
consistency-check
Requires 5.0(2)N2(1) on the Nexus 5k Requires 5.2 on the Nexus 7k
vPC vPC vPC
vPC primary
vPC secondary
Cisco Confidential
38
5.2
5.0(2)N2(1)
Check whether STP is enabled or disabled on per-VLAN basis. VLANs that have mismatched status will be suspended on both switches
Disable STP on VLAN 5
Rest of VLANs wont be affected Prior to this change all VLANs are affected
Cisco Confidential
39
Config-sync allows administrators to make configuration changes on one switch and have the system automatically synchronize to its peers. This eliminates any user prone errors & reduces the administrative overhead of having to configure both vPC members simultaneously. Config-sync and Graceful conflict resolution are complementary features Config-sync traffic is carried over the peer keepalive link
vPC vPC
+ vlan 12
mst region vlans 1-5
vPC
Cisco Confidential
40
Global Configurations:
Which configurations are not
VLANs ACLs STP configurations QOS

Interface Level Configurations:
synchronized?
Enabling Feature
vPC domain configuration FCoE configuration
Ethernet Interfaces Port Channel Interfaces vPC Interfaces
Cisco Confidential
41
N5000-1# feature vpc vpc domain 10 peer-keepalive destination 10.29.170.8 N5000-1#sh run switch-profile Switch-profile Apple sync-peers destination 10.29.170.8 N5000-1(config-if)# config sync N5000-1(config-sync)# switch-profile Apple N5000-1(config-sync-sp)# int ethernet 100/1/3 N5000-1(config-sync-sp-if)# switch mode trunk N5000-1(config-sync-sp-if)# verify Verify Successful N5000-1(config-if)# config sync N5000-1(config-sync)# switch-profile Apple N5000-1(config-sync-sp)# commit Commit Successful
N5000-2# feature vpc vpc domain 10 peer-keepalive destination 10.29.170.7 N5000-2#sh run switch-profile Switch-profile Apple sync-peers destination 10.29.170.7
NOTE: Verify does not push the config to peer, user must issue commit for sync to take place If sync fails, then the config is in the BUFFER
N5000-1#sh run switch-profile interface ethernet 100/1/3 switchport mode trunk

N5000-2#sh run switch-profile interface ethernet 100/1/3 switchport mode trunk

Configuration is stored in a buffer until commit is applied. User can add/delete/move configuration. Once the config has been pushed via commit, it will no longer show up in buffer (it will show up in show runningconfig switch-profile X) If the commit fails due to mutex check or other reasons, the failed configuration still shows in the buffer, you have to explicitly remove it to continue
N5K-1(config-sync-sp-if)# sh switch-profile A buffer ----------------------------------------------------Seq-no Command ----------------------------------------------------1 interface Ethernet100/1/9 1.1 switchport mode trunk 1.2 switchport trunk allowed vlan 5-10 2 interface Ethernet100/1/10 2.1 switchport mode access
N5K-1(config-sync-sp)# ? buffer-delete Delete buffered command(s) buffer-move Move buffered command(s) N5K-1(config-sync-sp)# buffer-delete 1 N5K-1(config-sync-sp)# sh switch-profile A buffer ----------------------------------------------------Seq-no Command ----------------------------------------------------2 interface Ethernet100/1/10 2.1 switchport mode access
Cisco Confidential
43
Interface Ethernet1/11
config-t area This portion is not synchronized
fex associate 100 switchport mode fex-fabric channel-group 100
Interface Ethernet1/11
switch-profile area
shut/no shut
This portion is synchronized
Cisco Confidential
44
A port-channel may consist of port ethernet 1/1 on n5k01 And erthernet 1/2 on n5k02 FEX A/A has the same FEX configured to both N5ks, so Preprovisioning has to be configured identically
Cisco Confidential
45
If one vPC peer needs to
be disconnected completely from the vPC domain you can still operate the remaining one For this you need to leverage the commands reload restore and autorecovery
Reload restore deals
with the split brain scenario allowing a vPC peer to bring up new vPC ports even after a reload Autorecovery deals with the sequential loss of peer-link first, and peer-keepalive second, allowing the vPC secondary to bring up the vPC ports (which were down previously)
Cisco Confidential
46
VPC needs to be able to talk to the peer (over peer-link) before
bringing up VPC port-channels

Negotiate LACP/STP operating roles for the chassis
Wait for per-port peer parameters and handshake to bring up vPC ports
Performs peer parameters consistency check on each VPC bringup Only after VPC port-channels are brought up. What if after a full DC outage (both Nexus down), only one switch is
coming up ?
Will not bring up VPCs if after a datacenter outage, only one VPC
peer comes back up
Cisco Confidential
47
Switch1
Switch2
Switch1
Switch1
Switch3
Switch3
Switch3
Existing vPCs are brought up When adding a new vPC member port, the port goes up
1 Switch1 Switch2
Switch1
Switch2
Switch3
Switch1
Switch2
Switch3
Switch3
Keepalive S1 -Primary vPC peer-link S2-Secondary S1 -Primary
Keepalive S2-Secondary vPC peer-link
vPC 1
po1
vPC 1
po1
Peerlink down and keepalive working Secondary shuts vPCs
Primary fails Po1 is completely shut

Keepalive vPC Primary vPC peer-link vPC Secondary
vPC Operational Primary Keepalive
vPC peer-link
vPC 1
po1
po1
vPC 1
After 3 consecutive keepalive timeouts
Peerlink down and keepalive down
Secondary changes role and brings up vPCs
Cisco Confidential
51
STP for vPCs is controlled by
Primary vPC
Secondary vPC
the vPC operationally primary switch and only such device sends out BPDUs on STP designated ports.
This happens irrespectively of where
BPDUs
the designated STP Root is located

The vPC operationally secondary
device proxies STP BPDU messages from access switches toward the primary vPC
Cisco Confidential
52
vPC peer-link is a regular STP port

ECMP
Primary vPC
ECMP
Secondary vPC
SW1
Primary Root
L3 L2
vPC PK-Link vPC_PL
SW2
Secondary Root
vPC Primary Switch Source and controls STP for vPCs

vPC1 vPC2
The secondary vPC device does NOT source STP BPDUs on symmetrical vPCs
SW3 MAC_A
SW4 MAC_B
Cisco Confidential
53
Assume the following topology with
vPC enabled on the vPC

If the Primary fails over, the
Primary / Root
Secondary becomes primary and root
secondary needs to start sending BPDUs

If the Primary was also the STP root,
the secondary also has to overtake the role as a root

If this process lasts too long, the
7k01
7k02
uplink port on 5k02 may go into BA_Inconsistent state

Better not use Bridge Assurance
BPDUs prior to the failure

5k01 5k02
BA Inconsistent
with vPC
Bridge Assurance on peer-link is
fine (and is the default)
Cisco Confidential
54
Primary
left# sh span vlan 101 VLAN0101 Spanning tree enabled protocol rstp Root ID Priority 8293 Address 0023.04ee.be01 This bridge is the root ... Bridge ID ... Interface ---------------Po1 Po100 Role ---Desg Root Sts --FWD FWD Cost --------1 2 Prio.Nbr -------128.4096 128.4195 Type --------------(vPC) P2p (vPC peer-link) Priority Address 8293 (priority 8192) 0023.04ee.be01
Secondary ROOT
ROOT
left# sh vpc role | i mac vPC system-mac vPC local system-mac
: 00:23:04:ee:be:01 : 00:1b:54:c2:42:43
right# sh span vlan 101 VLAN0101 Spanning tree enabled protocol rstp Root ID Priority 8293 Address 0023.04ee.be01 This bridge is the root ... Bridge ID Priority Address Role ---Desg Desg 8293 (priority 8192) 0023.04ee.be01 Sts --FWD FWD Cost --------1 2 Prio.Nbr -------128.4096 128.4195 Type --------------(vPC) P2p (vPC peer-link)
In Peer-Switch mode bridge-ID comes from system-mac as opposed to local mac in normal mode
... Interface ---------------Po1 Po100
BA is default enabled on Peer-Link (and recommended to remain enable), not
recommended for VPCs unless Peer-Switch feature is used

Without Peer-switch BA should be kept only on Peer-Link (no BA/Loop guard on VPCs)
Dispute is default enabled (for both RSTP and MST on VPC) UDLD [normal mode] is recommended to take out bad links from channels BA + UDLD + Dispute (on all interswitch links when using Peer-switch) when all
switches support this (nexus7000/5000)
Cisco Confidential
56
By default on the Nexus 5x00 series, LACP sets a port to I state if it
does not receive an LACP PDU from the peer. This behavior is different on the Nexus 7000 series where the default is to suspend a port if it doesnt receive LACP PDUs.
For server facing port-channels it is better to allow LACP ports to revert
to I-state if the server doesnt send LACP PDUs. By doing this the I-state port can operate like a regular Spanning-Tree port. Also this allows immediate server connectivity when it boots up before the full LACP negotiation has taken place.
For network facing ports, allowing ports to revert to I-state creates
additional Spanning-tree state without any real benefit.

This behavior can be configured on a per Port-Channel basis with the
configuration [no] lacp suspend-individual (which is the equivalent of the Catalyst IOS command port-channel standalone-disable.
57
Cisco Confidential
IGMP snooping shares the snooped reports with the peer vPC
switch to help with multicast forwarding

Forwarding of IGMP protocol packets is tweaked so that IGMP
reports received on one vPC switch is also forwarded to the vPC peer. Thus multicast forwarding state remains in sync on both the vPC switches.
Do NOT DISABLE IGMP Snooping!
If you need to support Firewalls / Clusters: Use static IGMP entries OR Create an IGMP querier!
Cisco Confidential
58
vPC maintains dual active control
planes and STP still runs on both switches

IGMP join/leave messages received
vPC Primary
vPC Secondary
on one peer is forwarded to another peer via peer link

IP muticast packets are sent to host
through local port

Non-IP multicast and broadcast
vPC Primary
vPC Secondary
packets are flooded

IGMP join/leave
IGMP join/leave
Cisco Confidential
59
So is the multicast traffic going
to the peer link?

Yes, but duplicates are avoided by using the vPC loop prevention technique, which should rather be called duplicate prevention
N7k01 N7k02
And how about orphan ports?

Orphan ports receive traffic because the multicast traffic is always sent over the peer-link
N5k01
N5k02
Cisco Confidential
60
Assuming that there are no orphan ports it is possible to remove
multicast traffic from crossing the peer-link with the command

no ip igmp snooping mrouter vpc-peer-link (Nexus 5k) ip igmp snooping vpc peer-link-exclude (hidden command on the
Nexus 7k, not supported)
Cisco Confidential
61
VPC peer-link is considered as mrouter
port. Therefore all multicast traffic is flooded over peer-link

A CLI was introduced in 5.0(3)N1(1) to
N5k-1
N5k-2
avoid that. With the CLI multicast traffic is sent to vPC peer-link only when it is necessary, such as, there is singly connected host
Improving multicast convergence time
IGMP Group sync
with peer-link down/up and switch reload

The CLI is not supported for FEX dual-
home topology in 5.0(3)N1(1). The limitation will be removed in upcoming release-5.0(3)N2(1)
Cisco Confidential
62
If the peer-link is lost the vPC
vPC on the N7k N7k01 N7k02
secondary is going to shut down the vPC member ports

For single attached hosts, pls
see
CSCtc49559
1 2 3 4
and Orphan ports suspend
feature
vPC on the N5k
N5k01
N5k02
Cisco Confidential
63
S1 -Primary
Keepalive
S2-Secondary
Intended for devices that do not support port-channel. Other devices should be dually connected by vPCs (Orphanport CLI is available only on physical ports, not on portchannels) Configure single attached devices (like FW or LB) port as orphan-port When vPC peer-link goes down, vPC secondary peer device shuts all its vPC member ports as well as orphan ports
vPC peer-link
Active or Standby Active or Standby
vPC 1
po1 CE-1 Orphan port
S1(config)# int eth 1/1 S1(config-if)# vpc orphan-ports suspend S2(config)# int eth 1/1 S2(config-if)# vpc orphan-ports suspend
Cisco Confidential
64

Adding FEX
Summary designs
Cisco Confidential
66
vPC maintains dual active control
HW Programmed to forward frames sent to the FHRP MAC address on BOTH Switches
HSRP Active HSRP Standby
planes and STP still runs on both switches

HSRP active process communicates
the active MAC to its neighbour

Only the HSRP active process
responds to ARP requests

HSRP active MAC is populated into
the L3 hardware forwarding tables, creating a local forwarding capability on the HSRP standby device
Consistent behavior for HSRP, VRRP
and GLBP
No need to configure aggressive
FHRP hello timers as both switches are active
Cisco Confidential
67
It recommended to not use HSRP link tracking in a vPC configuration Reason: vPC will not forward a packet back on a vPC once it has crossed the peer-link, except in the case of a remote member port failure Use an L3 point-to-point link between the vPC peers to establish a L3 backup path to the Core in case of uplinks failure A single point-to-point VLAN/SVI will suffice to establish a L3 neighbor
SVI VLAN 300
VLAN 100, 200,300
SVI VLAN 300
VLAN 100
VLAN 200
Cisco Confidential
68
Non-RFC compliant end hosts Device required to send packets to the MAC address returned in ARP response (HSRP virtual MAC) Some non-compliant devices use the MAC address of the sender device (Switch physical MAC) NAS devices (i.e. NETAPP Fast-Path or EMC IP-Reflect) have been found to do this vPC Peer Gateway - NX-OS 4.2(1) Allows a vPC peer to respond both the the HSRP virtual and the real MAC address of both itself and its peer
peer-gateway command tells the vPC to respond to the physical MAC address of its peer
L3 L2
VLAN 200
VLAN 100
Cisco Confidential
69
Not enabled by default After the peer-link comes up perform an ARP bulk sync over CFSoE to the peer switch Improve Convergence for Layer 3 flows
ARP TABLE
IP1 IP2 MAC1 MAC2 VLAN 100 VLAN 200 IP1 IP2
ARP TABLE
MAC1 MAC2 VLAN 100 VLAN 200
SVIs
P
S
S1(config-vpc-domain)# ip arp synchronize

P S
S2(config-vpc-domain)# ip arp synchronize

Primary vPC Secondary vPC
IP1
MAC1
IP2
MAC2
Note: CSCti06907 has been fixed
ARP Synchronization Process

Feature
VPC interaction with FHRP Peer-gateway vPC delay restore vPC exclude VLAN ARP synchronization PIM pre-built-SPT PIM dual DR
Function
Both active and standby peer function as gateway L3 forwarding when the DMAC is peers MAC Delay bringing up vPC ports CLI to specify SVI interfaces wont be suspended when peer-link fails Synchronize ARP between two peer switches Both N5k joins source tree as PIM last hop router Both N5k can be DR when it is first hop router
Availability
HSRP VRRP
Roadmap
Cisco Confidential
71

Adding FEX
Summary designs
Cisco Confidential
72
2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
FEX 2248, 2232, 2224 from 4.2(1)N1(1)

73
FEX2148T starting from 4.1(3)N1(1) FEX 2248, 2232 from 4.2(1)N1(1)
Fairhaven
Cisco Confidential
74
N7K NXOS 5.1(1)
N7K NXOS 5.2
Future
active
active
active
active
active
active
active
active
active
active
active
active
N
radar
Cisco Confidential
75
Nexus 5000 Topologies (Nexus 2248TP & 2232PP)

Straight Through Dual Homed
vPC Supported with up to 2 x 8 links
FCoE Adapters supported on 10G N2K interfaces
Local Etherchannel with up to 8 links
Redundancy model Dual Switch with redundant fabric
Provides isolation for Storage topologies (SAN A and B)

Port Channel and Pinning supported for Fabric Link
Redundancy model Single switch with dual supervisor for fabric, data control & management planes No SAN A and B isolation (VSAN isolation sufficient in the future?)
Cisco Confidential
76
Nexus 7000 Topologies (Nexus 2248TP & 2232PP)

NXOS 5.2 Nexus 2248TP & 2232PP
NIC Teaming: TLB/ALB
Local Etherchannel with up to 8 links
Fabric links supported on N7K-M132XP-12 & N7K-M132XP-12L Port Channel only supported for Fabric Links
Local port channel support on 2248 & 2232
No support for DCB and FCoE (parent switch fabric ports not DCB capable yet)
Nexus 7000 - vPC NXOS 5.2
Nexus 5000 Fairhaven
MCEC Etherchannel with up to 16 links
MCEC Etherchannel with up to 16 links
Redundancy model Dual Switch (each switch supports redundant supervisors)
Redundancy model Single switch with dual supervisor, fabric, line card, data control & management planes
Cisco Confidential
78
24 FEX
Nexus 2000 Straight-through deployment
n5k01
n5k01
max 4/8 fabric links
n5k02
FEX100 FEX101
FEX102
FEX120
FEX121
FEX122
FEX100 FEX101
FEX102
max 24 x 2
max 24 with Nexus 5500 = 768 ports
Active/Standby
Cisco Confidential
79
Peer Keepalive
FEX 2248
Cisco Nexus 2000 Series Straight-Through vPC Cisco Nexus 2000 Active-Active
Peer Link vPC Member Port
vPC Primary
vPC Secondary
vPC Primary
vPC Secondary
up to 4 ports
up to 4 ports
Fabric Links
Fabric Links
up to 4 ports up to 4 ports
FEX100
HIF
up to 8 ports
FEX120
HIF
up to 8 ports up to 24 PC per FEX up to 24 PC per FEX
vPC 1
vPC 2
FEX100
HIF
FEX120
HIF
up to 8 ports
up to 8 ports
Cisco Confidential
80
Peer Keepalive
FEX 2232
Cisco Nexus 2000 Series Straight-Through vPC Cisco Nexus 2000 Active-Active
Peer Link vPC Member Port
vPC Primary
vPC Secondary
vPC Primary
vPC Secondary
up to 8 ports
up to 8 ports
Fabric Links
Fabric Links
up to 8 ports up to 8 ports
FEX100
HIF
up to 8 ports
FEX120
HIF
up to 8 ports up to 16 PC per FEX up to 16 PC per FEX
vPC 1
vPC 2
FEX100
HIF
FEX120
HIF
up to 8 ports
up to 8 ports
Compatible with FCoE IF server uses 2 uplinks

Doesnt support FCoE, today

SAN A
SAN B
In a Dual Tier vPC configuration
FCoE traffic will NOT be load shared across both sets of fabric links
SAN A and B isolation is
LAN traffic
maintained
This may result in un-even sharing
of traffic across the multiple fabric links

FCoE + LAN on one set of fabric links LAN only on the other set of fabric links
Need to plan for the aggregate
LAN & SAN traffic
traffic capacity
Cisco Confidential
82

Adding FEX
Summary designs
Cisco Confidential
83
root
vPC on the N7k Root
N7k01 2/9 2/10 2/9
N7k02 2/10
logical equivalent
2/1 Po51,2 N5k01
2/2
2/1
2/2
N5k02
Cisco Confidential
84
root
vPC on the N7k Root
N7k01 2/9 2/10 2/9
N7k02 2/10
logical equivalent
2/1 Po51 N5k01
2/2 Po10
2/1
2/2
Peer Link
N5k02
primary
secondary
regular STP priority

16 port HW Etherchan nel
vPC peer link
7010s with F1linecards

x8 32 PORTS
x8
x8
x8
16 port HW Etherchan nel
vPC peer link Running vPC only for server attach ports
5500 or 50x0
...
...
2248TPs
Cisco Confidential
86
Clear access VLANs to create a Loop Free Topology
logical equivalent
Root HSRP primary
Secondary Root HSRP secondary
Root HSRP primary
SW01 2/9 2/10 2/9
SW02 SW01 2/10 2/9
SW02 2/10 2/9 2/10
2/1 Po51 N5k01
2/2 Po10 Peer Link
2/1
2/2
N5k02
primary
secondary
regular STP priority

Traffic flows are symmetric from access
to aggregation
vPC is still useful to optimize traffic
Root HSRP primary
flows from access to aggregation

All traffic flows through the active HSRP
SW01 2/9 2/10 2/9
SW02
switch, in this case SW01

Client-to-Server traffic uses both SW01
2/10
and SW02
Peer-link is almost unutilized
Po51
2/1
2/2
Po10
2/1
2/2
N5k01
Peer Link
N5k02
Cisco Confidential
88
Following steps are needed to build a vPC
Define domains
Establish Peer Keepalive connectivity Create a Peer link Create vPCs Make Sure Configurations are
N7k01
N7k02
Consistent / leverage config-sync / configure graceful conflict resolution
5 6 N2k01
7 8 N2k02
Cisco Confidential
89
Ensure domain-id or system-mac differs
N7k01
N7k02
between Agg pair and Access pair

Connect the N7ks with redundant peer-links
across linecards
Connect the N5ks with redundant peer-links Create a single Port-channel leveraging LACP
1
LACP
2 3 4
between Aggregation and Access

Do not forget that putting a VLAN on a vPC
N5k01 5 6 N2k01 N5k02 7 8 N2k02
requires that that VLAN be on the Peer-link too

If you foresee significant multicast traffic, or
there is a high percentage of single attached devices, you may want to size the peer-link to match the uplink bandwidth utilization
Cisco Confidential
90
If you use the peer switch functionality, then define Identical Priorities on the Aggregation Layer switches, to make them the root Do not use Bridge Assurance Keep the default STP priorities on the access layer switches IF using MST, Make Sure that VLAN range Configurations are Consistent With MST be aware of the NXOS VLAN range and of the Global Type-1 Inconsistencies, hence configure VLAN-to-region mappings from day 1 Use pathcost method long Configure STP port type edge or port type edge trunk
N7k01
N7k02
5 6 N2k01
7 8 N2k02
Cisco Confidential
91
Configure HSRP priorities as usual, both
peers forward L3 traffic

N7k01 N7k02
Configure vPC delay restore to avoid L3
traffic loss upon reboot

Create a L3 backup link between the N7k Configure peer-gateway for firewalls, load
balancers, filers
Configure regular L3 ECMP from the core to
the aggregation layer
N5k01 5 6 N2k01
N5k02 7 8 N2k02
1
3
Make sure to leverage Reload
N7k01 2/9 2/10 2/9
N7k02 2/10
Restore and auto-recovery

Make sure to have mgmt0
connectivity for config-sync to work (you may want to use the same mgmt0 for vPC peer keepalive)
FEX A/A provides redundancy and
2/1
2/2
2/1
2/2
each HIF
Config-sync also helps with regular
N5k01
N5k02
port channels
FEX pre-provisioning is highly
recommended
Cisco Confidential
93
Thank You

Cisco - VPC Concepts

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cisco - VPC Concepts

Uploaded by

Copyright:

Available Formats

vPC Best Practices with Nexus

SAVBU TME Team

2010 Cisco and/or its affiliates. All rights reserved.

E-Rocks Nexus 3000

Complete sync done at major releases

Partial sync done at minor releases

2010 Cisco and/or its affiliates. All rights reserved.

vPC is a Port-channeling concept extending link

aggregation to two separate physical switches

Provides increased bandwidth

Virtual Port Channel

vPC peer keepalive link

vPC peer link

channels) that form a vPC

vPC peers and the downstream device vPC peer

vPC member port

vPC peer keepalive link the peer keepalive

link between vPC peer switches. It is used to carry heartbeat packets

2010 Cisco and/or its affiliates. All rights reserved.

Graceful consistency check:

Per VLAN consistency check:

FEX preprovisioning: On N7k: Freetown On N5k: NXOS 5.0(2)N1(1)

Dual Layer vPC: On N7k: TBD On N5k: Fairhaven

Orphan Ports shutdown:

2010 Cisco and/or its affiliates. All rights reserved.

vPC allows a single device to use a port

channel across two neighbor switches (vPC peers)

Portchannel vPC Peers

2010 Cisco and/or its affiliates. All rights reserved.

Peer Link carries both vPC data and control

traffic between peer switches

Minimum 2 x 10GbE ports ALL VLANS used on vPC PORTS MUST BE

PRESENT ON THE PEER-LINK

2010 Cisco and/or its affiliates. All rights reserved.

Peer Keep-alive provides and out of band heartbeat

between vPC peers

Purpose is to detect and resolve roles if a Split Brain

(Dual Active) occurs timeout

Messages sent on 1 second interval with 5 second

(first 16 ports on 5020 are 1/10GE ports)

3rd option, use a routed inband connection over L3

infrastructure (using SVIs in the default VRF)

2010 Cisco and/or its affiliates. All rights reserved.

Peer keep-alive is a routable protocol (both N5K

Standby Management Interface Active Management Interface

different path than all other vPC traffic

supervisor management interfaces back to back

vPC basic components Hardware Specific Considerations

vPC forwarding rules

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Nexus 5000 Series

1st option dedicated front panel port in dedicated VRF.

2nd option is management interface.

peer-link should be made on either all F1 cards or all M1 cards

2010 Cisco and/or its affiliates. All rights reserved.

NEXUS 7000 I/O modules

N7K-M148GT-11 N7K-M148GT-11L N7K-M148GS-11 N7K-M148GS-11L

2010 Cisco and/or its affiliates. All rights reserved.

Mixed Chassis Mode

F-Series Mode vPC Peer-link on F-Series Modules

Mixed Chassis Mode vPC Peer-link on F-Series Modules (*)

vPC Peer-link on M-Series Modules

vPC Peer-link on M-Series Modules