VSS VS VPC PDF

VPC & VSS: Operation and Troubleshooting
BRKCRS-1930
VSS and VPC

enable us to build EtherChannel to 2 separate
switches and transform network building block
from this
to this
or, logically
No blocked ports, More usable bandwidth, Load-sharing

Distribution or link failure != network reconvergence
BRKCRS-1930
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Goals
Understand general
concepts of VPC on Nexus
7000 and VSS on Catalyst
6500
Study the impact of VPC and
VSS on bridging and routing
Learn how to troubleshoot
VPC and VSS
BRKCRS-1930
Cisco Public
Spirit of this session
Simple description on how things work

Special cases
Troubleshooting
More on the topic

Cisco Catalyst Virtual Switching System
(BRKCRS-3468)
Advanced Enterprise Campus Design: Virtual Switching System
(BRKCRS-3035)
Deploying Virtual Port Channel in NXOS
(BRKDCT-2048)
BRKCRS-1930
Cisco Public
VSS
BRKCRS-1930
Cisco Public
VSS Agenda
Initialization
Internal redundancy considerations
Spanning Tree
1st hop redundancy

Traffic forwarding
Multicast considerations
BRKCRS-1930
Cisco Public
VSS
1 active redundant control plane
single config
single point of management
VSS domain
2 active data planes
Active
Standby
Active
Control Plane
Standby switch is essentially a

set of additional linecards
Control messages and Data
frames flow between active and
standby via VSL
Standby
VSL
Control Plane
Active
Active
Data Plane
Data Plane
Dual-Active
detection link
MEC
(can be seen as backplane

extension)
Special encapsulation on VSL

frames to carry additional
information
BRKCRS-1930
Cisco Public
VSS initialization
Before the Virtual Switch domain can become active, the Virtual Switch Link
(VSL) must be brought online to determine Active and Standby roles. The
initialization process essentially consists of 3 steps:
1
Link Bringup to establish connectivity with remote chassis
Link Management Protocol (LMP) used to track and reject Unidirectional Links,
Exchange Chassis ID and other information between the 2 switches
LMP
LMP
RRP
RRP
Role Resolution Protocol (RRP) used to determine compatible Hardware and

Software versions to form the VSL as well as determine which switch becomes
Active and Hot Standby from a control plane perspective
BRKCRS-1930
Cisco Public
Troubleshooting VSS: quick sanity check

vss# sh switch virtual
Switch mode
:
Virtual switch domain number :
Local switch number
:
Local switch operational role:
Peer switch number
:
Virtual Switch
111
1
Virtual Switch Active
2
vss# sh switch virtual link

VSL Status : UP
VSL Uptime : 18 hours, 38 minutes
VSL SCP Ping : Pass
VSL ICC Ping : Pass
VSL Control Link : Te1/6/1
In VSS mode?
Domain# unique for each VSS?
Role of this switch
Peer-switch visible?
VSL is up?
Link used to carry control plane
messages (ICC, IPC, SCP)
VSL member-links state
Redundancy mode SSO?
vss# sh switch virtual link port

LMP summary
Link info:
Configured: 2
Operational: 1
Peer Peer
Peer
Peer
Timer(s)running
Interface Flag State
Flag MAC
Switch Interface (Time remaining)
-------------------------------------------------------------------------------Te1/5/4
v
link_down
Te1/6/1
vfs operational vfs 0007.0d72.4800 2
Te2/6/1
T4(960ms)
T5(29.98s)
...
vss# sh redundancy states
my state = 13 -ACTIVE
peer state = 4 -STANDBY COLD
Mode = Duplex
...
BRKCRS-1930
Cisco Public
Aside from packet/bit rate this is

one-stop-shop command for VSL
packet and error counters
Always take 2-3 samples
All errors should be at or near zero
and most importantly not
incrementing (giants are ok)
Troubleshooting VSL:
counters
vss# sh switch virtual link counters

Port
Po10
Te1/6/4
Te1/6/5
InOctets
3084500343
523470151
2814244020
InUcastPkts
31059
139662
11346
InMcastPkts
7382085
1323349
6883221
InBcastPkts
1046088
1045940
258
Port
Po10
Te1/6/4
Te1/6/5
...
OutOctets
1457635126
363835687
1214900160
OutUcastPkts
1467466
264788
1202788
OutMcastPkts
9890548
2732502
8103037
OutBcastPkts
0
0
0
Port
Po10
Te1/6/4
Te1/6/5
Port
Po10
Te1/6/4
Te1/6/5
Port
Po10
Te1/6/4
Te1/6/5
BRKCRS-1930
Align-Err
0
0
0
Single-Col
0
0
0
SQETest-Err
0
0
0
FCS-Err
0
0
0
Multi-Col
0
0
0
Deferred-Tx
0
0
0
Cisco Public
Xmit-Err
0
0
0
Late-Col
0
0
0
IntMacTx-Err
0
0
0
...
...
...
...
...
...
...
...
...
...
...
...
10
Complete information about LMP

layer of VSLP
At least 1 link should be operational
vss# sh switch virtual link detail
Should see a neighbor
...
LMP summary
Should not see any events except
...
t4_exp (hello tx timer expiry)
LMP neighbors
Non-zero (low number) error
Peer Group info:
# Groups: 1
(* => Preferred
PG) are acceptable as long as
counters
PG #
MAC
Switch Ctrl Interface Interfaces they do not increment (take 2-3
snapshots)
---------------------------------------------------------------
Troubleshooting VSL: LMP
*1
0004.9bbe.ac00
...
LMP hello timer
...
LMP FSM info
Te1/6/4
Te1/6/4, Te1/6/5
sm(vslp_lmp 6/4), running yes, state operational

Last transition recorded: (hello)-> operational (t4_exp)-> operational (hello)->
operational (hello)-> operational (t4_exp)-> operational (hello)-> operational
...
LMP counters
Tx
Rx
Interface
OK
Fail
Bidir
Uni
Fail
Bad
-------------------------------------------------------------------Te1/6/4
805969 0
806270 7
0
0
Te1/6/5
640674 0
640726 3
0
0
Rx error details
Interface My info
My info
Bad MAC
Bad switch Domain id
Peer info
mismatch
absent
Address
id
mismatch
mismatch
------------------------------------------------------------------------------Te1/6/4
0
7
0
0
0
0
Te1/6/5
0
3 its affiliates. All rights 0reserved.
0
0
BRKCRS-1930
Cisco0
Public
2011 Cisco and/or
11
Troubleshooting VSL: LMP

vss# sh switch virtual link port
LMP summary
Link info:
Configured: 2
Compared to previous command

this one provides details of the
previous failure (if there was any) of
VSL links
Rest of the information is identical
Operational: 2
Peer Peer
Peer
Peer
Timer(s)running
Interface Flag State
Flag MAC
Switch Interface (Time remaining)
-------------------------------------------------------------------------------Te1/6/4
vfsp operational vfsp 0004.9bbe.ac00 2
Te2/6/4
T4(756ms)
T5(29.98s)
Te1/6/5
vfsp operational vfsp 0004.9bbe.ac00 2
Te2/6/5
T4(756ms)
T5(29.92s)
Flags:
v - Valid flag set

s - Negotiation flag set
Timers: T4 - Hello Tx Timer
f - Bi-directional flag set

p - Peer detected flag set
T5 - Hello Rx Timer
LMP Status
Last operational
Current packet
Last Diag
Time since
Interface Failure state
State
Result
Last Diag
------------------------------------------------------------------------------Te1/6/4
Link down
Hello bidir
Never ran
-Te1/6/5
Link down
Hello bidir
Never ran
-LMP hello timer
Hello Tx (T4) ms
Hello Rx (T5*) ms
Interface
State
Cfg
Cur
Rem
Cfg
Cur
Rem
------------------------------------------------------------------------Te1/6/4
operational 1000
756
30000
29896
Te1/6/5
operational 1000
756
30000
29228
BRKCRS-1930
Cisco Public
12
One of the switches must be

standby. If both are active it means
VSS has recovered from dualactive condition, but new standby
has not been reloaded, most likely
due to unsaved config
This only refers to local switch
RRP
vss# sh switch virtual role detail

Switch
Switch Status Preempt

Priority Role
Session ID
Number
Oper(Conf) Oper(Conf)
Local Remote
-----------------------------------------------------------------LOCAL
1
UP
FALSE(N )
100(100) ACTIVE
0
0
REMOTE
2
UP
FALSE(N )
100(100) STANDBY 6480
9910
RRP Counters:
-------------------------------------------------------------------Inst. Peer Direction Req
Acc
Est
Rsugg
Racc
---------------------------------------------------------------------1
1
Tx
0
1
0
1
3
1
1
Rx
2
0
1
0
3
RRP FSM info:
-------------------------------------------------------------------sm(vslp_rrp RRP SM information for Instance 1, Peer 1), running yes, state role_res
Last transition recorded: (lmac)-> lstart (req)-> hold (srt_exp)-> hold (req)-> hold
(est)-> role_neg (srt_exp)-> role_neg (racc)-> role_res (racc)-> role_res (srt_exp)> role_res (racc)-> role_res (srt_exp)-> role_res (srt_exp)-> role_res
In dual-active recovery mode: No
BRKCRS-1930
Cisco Public
13
All ports on both sides of VSL

should be in bundled (P) state
Verify reliability of each individual
VSL link output interface specifies
egress link (one of the VSL
interfaces). VSLP ping should work
when VSL is up, even if remote is in
RPR mode etc
Troubleshooting VSL
vss# sh switch virtual link port-channel

Flags: D - down
P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3
S - Layer2
U - in use
N - not in use, no aggregation
w - waiting to be aggregated
Group Port-channel Protocol
Ports
------+-------------+-----------+------------------10
Po10(RU)
Te1/6/4(P)
Te1/6/5(P)
20
Po20(RU)
Te2/6/4(P)
Te2/6/5(P)
vss# ping vslp output interface t1/6/4 count 100 size 1388
Type escape sequence to abort.
Sending 100, 1388-byte VSLP ping to peer-sup via output port 1/6/4, timeout is 2
seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 12/12/28 ms
BRKCRS-1930
Cisco Public
14
what information to collect
Note: with VSS many commands use switch <#> module <#>
notation instead of just module <#>
In case of issues with VSL or VSS bring up, collect the following
information
sh tech
(if VSS is split, collect from both sides)
remote command switch sh monitor event vslp all detail
(if VSS is split, collect from both sides)
BRKCRS-1930
Cisco Public
15
VSS Agenda
Initialization
Spanning Tree
1st hop redundancy

Traffic forwarding
BRKCRS-1930
Cisco Public
16
High Availability
Redundancy Mechanisms
The default redundancy mechanism between the 2 VSS chassis and their associated
supervisors is NSF/SSO, allowing state information and configuration to be
synchronized. Additionally, only in NSF/SSO mode does the Standby supervisor PFC,
Switch Fabric, modules and their associated DFCs become active
Switch 2
SSO Standby
Switch 1
Active
VSL
Should a mismatch of information occur between the Active and Standby Chassis, the
Standby Chassis will revert to RPR mode, where only configuration is synchronized, but
PFC, Switch Fabric and modules will not be brought up
Switch 2
12.2(33)SXH2
RPR Standby
Switch 1
12.2(33)SXI3
Active
VSL
BRKCRS-1930
Cisco Public
17
Troubleshooting redundancy:
why standby is not in SSO mode
In case of certain mismatches standby will only boot to RPR mode
(fabric, PFC & modules will be down)
vss# show switch virtual redundancy
My Switch Id = 1
Peer Switch Id = 2
Last switchover reason = none
Configured Redundancy Mode = sso
Operating Redundancy Mode = rpr
...
vss# show switch virtual redundancy mismatch
Startup Config Mismatch:
Mismatch in config file between local Switch 1 and peer Switch 2:
ACTIVE : Interface TenGigabitEthernet1/6/5 shutdown
STANDBY : Interface TenGigabitEthernet1/6/5 not shut
Other possibilities
IOS version mismatch
Other VSL-related config mismatch
Non-SSO redundancy mode is configured
Forwarding engine (PFC) mismatch
BRKCRS-1930
Cisco Public
18
VSS with 4 supervisors

Initially in-chassis redundant
supervisors were kept in rommon
not used
Pre-12.2(33)SXI4
VSL
As of 12.2(33)SXI4 in-chassis
redundant supervisors function as
a linecard ports are useable
Active
SSO
rommon>
rommon>
Si
Si
Before switching to linecard mode

supervisors will boot to RPR-warm
mode meaning they will have their
configuration synchronized
If active supervisor fails entire
chassis is reloaded 2nd chassis
takes over same model as with
2 sups
VSL
If supervisor fails completely

(doesnt boot) or removed, the inchassis redundant supevisor will
boot as active supervisor no
need to follow procedure for
supervisor replacement
Active
SSO
RPR-warm
RPR-warm
Si
Si
12.2(33)SXI4 and later
BRKCRS-1930
Cisco Public
19
What is Dual-Active?
If VSL goes down standby needs
to know if it was just VSL or the
active switch that failed
Si
For faster failovers assumption is

that active switch fails Old
standby becomes Active a.s.a.p.
If old Active is still there however
we will have 2 devices with
identical config on the network
IGP adjacencies will start to flap
or will go down
Layer3-MEC
Standby
Active
Active
Si
L2 MEC will be error-disabled

after ~1 minute by EtherChannel
misconfig guard (because of
receiving 2 different BPDUs)
VSL
Si
Layer2-MEC
Dual-active, if not detected will cause severe network outage

Configure robust dual-active detection
BRKCRS-1930
Cisco Public
20
Dual-Active Detection options

Enhanced PAGP
Switch 1
Active
Switch 2
Hot Standby
Requires PAGP+ capable neighbor with

3750
12.2(46)SE
4500
12.2(44)SE
6500
12.2(33)SXH
VSLP Fast Hello
Switch 1
Active
Switch 2
VSLP
VSLP
IP-BFD
Switch 1
Switch 2
BFD
Hot Standby
BFD
Active
L2 Heart Beat Link
L3 Heart Beat Link
Software-12.2(33)SXI
Software -12.2(33)SXH1
Hot Standby
Enhanced subsecond detection in

12.2(33)SXI3
Software -12.2(33)SXH1
BRKCRS-1930
Cisco Public
21
Dual Active Recovery

Switch 1 detects that switch 2 is now also active triggering dual active
condition thus switch 1 brings down all the local interfaces to avoid network
instability. Until VSL link restoration occurs, switch 1 is isolated from the
network;
Once the VSL link comes up, the role negotiation determines that switch 1
needs to come up in STAND_BY mode hence it reboots itself; finally, all
interface on switch 1 are brought on line and switch 1 assumes STAND_BY
role
OLD
ACTIVE
New
ACTIVE
Switch 1
Reboot and
Comes Up in
STAND_BY
Mode
Switch 2 in
ACTIVE
Mode
Switch 1
All
Interfaces
Down
VSS Restoration
Dual Active Recovery

BRKCRS-1930
Cisco Public
22
Dual-active recovery,
If configuration was changed but has not been saved the would-bestandby switch will not be reloaded following VSL recovery
Save the config & reload standby
19:54:59: %VSLP-SW2_SP-5-RRP_MSG: Role change from Active to Standby and hence need
to reload
19:54:59: %VSLP-SW2_SP-5-RRP_UNSAVED_CONFIG: Ignoring system reload since there are
unsaved configurations. Please save the relevant configurations
19:54:59: %VSLP-SW2_SP-5-RRP_MSG: Use 'redundancy reload shelf' to bring this switch
to its preferred STANDBY role
Reload from active switch will not correct this

After reloading it might happen that config between Active and Standby
is not consistent Standby will come up in RPR mode
Save the config once again and reload standby again (redundancy
reload peer)
BRKCRS-1930
Cisco Public
23
Virtual Switching System

Which Dual Active Recovery Method Should I Use?
Since dual-active detection is important

redundancy is highly recommended
Si
Si
ePAgP
Use Fast-hello + e-PAgP

In case of all-LACP deployment, use Fasthello over port-channel
Only case where BFD had advantage was in
pre-SXI3 release with routed ECMP uplinks
and OSPF
Redundant
VSL Fiber
VSLP Fast-Hello
or BFD
ePAgP
BRKCRS-1930
Cisco Public
24
VSS Agenda
Initialization
Spanning Tree
1st hop redundancy

Traffic forwarding
BRKCRS-1930
Cisco Public
25
Spanning Tree and VSS

Physical
Active
Logical
Standby
4
1
STP process
BRKCRS-1930
VSS domain behaves as a single bridge

STP runs only on SP of active switch
VSL is not part on STP and will not be blocked
BPDUs will travel across single link of the MEC
STP will be blocking ports is there are redundant

links Keep STP enabled
Cisco Public
26
Troubleshooting STP
vss#sh spanning-tree interface po201 detail
Port 5767 (Port-channel201) of VLAN0001 is designated forwarding
Port path cost 3, Port priority 128, Port Identifier 128.5767.
Designated root has priority 0, address 001e.4963.7b94
Designated bridge has priority 32768, address 0008.e3ff.fdbd
Designated port id is 128.5767, designated path cost 16
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1

Link type is point-to-point by default
BPDU: sent 4447, received 12
...
STP state, role and BPDU counters

for given port
All debugging for STP is on active
SP
Limit debugs to port in question
Abbreviated BPDU debug
vss# remote login switch
Detailed BPDU debug (when
vss-sp# debug interface po201
enabled together with abbreviated
Condition 1 set
one)
vss-sp# debug spanning-tree switch tx
Observe normal precautions
Spanning Tree Switch Shim transmit bpdu debugging is on
regarding debugs
Dec 6 14:59:22.594: SW1_SP: STP SW: FAST TX: VLAN 555 Port-channel201: bpdu size 116, refcnt 1
vss-sp# debug spanning-tree switch tx decode
Spanning Tree Switch Shim decode transmitted packets debugging is on
Dec 6 14:59:43.510: SW1_SP: STP SW: FAST TX: 0180.c200.0000<-0015.6301.26f8 type/len 0026
Dec 6 14:59:43.510: SW1_SP:
encap SAP linktype ieee-st vlan 1 len 112 on v1 Po201
Dec 6 14:59:43.510: SW1_SP:
42 42 03 SPAN
Dec 6 14:59:43.510: SW1_SP:
CFG P:0000 V:00 T:00 F:00 R:0000 001e.4963.7b94 00000010
Dec 6 14:59:43.510: SW1_SP:
B:8000 0008.e3ff.fdbd 96.87 A:0400 M:1400 H:0200 F:0F00
...
vss-sp# undebug all
All possible debugging has been turned off
BRKCRS-1930
Cisco Public
27
Spanning Tree stability features recap

Feature
UDLD
Bridge
Assurance
(BA)
Dispute
Loop
Guard
Condition
Detects if link becomes
unidirectional
I.e. link cannot carry BPDUs
both ways causes loops
Expects to receive a BPDU
every hello_time from the
peer.
I.e. cases of dead control
plane on the remote side,
also BPDU loss
Checks the remote port role
in the received BPDU, role
should not be designated in
BPDU received on
designated port
Cases of unidirectional
communication
Doesnt allow port to take
designated role if it stopped
receiving BPDUs
Unidirectional
communication, control plane
issues on remote
BRKCRS-1930
Works on
Effect
Note
Physical
port
Error-disables
unidirectional
links
Useful on port-channels to
take out broken links,
alternative fast-timers
PAGP/LACP
Blocks port at
STP level
(BAinconsistent
state)
Main protection mechanism

where supported, alternative
is Loop Guard
Logical
port
Blocks port at
STP level
(Disputed
state)
Complements BA, on by
default. Somewhat overlaps
with UDLD, but not as
effective on port-channels.
Only works with RSTP/MST
BPDUs
Logical
port
Blocks port at
STP level
(Loopinconsistent)
Superseded by BA + Dispute,
use with PVST+ or when BA
is not supported
Logical
port
Cisco Public
28
Bridge assurance, Dispute & UDLD
Preferred combination is Bridge Assurance + UDLD normal

mode + Dispute (on all interswitch links) when both sides
support it
UDLD is needed to take out bad links from port-channels
(otherwise BA or Dispute will keep whole port-channel
blocked). PAgP/LACP will take out bad links, but will take
longer (~105sec vs ~20sec for UDLD with 7 sec timer)
If preferred config is not supported use Loop Guard + UDLD
(supported by all Cisco switches)
Defaults: BA/UDLD disabled, Dispute - enabled
BRKCRS-1930
Cisco Public
29
VSS Agenda
Initialization

Spanning Tree
1st hop redundancy
Traffic forwarding
BRKCRS-1930
Cisco Public
30
Asymmetric Routing
Alternating HSRP Active between
distribution switches can be used
for upstream load balancing,
however downstream traffic hits
both distribution block switches
This can cause a problem
with unicast flooding
ARP entries age in 4 hours while
L2 entries age in 5 minutes
ARP entry with no matching L2
entry unicast flooding
In many cases when the HSRP
standby needs to forward a frame
it will have to unicast flood the
frame since its CAM table is
empty
Switch 1: Active
HSRP and Root
Bridge VLAN 3
CAM Table
Empty for
VLAN 2
Switch 2: Active
HSRP and Root
Bridge VLAN 2
Si
CAM Table
Empty for
VLAN 3
Si
VLAN 3
VLAN 2
VLAN 3
VLAN 2
With VSS there is single logical router thus no asymmetric routing

BRKCRS-1930
Cisco Public
31
1st hop redundancy with VSS

MAC_B Router MAC
IP B IP A
PC B
Router MAC
0001.0002.0003
VSS acts as 1 router there is 1 router MAC

address, both switches will L3 switch packets
destined to that MAC address
Router MAC
0001.0002.0003
Once either switch learns dynamic MAC address,

other switch will also learn no unicast floods
due to asymmetry of traffic between switches
In case of failover router MAC address does not
change Inherrent 1st hop redundancy
PC A
MAC_A Router MAC

IP A IP B
BRKCRS-1930
Cisco Public
32
VSS mac-address
By default VSS will use Router mac-address from active switch backplane
Router mac-address is maintained across switchovers no 1st hop redundancy
protocol is needed
If entire VSS system is brought down and then up again and switch 2 ends up
being active router mac-address might change (this will only have impact on
devices that ignore gratuitous ARPs)
To avoid such change, use mac-address use-virtual with this command VSS will
use special mac-address reserved for VSS
vss(config)#switch virtual domain 111
vss(config-vs-domain)#mac-address use-virtual
Configured Router mac address is different from operational value. Change will take
effect after config is saved and the entire Virtual Switching System (Active and
Standby) is reloaded.
Virtual mac is based on 0008.e3ff.fc00

Alternatively router-mac maybe statically configured with mac-address
<address> in the domain config context
BRKCRS-1930
Cisco Public
33
Troubleshooting Router-MAC
When VSS receives a packet destined to Router-MAC it will try to L3 switch
(route in hardware) the packet, else the packet will be bridged
vss# sh interface vlan 226
What is router MAC for given
Vlan226 is up, line protocol is up
Hardware is EtherSVI, address is 0008.e3ff.fdbc (bia 0008.e3ff.fdbc) interface
Internet address is 192.168.222.18/30
It should be pointing to the Router
...
vss# sh mac-address-table address 0008.e3ff.fdbc vlan 226 all
Legend: * - primary entry
age - seconds since last seen
n/a - not available
Actual hardware L2 entry must

have non-zero Xtag in order for
forwarding engine to consider such
packets for L3 switching
vlan
mac address
type
learn
age
ports
------+----------------+--------+-----+----------+-------------------------Supervisor switch 1 Module 6
* 226 0008.e3ff.fdbc
static No
Router
Supervisor switch 2 Module 6
* 226 0008.e3ff.fdbc
static No
Router
vss# sh mac-address-table address 0008.e3ff.fdbc vlan 226 detail switch 2 module 6
MAC Table shown in details
========================================
PI_E RM RMA Type Alw-Lrn Trap Modified Notify Capture Flood
Mac Address Age Pvlan SWbits Index XTag
----+---+---+----+-------+----+--------+------+-------+------+--------------+----+------+------+------+---Supervisor switch 2 Module 6
Yes No
No ST
No
No
No
No
No
No
0008.e3ff.fdbc 0xE8 226
0
0x380 1
BRKCRS-1930
Cisco Public
34
MAC address learning with VSS
MAC A is learned on lower MEC, triggering the

frame to be sent to every forwarding engine
(DFC/PFC) Flood to Fabric mechanism (HW)
Internal frame header (carried over VSL) includes

source index which identifies source port and
hence the MAC is learned on lower MEC although
the frame is received on VSL
PC B
Depending on how traffic is flowing through VSS

some forwarding engines might not see the
packets from A after initial flood to fabric which
might lead to aging of address and flooding
MAC synchronization feature keeps address from
expiring as long as traffic from that address is
seen anywhere in the system
PC A
BRKCRS-1930
Cisco Public
35
MAC address synchronization

Initial new learns are syncronized between switch 1 and switch 2
However if only switch 1 or switch 2 sees the traffic for given address L2 entry might age out
in one of the switches (this behavior is per forwarding engine: PFC/DFC)
In order to reduce chance of unicast flooding we need to keep L2 entries consistent access
both switches
mac-address-table synchronize feature will keep L2 tables synchronized
Enabled by default when WS-X6708 linecard is present in the chassis
Enabled by default in VSS as of 12.2(33)SXI4
Recommended in all cases
Make sure there is at least 2x aging intervals in synchonization interval
(i.e. for sync interval 160, L2 aging is >320 seconds, 480 recommended)
vss(config)# mac-address-table synchronize
% Current OOB activity time is [160] seconds
% Recommended aging time for all vlans is atleast three times the activity interval
and global aging time will be changed automatically if required
When troubleshooting unicast flooding, 2 items are very important
What module traffic arrives to (use commands to check ether-channel load-balancing)
Whether the module in question has the mac-address learned

(use sh mac-address address <mac> all)
BRKCRS-1930
Cisco Public
36
VSS Agenda
Initialization
Spanning Tree
1st hop redundancy

Traffic forwarding
BRKCRS-1930
Cisco Public
37
Ingress forwarding model

Distributed architecture. Ingress forwarding engine makes
forwarding, ingress *and* egress ACL/QOS decisions
IMPORTANT: If the linecard where packet is received has DFC
entries on that linecard need to be looked at when troubleshooting.
Otherwise look at active supervisors forwarding entries
i.e. sh mls cef <prefix> module <mod#>
or sh mls cef <prefix>
DFC
Ingress
Fabric
DFC
Egress
Traffic flow
BRKCRS-1930
Cisco Public
38
Traffic locality
Main concept for traffic forwarding is locality
Only local ports are used to send traffic out
except when there are no local ports, this is when traffic will cross
VSL/Peer-link
BRKCRS-1930
Cisco Public
39
Traffic locality for ECMP routes

ECMP follows a similar behavior, local
links are preferred and all traffic is
forwarded out of a locally attached link
Hardware FIB inserts entries for ECMP
routes using locally attached links
If all local links fail the FIB is programmed
to forward across the VSL link
Si
Si
Te1/2/1
Te1/2/2
SW1
vss# sh ip route 10.121.0.0 255.255.128.0 longer-prefixes

D
10.121.0.0/17
[90/3328] via 10.122.0.33, 2d10h, TenGigabitEthernet2/2/1
Four ECMP
Entries
vss# sh mls cef 10.121.0.0 17 switch 1

Codes: decap - Decapsulation, + - Push Label
Index Prefix
Adjacency
102400 10.121.0.0/17
Te1/2/2
, 0012.da67.7e40 (Hash: 0001)
Te1/2/1
, 0018.b966.e988 (Hash: 0002)
BRKCRS-1930
Cisco Public
Two FIB
Entries
40
VSS L2/L3 Forwarding (Data Plane)

VSS Data Plane Troubleshooting L2 MEC
VSS specific commands
augmented with switch id
Verify the load-balance algorithm used

vss# show etherchannel load-balance switch 2 module 2
EtherChannel Load-Balancing Configuration:
src-dst-ip vlan included
mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
MPLS: Label or IP
Important:: Only use parameters

consistent with the configured
load-balancing algorithm.
Command uses all the specified
arguments to calculate the hash.
Identify the physical path for flow from host 2 host 1 (out of Port-channel 2)
vss# show etherchannel load-balance hash-result interface Port-channel 2 switch 1
ip 9.0.1.2 vlan 705 8.0.1.1
Packet coming in on switch 1, needing to go
Computed RBH: 0x6
out on Po2 will select Gi1/6/2
Would select Gi1/6/2 of Po2
vss# show etherchannel load-balance hash-result interface Port-channel 2 switch 2
ip 9.0.1.2 vlan 705 8.0.1.1
Computed RBH: 0x6
Packet coming in on switch id 2, needing to
Would select Gi2/9/15 of Po2
go out on Po2 will select Gi2/9/15
BRKCRS-1930
Cisco Public
41
VSS L2/L3 Forwarding (Data Plane)

VSS Data Plane Troubleshooting ECMP: Host 1 Host 2
Routing table shows two Equal Cost Paths to 9.0.0.0/8
vss# show ip route 9.0.0.0 | i via
Known via "eigrp 101", distance 90, metric 3072, type internal
Redistributing via eigrp 101
7.7.1.2, from 7.7.1.2, 1d00h ago, via TenGigabitEthernet2/2/7
* 7.6.1.2, from 7.6.1.2, 1d00h ago, via TenGigabitEthernet1/3/2
Looking at the HW table shows next hop directly attached to local switch
is preferred
vss# show mls cef lookup 9.0.1.0 switch 1 mod 3
Packet coming in on switch 1 module 3, for 9.0.0.0/8

prefers next hop attached to local switch id 1

Index Prefix
Adjacency
108775 9.0.0.0/8
Te1/3/2
, 000f.35ed.7c00
vss# show mls cef lookup 9.0.1.0 switch 2 mod 2
Packet coming in on switch 2 module 2, for 9.0.0.0/8

prefers next hop attached to local switch id 2
Index Prefix
Adjacency
108775 9.0.0.0/8
Te2/2/7
, 000f.35ed.7c00
DUT# show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 1 mod 3
Interface: Te1/3/2, Next Hop: 7.6.1.2, Vlan: 4064, Destination Mac: 000f.35ed.7c00
DUT# show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 2 mod 2
Interface: Te2/2/7, Next Hop: 7.7.1.2, Vlan: 4056, Destination Mac: 000f.35ed.7c00
BRKCRS-1930
Cisco Public
42
1/1/33
Will the
packet cross
VSL link?
1/1/15
Po4
2/4/33
Po3
VSS
2/6/3
vss# sh mac-address-table address 0005.9a3b.6c80 vlan 226

...
vlan
mac address
type
learn
age
ports
------+----------------+--------+-----+----------+-------------------------Supervisor switch 1 Module 6
* 226 0005.9a3b.6c80
dynamic Yes
10
Po3
Supervisor switch 2 Module 6
What is the port
* 226 0005.9a3b.6c80
dynamic Yes
10
Po3
0005.9a3b.6c80
for this mac
address
What are physical ports of portchannel
vss# sh etherchannel 3 summary

...
Ports
------+-------------+-----------+----------------------------------------------3
Po3(SU)
PAgP
Gi1/1/15(D)
Gi2/6/3(P)
All ports on switch1 side are

down
If packet will arrive to switch1 to
be switched to po3, packet will
cross VSL
BRKCRS-1930
Cisco Public
43
1/1/33
Will the
packet cross
VSL link?
1/1/15
Po4
2/4/33
Po3
VSS
2/6/3
0005.9a3b.6c80
vss# sh mac-address-table address 0005.9a3b.6c80 vlan 226 detail switch 1 module 6

MAC Table shown in details
========================================
PI_E RM RMA Type Alw-Lrn Trap Modified Notify Flood
Mac Address Age Pvlan Index XTag
----+---+---+----+-------+----+--------+------+------+--------------+----+------+------+---Supervisor switch 1 Module 6
Yes No
No DY
No
No
Yes
No
No
0005.9a3b.6c80 0x86 226
0xB40 0
vss# remote command switch test switch virtual ltl index 0xB40
...
Unmapped index: 0xB40
------+---------------------------------------SW view
Index | Ports
------+---------------------------------------0x0B40 Po3[Gi2/6/3],Po10[Te1/6/4]
...
------+---------------------------------------HW view
Index | Ports
------+---------------------------------------0x0B40 Te1/6/4,Gi2/6/3
...
vss# sh switch virtual link port-channel | i Po
Ports
10
Po10(RU)
Te1/6/4(P)
20
Po20(RU)
Te2/6/4(P)
BRKCRS-1930
Find the index for given mac

address on ingress forwarding
engine
Find what ports on the local
switch (1) this index includes
Index should include VSL ports
How to verify if the packet from

switch 1 will cross VSL in order to
reach next-hop mac-address?
Cisco Public
44
VSS forwarding troubleshooting

summary
Unless the traffic is crossing VSL,
troubleshooting VSS packet forwarding is
exactly the same as troubleshooting
standalone cat6500
When traffic crosses VSL, verify
L3 entries on the ingress forwarding
engine (PFC or DFC)
L2 entries (for next hop destination mac)
on forwarding engine servicing the VSL on
the 2nd chassis (strictly speaking L2 entries
need to be checked on all DFCs along the
packet path)
BRKCRS-1930
Cisco Public
45
Special case for flooding
MAC_B
3
1
MAC B is not known flood the frame
Internal frame header (carried over VSL) includes

destination index which is remapped by egress
switch to another index that does not include any
MEC that has operational ports on ingress switch
Frame is flooded to devices that are single

connected to egress switch (on the right)
MAC_A
BRKCRS-1930
Cisco Public
46
EtherChannel Adaptive Hash

Each flow is assigned to 1 of 8 buckets
Each port in port-channel transmits traffic for some buckets (i.e. 4 for 2-port channel, 2
for 4-port etc)
When ports are joining/leaving channel the buckets are redistributed among operational
ports in deterministic fashion
Flows that remain on operational ports might be disturbed while ASICs are being
programmed
Member 1
Member 2
buckets that must move
New member
joins
Member 1
Member 2
Member 3
buckets moving between

operational ports
With adaptive hash option, only buckets that must move are reprogrammed
Member 1
Member 2
BRKCRS-1930
buckets that must move
New member
joins
Cisco Public
Member 1
Member 2
Member 3
47
EtherChannel Adaptive Hash

Adaptive hash is enabled by default on VSL link
If there is 1 link / chassis / MEC adaptive hash on MEC will not make any difference
If the network consists of several adjacent VSS systems, adaptive hash was enhanced
to avoid traffic polarization (as of 12.2(33)SXI)
Configured per port-channel
vss(config)#int port-channel200
vss(config-if)#port-channel port hash-distribution adaptive
With adaptive hash less flows should be impacted when ports join or leave portchannels
This is mostly evident when control-plane is busy (i.e. when many changes are
happening at the same time during failovers etc)
BRKCRS-1930
Cisco Public
48
SPAN
When SPANed traffic is crossing VSL it is transmitted
over single link this might cause oversubscription of
VSL link if amount of SPANed traffic is significant
Use MEC as SPAN destination to prevent SPANed
traffic from crossing VSL
If one side of the MEC goes down SPANed traffic will
cross VSL
Provision enough bandwidth on VSL
Use port-channel min-links LACP feature on SPAN
destination MEC to bring down MEC if link is down on one
side
Use EEM script to shut down MEC or SPAN session when
one side of SPAN destination MEC goes down
BRKCRS-1930
Cisco Public
49
VSS Agenda
Initialization
Spanning Tree
1st hop redundancy

Traffic forwarding
BRKCRS-1930
Cisco Public
50
Multicast forwarding
Layer 2 access has two multicast routers on the access subnet, RPF
checks and split roles between high and low IP address routers
VSS has a single multicast router which simplifies multicast topology
The multicast forwarder is selected based on which member of VSS
link receives multicast traffic
IGMP Querier
(Low IP address)
Si
Non-DR Has to
Drop All
Non-RPF Traffic
BRKCRS-1930
Single Logical Multicast

Designated Router and IGMP Querier
Si
Designated
Router
(High IP Address)
Cisco Public
51
MEC behavior upon

VSS recovery after SSO switchover
3
2
1
Following SSO switchover left switch comes up

after reload
MEC link from left switch is brought up and joins

the bundle
Top switch starts sending a share of traffic to the left

switch, but the left switch might still be converging
(loading FIB tables, programming ASICs etc), so it
might not be fully ready to correctly forward the this
traffic
this might cause part of traffic to be lost for
some time after the switch recovery
To prevent this issue, configure port-channel load-defer feature on upstream switch

Upstream switch will delay sending traffic to newly bundled port for configured duration
vss(config)#port-channel load-defer 120
vss(config)#int po200
vss(config-if)#port-channel port load-defer
This will enable the load share deferral feature on this port-channel.
The port-channel should connect to a Virtual Switch (VSS).
Do you wish to proceed? [yes/no]: y
BRKCRS-1930
Cisco Public
52
Multicast fast-redirect
When a member of egress
Layer2 port-channel (MEC or
DEC) is unbundled/bundled On
VSS replicating multicast traffic in
egress mode it might take
noticeable time to reprogram
hardware to send traffic via
remaining links (local or across
VSL)
Sources
MEC
Si
Fast-redirect feature shortens

reprogramming time by
preprogramming most of the
needed changes
Si
MEC
Receivers
vss(config)#interface port-channel 40
vss(config-if)#mls ip multicast egress fast-redirect
BRKCRS-1930
Cisco Public
53
VSS: summary
1 active redundant control plane
single config
single point of management
VSS domain
Active
Standby
Active
Control Plane
Standby switch is essentially a

set of additional linecards
Control messages and Data
frames flow between active and
standby via VSL
Standby
VSL
Control Plane
Active
Active
Data Plane
Data Plane
Dual-Active
detection link
MEC
(can be seen as backplane

extension)
Special encapsulation on VSL

frames to carry additional
information
BRKCRS-1930
Cisco Public
54
V PC
BRKCRS-1930
Cisco Public
Both VPC and VSS

simplify logical Layer 2 topology
use Traffic Locality for efficient shortest path
forwarding
BRKCRS-1930
Cisco Public
56
VPC Agenda
Initialization
Redundancy considerations
Spanning Tree
Traffic forwarding
1st hop redundancy
BRKCRS-1930
Cisco Public
57
VPC Virtual Port channel

2 active control planes
2 configs
2 points of management
VPC domain
Primary-Secondary notion for some

aspects of operation
Primary
Secondary
Active
Control Plane
Control messages and Data frames

flow between active and standby via
Peer-Link
Peer-Link is L2 trunk with plain 802.1q
encapsulation
Control messages are carried by CFS
over Peer Link
Active
Peer-Link
Control Plane
Active
Active
Data Plane
Data Plane
Peer
Keepalive link
VPC
Peer keepalive link to detect dualactive condition

We call VPC the MCEC between VPC
domain and access switches
BRKCRS-1930
Cisco Public
58
VPC initialization
VPC init is largely independent of NXOS boot each
switch boots on its own
VPC feature starts
Keep-alive linkup / peer communication is established

Peer-link linkup / CFS communication is established
Primary/Secondary role is resolved
Consistency is checked via CFS and applications synced

Peer-Link brought UP for data
VPCs brought UP
BRKCRS-1930
Cisco Public
59
Cisco Fabric Services

CFS
CFS messaging
Uses
Configuration validation
MAC member port synchronization

vPC member port status
IGMP snooping synchronization
vPC status
For VPC CFS messages are encapsulated in Ethernet frames

delivered between peers on the peer-link
Nexus# sh cfs application
---------------------------------------------Application
Enabled
Scope
---------------------------------------------arp
Yes
Physical-eth
stp
Yes
Physical-eth
vpc
Yes
Physical-eth
igmp
Yes
Physical-eth
l2fm
Yes
Physical-eth
...
BRKCRS-1930
Cisco Public
60
VPC Configuration consistency

VPC has distributed management plane. Configurations of both
switches are managed separately
Some configurations inconsistencies could lead to undesirable
forwarding implications (packet duplication, blackholing etc). VPC
takes different action depending on the type of inconsistency
Type 1: VPC will not come up
Type 2: VPC will come up, but undesirable forwarding implications

might occur, syslog will be printed upon detected inconsistency
Nexus#
Nexus# sh
sh vpc
vpc consistency-parameters
consistency-parameters global
interface port-channel 1
Name
Name
Type
Type Local
Local Value
Value
Peer
Peer Value
Value
------------------------------- ------------------------------------------- --------------------------------------------STP
lag-id
Mode
11
Rapid-PVST
[(7f9b,
Rapid-PVST
[(7f9b,
STP
... Disabled
1
None
None
STP
modeMST Region Name
11
""
active
""
active
STP
STP MST
PortRegion
Type Revision
11
0Default
0Default
STP
STP MST
PortRegion
Guard Instance to 11
None
None
STP
VLANMST
Mapping
Simulate PVST
1
Default
Default
STP
Native
Loopguard
Vlan
11
Disabled
1
Disabled
1
STP
PortBridge
Mode Assurance
11
Enabled
trunk
Enabled
trunk
STP
MTU Port Type, Edge
11
Normal,
1500
Disabled,
Normal,
1500
Disabled,
BPDUFilter,
Duplex
Edge BPDUGuard 1
Disabled
full
Disabled
full
STP
Speed
MST Simulate PVST
11
Enabled
10 Gb/s
Enabled
10 Gb/s
Interface-vlan
Allowed VLANs admin up
2101
101
101
101
Interface-vlan routing
2
1,101
1,101
BRKCRS-1930
Cisco Public
61
Troubleshooting VPC initialization

Use sh vpc to check the feature status
vpc1# show feature | i vpc
vpc
1
enabled
vpc1# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id
: 1
Peer status
: peer adjacency formed ok
vPC keep-alive status
: peer is alive
Configuration consistency status: success

Type-2 consistency reason
: Consistency Check Not Performed
vPC role
: primary
Number of vPCs configured
: 1
Peer Gateway
: Disabled
Dual-active excluded VLANs
: -
vPC Peer-link status

--------------------------------------------------------------------id
Port
Status Active vlans
--
----
------ --------------------------------------------------
Po100
up
CFS can communicate with the

peer
We hear peer-alives
Configs are compatible
Master/Slave for certain apps
Peer-Link will come up after CFS +
Peer-Keepalive + Config check are
ok
1,101
vPC status
---------------------------------------------------------------------id
Port
Status Consistency Reason
Active vlans
--
----
------ ----------- ------
------------
Po1
up
101
BRKCRS-1930
success
success
Cisco Public
62
Troubleshooting VPC initialization

Stable, not expecting issues here
Set VPC logging level to 5 (default) to see more verbose messaging during the
VPC bringup
vpc1(config)# logging level vpc 5
08:18:47 %ETHPORT-5-SPEED: Interface port-channel100, operational speed changed to 10 Gbps Peer-Link comes up
08:18:51 %VPC-3-PEER_UNREACHABLE: Remote Switch Unreachable
08:18:51 %VPC-3-VPC_PEER_LINK_BRINGUP_FAILED: vPC peer-link bringup failed (vPC peer is not reachable over cfs)
08:18:51 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1,100-101 on Interface port-channel100 are being suspended.
(Reason: vPC peer is not reachable over cfs)
08:18:51 %ETHPORT-5-IF_UP: Interface port-channel100 is up in mode trunk
08:18:58 %VPC-4-VPC_ROLE_CHANGE: In domain 1, VPC role status has changed to primary
08:18:58 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 1,100-101 on Interface port-channel100 are removed from
suspended state.
08:18:58 %VPC-5-VPC_DELAY_SVI_BUP_TIMER_START: vPC restore, delay interface-vlan bringup timer started
08:19:08 %VPC-5-VPC_DELAY_SVI_BUP_TIMER_EXPIRED: vPC restore, delay interface-vlan bringup timer expired,
reiniting interface-vlans
08:19:08 %VPC-5-VPC_RESTORE_TIMER_START: vPC restore timer started to reinit vPCs
08:19:38 %VPC-5-VPC_RESTORE_TIMER_EXPIRED: vPC restore timer expired, reiniting vPCs
In case process does not go beyond certain stage, one should look at
communication between the peers (CFS)
BRKCRS-1930
Cisco Public
63
VPC config remarks

Check config consistency using sh vpc consistency-parameters
Complete list of parameters which should be consistent is quite
extensive: physical port config, QOS, security, STP, routing
protocols etc
check config guide for specific NXOS version
Domain id must be unique for each domain reachable adjacent

at Layer 2
VPC domain 100
Domain id MUST be
different
(cant be 100 on both
Pair)
VPC
BRKCRS-1930
2011 Cisco and/or its affiliates. AllVPC

rights reserved.
domain
200
Cisco Public
64
VPC: CFS troubleshooting

Cisco Fabric Services Transport of control messages between VPC peers
Nexus# show cfs status
Distribution : Enabled
Distribution over IP : Disabled
IPv4 multicast address : 239.255.70.83
IPv6 multicast address : ff15::efff:4653
Distribution over Ethernet : Enabled
Nexus# show cfs peers
Physical Fabric
--------------------------------------------Switch WWN
IP Address
--------------------------------------------20:00:00:1b:54:c2:42:41 10.48.73.222 [Local]
Nexus
20:00:00:1b:54:c2:42:44 0.0.0.0
Total number of entries = 2
Nexus# show cfs internal ethernet-peer statistics

| i Trans|Rece
Number of Segments Transmitted
: 218
Number of Acks Transmitted
: 223
Maximum Segment Size Transmitted
: 0
Number of Transmission Timeouts
: 0
Number of segments in Transmit Queue
: 0
Number of segments in Re-Transmit Queue
: 0
Total Number of Segments Received
: 441
Number of Acks Received
: 217
Number of Duplicate Messages Received
: 0
Number of Unexpected Segments Received
: 0
Number of fragmented segments Received
: 2
Number of duplicate fragments Received
: 0
Number of unfragmented segments Received
: 210
Number of Received Segments Dropped
: 0
TX/RX
counters
should move
when
Number of Unreliable
segments
Transmitted
: 1
Number of UnreliableVPC
segments
Received
is active
or coming up : 1
Nexus# sh cfs internal notification log name vpc

Sun Nov 14 15:27:22 2010: Peer add 20:00:00:1b:54:c2:42:44
Sun Nov 14 19:05:25 2010: Peer gone 20:00:00:1b:54:c2:42:44
Sun Nov 14 19:08:03 2010: Peer add 20:00:00:1b:54:c2:42:44
BRKCRS-1930
Cisco Public
Remote peer should be seen

Shows timestamps for when CFS
communication for VPC was
interrupted (peer-reload, peer-link
issues etc)
65
More information
sh tech
(collect for offline analysis, takes ~5 min when redirected to file)
sh tech vpc
(collect when there is no time for big sh tech)
debug vpc peer
(peer events, useful for indepth vpc troubleshooting)
debug vpc peer-link
(peer-link events, for indepth vpc bringup troubleshooting)
debug cfs event ethernet
(cfs event peer communication)
BRKCRS-1930
Cisco Public
66
VPC Agenda
Initialization
Spanning Tree
Traffic forwarding
1st hop redundancy
BRKCRS-1930
Cisco Public
67
VPC redundancy model

Process restartability
Supervisor redundancy
VPC redundancy
Processes checkpoint their runtime state

Crashing process is restarted statefully by
system manager
VPC Domain
Switch 1
HA-policy will trigger
supervisor switchover
in response to
excessive process
crashing, software,
hardware or
diagnostic failure
BRKCRS-1930
Active
Switch 2
Process 1
Process 1
Active
Process 2
Process 2
Process X
Process X
Standby(SSO)
Standby(SSO)
Devices dual-attached to VPC domain are protected against

single switch failureCisco
(power,
hardware, maintenance etc)
Public
68
Peer-link failure handling

(similar to dual-active detection in VSS)
Primary is alive
VPC peer-link failure
I am primary
Primary is gone
Receiving
Keepalives*
2ndary
no
Become primary
yes
primary
Bring down all VPC ports
Done
VPC peers do not require reload following
peer-link failure or recovery
BRKCRS-1930
Cisco Public
69
Keepalive link
Peer Keepalives
Heartbeat between vPC peers to prevent dual-active scenario

Keepalives are sent every second by default on UDP port 3200
3 second hold timeout on peer-link loss (ignore keepalive to leave

time for convergence before taking action)
5 seconds keepalive timeout (starts after hold timeout after peer-link
down) if no keepalive received during this timeout dual active
detection seconday bring down VPC
Use dedicated link, though NXOS does not enforce this just IP
connectivity is verified
Mgmt interface can be used as keepalive link, but do not connect the
managemet interfaces together directly (only active supervisor
management interface is up)
vpc1# debug vpc
13:10:54.257099
your_context(0)
13:10:54.257126
13:10:55.257442
your_context(0)
13:10:55.257469
13:10:56.257324
your_context(0)
13:10:56.257351
BRKCRS-1930
peer-keepalive
vpc: received new OOB packet, version(0) flags(0) my_context(0)
my_epoch(604049) your_epoch(604104) my_ip(1.1.1.2)
vpc: your_ip(1.1.1.1) domainId(1)
Cisco Public
70
Troubleshooting VPC peer-keepalives

Nexus# show vpc peer-keepalive
--Send status
--Last send at
--Sent on interface
--Receive status
--Last receive at
--Received on interface
--Last update from peer
:
:
:
:
:
:
:
:
peer is alive
Success
2009.06.19 00:41:15 589 ms
Eth2/35
Success
2009.06.19 00:41:14 580 ms
Eth2/35
(1) seconds, (9) msec
vPC Keep-alive parameters

--Destination
: 7.7.7.77
--Keepalive interval
: 1000 msec
--Keepalive timeout
: 5 seconds
--Keepalive hold timeout
: 3 seconds
--Keepalive vrf
: v1
--Keepalive udp port
: 3200
--Keepalive tos
: 192
Nexus# show vpc statistics peer-keepalive
Peer-keepalive is only essential at

the time when peer-link goes down
At any other time peer-keepalive
failure will only trigger syslog
Peer-keepalives might be affected
by extreme control plane load
(check CPU utilization & COPP)
Number of keepalive state
transitions, closer to 0 - better

: peer is alive
vPC keep-alive statistics
---------------------------------------------------peer-keepalive tx count:
9773
peer-keepalive rx count:
8985
average interval for peer rx:
991
Count of peer state changes:
0
BRKCRS-1930
Cisco Public
71
VPC behavior at initialization

(default)
VPC needs to be able to talk to the
peer (over peer-link) before bringing
up VPC port-channels
Negotiate LACP/STP operating roles for
the chassis
Wait for per-port peer parameters and
handshake to bring up vPC ports
Performs peer parameters consistency

check on each VPC bringup
Only after VPC port-channels are

brought up.
What if after a full DC outage (both
Nexus down), only one switch is coming
up ?
Will not bring up VPCs if after a
datacenter outage, only one VPC peer
comes back up
BRKCRS-1930
Cisco Public
73
VPC Reload Restore

Allows to bring up VPCs after timeout
if peer is presumed dead
Default timeout 240 sec

Assumes primary role for STP and
LACP
Nexus(config)# vpc domain 1
Nexus(config-vpc-domain)# reload restore ?
<CR>
delay Duration to wait before assuming
peer dead and restoring vpcs
Nexus(config-vpc-domain)# reload restore delay ?
<240-3600> Time-out for restoring vPC links
(in seconds)
BRKCRS-1930
Cisco Public
74
ARP synchronization
PC B
ARP
Ip B ???
Needs to be
Resolved ?
ARP
Ip B Mac B
PC A
BRKCRS-1930
When traffic pattern

changes (due to VPC links
going up/down, due to
failover etc) the peer that
handles the traffic might
need to resolve ARP before
being able to forward
packets
This might introduce
additional delay to traffic
recovery
ARP sync feature is
supported as of 4.2(6), and
allows VPC peers to
synchronize their ARP
tables over CFS
vpc(config)# vpc domain 1

vpc(config-vpc-domain)# ip arp synchronize
Cisco Public
75
More information
sh log last <x>
(review sequence of events)
show file logflash://sup-standby/log/messages
(in case other supervisor was active when everything started)
sh process log
(which processes have crashed when)
sh redundancy status
(status of supervisor redundancy & last switchover data)
sh system reset-reason
(last reset/switchover reason per module)
sh logging onboard internal reset-reason
(reset reason from different components point of view useful
for complex cases)
sh tech /from main VDC/
(collects most of the above for offline analysis)
BRKCRS-1930
Cisco Public
76
VPC Agenda
Initialization
Spanning Tree
Traffic forwarding
1st hop redundancy
BRKCRS-1930
Cisco Public
77
Handling of Spanning Tree: VPC

1
Primary
Secondary
STP process
STP runs on both switches (2 active control

planes) but only primary switch controls VPCs.
(even if root is secondary , then Primary will send
bpdu with root info being secondary)
VPC port states changes are communicated to
secondary via CFS messages.
For non-VPC ports domain appears as 2 bridges
STP process
Peer-link is part of STP. BPDU handling is

modified such that Peer-link will never be blocked
(similar to MST implementation of IST)
Non-VPC ports are managed independently by
local STP process on each switch
BRKCRS-1930
Cisco Public
78
STP troubleshooting
Left-Root# sh spanning vlan 35
VLAN0035
Spanning tree enabled protocol rstp
Root ID
Priority
24611
Address
001b.54c2.4241
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority
24611 (priority 24576 sys-id-ext 35)
Address
001b.54c2.4241
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Desg FWD 1
128.4096 (vPC) P2p
Po100
Desg FWD 2
128.4195 (vPC peer-link) Network P2p
Peer link is running STP

Right# sh spanning-tree vl 35 detail | i "^ Port|BPDU"
Port 4096 (port-channel1, vPC) of VLAN0035 is designated forwarding
Port 4195 (port-channel100, vPC Peer-link) of VLAN0035 is root forwarding
On the other end of peer-link po1 is designated despite not sending or

receiving single BPDU
BRKCRS-1930
Cisco Public
79
STP troubleshooting
Looking at BPDUs
Left-Root# debug spanning-tree bpdu_tx tree 101
14:20:37.556707 stp: RSTP(101): transmitting RSTP BPDU
14:20:37.556750 stp: vb_vlan_shim_send_bpdu(1933): VDC
channel100 enc_type 1 len 42
14:20:37.556834 stp: RSTP(101): transmitting RSTP BPDU
14:20:37.556863 stp: vb_vlan_shim_send_bpdu(1933): VDC
enc_type 2 len 36
This output can be easily limited to

necessary Vlan/Interface, but it
doesnt dump the BPDU
Very chatty use debug logfile
<file> to redirect output to a file
on port-channel100
4 Vlan 101 port port-
on port-channel1
4 Vlan 101 port port-channel1
Left-Root# debug spanning-tree all

14:22:23.560147 stp: RSTP(1): transmitting RSTP BPDU on port-channel100
14:22:23.560169 stp: vb_vlan_shim_send_bpdu(1933): VDC 4 Vlan 1 port port-channel100
enc_type 2 len 36
14:22:23.560219 stp: BPDU TX: vb 1 vlan 1 port port-channel100 len 36 ->0180c2000000
CFG P:0000 V:02 T:02 F:78 R:80:01:00:1b:54:c2:42:43 00000002
B:80:01:00:1b:54:c2:42:44 9063 A:0000 M:0014 H:0002 F:000f
Looking at past events

Left-Root# sh spanning-tree internal event-history tree 0 interface port-channel 50
VDC02 MST0000 <port-channel50>
0) Transition at 497772 usecs after Tue Oct 20 17:42:01 2009
State: FWD Role: Root Age: 5 Inc: no [STP_PORT_STATE_CHANGE]
State: FWD Role: Root Age: 4 Inc: no [STP_PORT_ROLE_CHANGE]
State: BLK Role: Root Age: 5 Inc: no [STP_PORT_STATE_CHANGE]
...
BRKCRS-1930
Cisco Public
80
STP inconsistencies
When STP detects certain abnormal situations it may
mark ports as inconsistent and block them to prevent
forwarding loops
Root Root Guard feature detected inconsistency
(unwanted bridge tries to become root)
Loop Loop Guard feature detected inconsistency
(port becomes designated because no BPDUs are being
received)
Bridge Assurance (BA)
(no BPDUs are received from remote side)
VPC Peer-link
(any of above inconsistencies happened on VPC peer-link)
%STP-2-VPC_PEER_LINK_INCONSIST_BLOCK: vPC peer-link detected BPDU receive timeout
blocking port-channel11 VLAN0121.
BRKCRS-1930
Cisco Public
81
Primary
BRKCRS-1930
inconsistency
Handling Peer-Link STP inconsistencies

on Primary switch
1
Secondary
When peer-link STP inconsistency is detected on

primary switch the link will be put in inconsistent
STP state (effectively blocking state)
BPDUs are not sent on peer-link when it is
inconsistent. This is to allow secondary switch to
detect inconsistency and react
Cisco Public
82
inconsistency
Primary
inconsistency
Handling Peer-Link STP inconsistencies

on Secondary switch
2Secondary 1
1
2
BRKCRS-1930
When peer-link STP inconsistency is detected on

secondary switch the peer link will be put in
inconsistent STP state (effectively blocking
state)
Respective vlans or MST instances are also
blocked on all VPCs
Cisco Public
83
Bridge assurance, Dispute & UDLD

BA is default enabled on Peer-Link (and recommended to remain
enable), not recommended for VPCs unless Peer-Switch feature is
used
Dispute is default enabled (for both RSTP and MST on VPC)
UDLD [normal mode] is recommended to take out bad links from
channels (otherwise LACP takes ~100sec vs ~20 with UDLD)
Recommendation
Preferred BA + UDLD + Dispute (on all interswitch links when using
Peer-switch) when all switches support this (nexus7000/5000 and
cat6500/VSS do support)
Without Peer-switch BA should be kept only on Peer-Link (no
BA/Loop guard on VPCs)
If preferred config is not supported use Loop Guard + UDLD
(supported by all Cisco switches)
BRKCRS-1930
Cisco Public
85
STP behavior upon VPC primary failure
Primary
ROOT
OP-Primary
Secondary
Backup
ROOT
ROOT
Primary switch (STP root) fails
Secondary switch becomes operational primary

and STP root
STP root port doesnt change for access switch

nor any STP port states for VPCs, forwarding
continues
Depending on control plane load it might take few
seconds for Op-primary to start sending BPDUs.
This might cause STP reconvergence on
connected switches hence increasing hello time
or peer-switch feature might be considered in
large deployments
BRKCRS-1930
Cisco Public
86
STP behavior upon VPC primary recovery
OP-Secondary
ROOT
2
SYNC
OP-Primary
Secondary
Backup
Left switch comes back up
Peer-Link comes back up
VPC role is resolved as Operational-secondary
Left switch has better STP priority becomes

STP root
STP root port of right switch will change and that

will trigger SYNC: all non-edge STP ports will be
temporarily blocked
ROOT
ROOT
Once sync is complete ports will resume

forwarding
BRKCRS-1930
Cisco Public
87
VPC Peer-Switch feature

Both VPC switches originate BPDUs with preconfigured information. This
allows to keep the same BPDU when primary fails/recovers no extra
SYNC required avoid short interruption in forwarding described on
previous slide is avoided
Both left and right switches consider themselves root

Both left and right switches send BPDUs all the time no need to raise
hello time
Available 4.2(6) 5.x software

Primary
Secondary
ROOT
ROOT
spanning-tree vlan 1-1000 priority 8192

vpc domain 1
peer-switch
BRKCRS-1930
spanning-tree vlan 1-1000 priority 8192

vpc domain 1
peer-switch
Cisco Public
88
VPC Peer-Switch feature

Primary
Secondary
ROOT
ROOT
left# sh span vlan 101
VLAN0101
Root ID
Priority
8293
Address
0023.04ee.be01
...
Bridge ID
Priority
Address
8293
(priority 8192)
0023.04ee.be01
...
Interface
---------------Po1
Po100
Role
---Desg
Root
Sts
--FWD
FWD
left# sh vpc role | i mac

vPC system-mac
vPC local system-mac
Cost
--------1
2
Prio.Nbr
-------128.4096
128.4195
Type
--------------(vPC) P2p
(vPC peer-link)
: 00:23:04:ee:be:01
: 00:1b:54:c2:42:43
In Peer-Switch mode bridge-ID

comes from system-mac as
opposed to local mac in normal
mode
BRKCRS-1930
right# sh span vlan 101

VLAN0101
Root ID
Priority
8293
Address
0023.04ee.be01
...
Bridge ID
Priority
Address
...
Interface
---------------Po1
Po100
Cisco Public
Role
---Desg
Desg
8293
(priority 8192)
0023.04ee.be01
Sts
--FWD
FWD
Cost
--------1
2
Prio.Nbr
-------128.4096
128.4195
Type
--------------(vPC) P2p
(vPC peer-link)
89
More information
show spanning-tree internal event-history all
(allows to look back at past STP events, not included in sh tech)
sh tech stp
(from both sides of VPC)
sh tech
(from both sides of VPC, this will include in it sh tech stp, in case
VPC is is non-default VDC collect also sh tech from VDC 1)
BRKCRS-1930
Cisco Public
90
VPC Agenda
Initialization
Spanning Tree
Traffic forwarding
1st hop redundancy
BRKCRS-1930
Cisco Public
91
Special case for forwarding

x
PC B
PC A ends a packet to PC B
MAC B is not known by left switch flood
MAC B is not known by right switch flood
B receives duplicate frames
MAC A will be learned on wrong port on the lower

access switch blackholing traffic to A
PC A
5
Frames received on Peer-Link may not be flooded
out of VPCs
BRKCRS-1930
Cisco Public
92
Special case for forwarding:

VPC implementation
PC B
MAC B is not known by left switch flood
Frames received from Peer-Link are never sent

out of VPC (except those without operational
ports on ingress switch)
Egress port ASICs will drop the frame
Frame is still flooded to devices that are solely

connected to egress switch
2
PC A
This rule (called VPC check) stands for all traffic

(L2, L3, unicast, multicast, broadcast, flooded etc)
BRKCRS-1930
Cisco Public
93
Summary: VPC traffic forwarding
BRKCRS-1930
X
Cisco Public
94
VPC forwarding and L3 implication

vPC view
Layer 2 topology
Layer 3 topology
7k vPC
7k1
7k1
7k2
7k2
R
R
R could be any router,

L3 switch or VSS
building a port-channel
Port-channel looks like

a single L2 pipe.
Hashing will decide
which link to chose
Layer 3 will use ECMP

for northbound traffic
R can Decide to send to 7k1 at L3 (next-hop = 7k1 if Po) and

uses link to 7k2 at L2 level !!!
Path is R 7k2 7k1 DROPPED (per VPC check) as
incoming on peer-link if it must be routed to another VPC
BRKCRS-1930
Cisco Public
95
Layer 3 and vPC Design update

Use L3 links to hook up routers and peer with a vPC domain
Dont use L2 port channel to attach routers to a vPC domain unless you statically route to
HSRP address
If both, routed and bridged traffic is required, use individual L3 links for routed traffic and L2
port-channel for bridged traffic
Use of peer-gateway does NOT change above recommendations
Switch
Switch
Po2
Po2
7k1
7k2
Po1
L3 ECMP
P
Routing Protocol Peer

Dynamic Peering Relationship
P
BRKCRS-1930
Router
Router
Cisco Public
96
Layer 3 and VPC consideration

Best : use Routed links from VPC pair to routers
Alternative : VPC in a pure L2 VDC and routing in a
separate VDC
Do not make L3 routing protocol peering between
VPC pair of switches on a VPC vlan.
May lead to routing frame towards Peer-link leading to drop
per VPC-Check
If peering between VPC devices is needed, must be done
outside of the peer link
Keep SVI interface administrative status in sync

(both up or both down) This is a type 2
consistency check
BRKCRS-1930
Cisco Public
97
Special case for L2 learning
PC B
2
1
MAC A is learned on lower VPC
MAC A is learned on Peer-Link
Frame destined to A arriving to right switch will be

sent to Peer-Link
A
A
PC A
BRKCRS-1930
Traffic should prefer local links when available

(traffic locality rule)
Cisco Public
99
L2 learning: VPC implementation
PC B
1
2
A
CFS message
MAC A is learned on lower VPC

MAC addresses are never learned from traffic on
Peer-Link
Left switch sends a CFS message to right switch

telling about MAC A learned on lower VPC. Right
switch updates MAC address table
Frame destined to A arriving to right switch will be

sent out of lower VPC
PC A
BRKCRS-1930
Cisco Public
100
Po50
Vlan 50
Troubleshooting
Layer 2
Po22
Vlan 20
VPC
91.0.0.10
0013.1908.e246
20.1.2.3
nexus# sh mac address-table address 0013.1908.e246 vlan 50

VLAN
MAC Address
Type
age
Secure NTFY
Ports
---------+-----------------+--------+---------+------+----+-----------------* 50
0013.1908.e246
dynamic
0
F
F Po50
nexus# sh spanning-tree vlan 50 interface port-channel 50
Mst Instance
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------MST0002
Desg FWD 200
128.4145 (vPC) P2p
nexus# sh hardware mac address-table 2 address 0013.1908.e246 vlan 50
MAC addresses should point

to expected ports in expected
vlans (path towards source)
nexus# sh system internal pixm info ltl 0x00a36 | i Eth.*,
0x0a36
Eth2/36,
The ports should be in STP
nexus# sh mac address-table address 0021.55e0.66c2 vlan 20
forwarding mode
VLAN
MAC Address
Type
age
Secure NTFY
Ports
Hardware MAC address
---------+-----------------+--------+---------+------+----+------------------
* 20
0021.55e0.66c2
dynamic
660
F
F Po22
table should be consistent
nexus# sh spanning-tree vlan 20 interface port-channel 22
with software table
Mst Instance
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Finding port# for given index
MST0000
Desg FWD 200
128.4117 (vPC) Network P2p
Valid| PI | BD
|
MAC
| Index | Stat| SW | Modi| Age | Tmr |
|
|
|
|
| ic |
| fied| Byte| Sel |
-----+----+-------+---------------+--------+-----+----+-----+-----+-----+
1
1
161
0013.1908.e246 0x00a36
0
3
0
141
1
nexus# sh hardware mac address-table 1 address 0021.55e0.66c2 vlan 20

Valid| PI | BD
|
MAC
| Index | Stat| SW | Modi| Age | Tmr |
|
|
|
|
| ic |
| fied| Byte| Sel |
-----+----+-------+---------------+--------+-----+----+-----+-----+-----+
1
1
18
0021.55e0.66c2 0x00a32
0
2
0
103
1
nexus# sh system internal pixm info ltl 0x00a32 | i Eth.*,
0x0a32
Eth1/13, Eth1/14,
BRKCRS-1930
Cisco Public
102
Po50
Vlan 50
Troubleshooting
Layer 3
Po22
Vlan 20
VPC
91.0.0.10
0013.1908.e246
20.1.2.3
nexus# sh routing ip 20.1.2.3

...
20.1.2.3/32, ubest/mbest: 1/0
*via 20.1.1.240, Vlan20, [1/0], 03:48:59, static
nexus# sh ip arp 20.1.1.240
Address
Age
MAC Address
20.1.1.240
00:02:17 0021.55e0.66c2
Interface
Vlan20
nexus# sh forwarding ip route 20.1.2.3 module 2

...
------------------+------------------+--------------------Prefix
| Next-hop
| Interface
------------------+------------------+--------------------20.1.2.3/32
20.1.1.240
Vlan20
nexus# sh forwarding adjacency 20.1.1.240 module 2
IPv4 adjacency information
next-hop
rewrite info
interface
-------------- --------------- ------------20.1.1.240
0021.55e0.66c2 Vlan20
nexus# sh int vl 20 | i address
Hardware is EtherSVI, address is
Is there route to
destination
Is the next hop resolved
Looking at module 2
because this is where
packets in question
should be received
Is adjacency consistent
with ARP
Router MAC must have
Gateway flag in order for
packet to be L3 switched
0023.ac66.1a42
nexus# sh mac address-table address 0023.ac66.1a42 vlan 20

VLAN
MAC Address
Type
age
Secure NTFY
Ports
---------+-----------------+--------+---------+------+----+-----------------G 20
0023.ac66.1a42
static
F
F sup-eth1(R)
BRKCRS-1930
Cisco Public
105
Where given packet will be load-balanced

For equal-cost routes
nexus# sh routing hash 91.0.0.10 20.1.2.3
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Universal-id seed: 0xcdb5769f
Hash for VRF "default"
Hashing to path *20.1.1.3 (hash: 0x2a), for route:
20.1.2.3/32, ubest/mbest: 2/0
*via 20.1.1.3, Vlan20, [1/0], 00:01:37, static
*via 20.1.1.240, Vlan20, [1/0], 16:32:42, static
Load-balancing is configurable
under ip load-sharing address in
default VDC and affects all VDCs
For port-channels
nexus# sh port-channel load-balance forwarding-path interface port-channel 22 dst-ip
20.1.2.3 src-ip 91.0.0.10 vlan 20 module 2
Load-balancing is configurable
Missing params will be substituted by 0's.
under port-channel load-balance
Module 2: Load-balance Algorithm: source-dest-ip-vlan
RBH: 0 Outgoing port id: Ethernet1/14
in default VDC and affects all VDCs
Use sh port-channel rbh-distribution to see which link sends traffic for

which of 8 available load-balancing buckets
BRKCRS-1930
Cisco Public
106
Hardware path packet drops
#1 command to look for hardware

packet drops
Not every drop listed here is actual
data packet drop
|------------------------------------------------------------------------|
| Device:R2D2
Role:MAC
| times to see if any
Run several
|------------------------------------------------------------------------|
Instance:7
counters increase at rate similar to
ID
Name
Value
Ports
traffic loss
------------28688 aric_no_port_select_error
0000000000000002
To clear1,3,5,7
counters,I2use
...
|------------------------------------------------------------------------|
clear statistics
module-all device all
| Device:Ashburton
Role:MAC
Mod: 1
|
nexus# sh hardware internal errors all
---------------------------------------Hardware errors as reported in module 1
----------------------------------------
|------------------------------------------------------------------------|
Instance:0
3629 Egress Port-1 VSL Dropped Packet Count
0000000853635833
5 3630 Egress Port-2 VSL Dropped Packet Count
0000000857893046
3 ...
|------------------------------------------------------------------------|
| Device:Naxos
Role:MAC SECURITY
|
|------------------------------------------------------------------------|
Instance:0
ID
Name
Value
Ports
------------106 m1_fab_p25_txq_tc0_drop_count
00000000000012af
2 ...
|------------------------------------------------------------------------|
| Device:Metropolis
Role:REWR
|
|------------------------------------------------------------------------|
Instance:1
ID
Name
Value
Ports
------------70
Krypton input controller zero portsel cnt
0000000000000038
18,20,22,24,26,28,30,32
|------------------------------------------------------------------------|
| Device:Lamira
Role:L3
|
|------------------------------------------------------------------------|
Instance:0
ID
Name
Value
Ports
------------93
CL2 Invalid Pkt count
00000008759cb9cb
1-32 I1
...
BRKCRS-1930
Cisco Public
109
VPC Agenda
Initialization
Spanning Tree
Traffic forwarding
1st hop redundancy
BRKCRS-1930
Cisco Public
112
1st hop redundancy with VPC

MAC_B vMAC
IP B IP A
PC B
Each of VPC peers will L3 forward packets

destined to its respective Router MAC address
HSRP/VRRP/GLBP used for 1st hop redundancy
Router MAC1
0001.0002.0003
Virtual MAC
0000.0c07.ac00
HSRP
Router MAC2
0005.0006.0007
Virtual MAC
0000.0c07.ac00
Both switches will L3 switch packets to vMAC

address as long as one of them is HSRP active or
HSRP standby.
If both switches are HSRP listening, they will not
L3 switch packets to vMAC
PC A
MAC_A vMAC
IP A IP B
BRKCRS-1930
Cisco Public
113
First hop redundancy troubleshooting

standby
Interface Vlan1
ip address 1.1.1.252/24
hsrp 1
ip 1.1.1.254
Left# sh hsrp brief

Interface
Grp Prio P State
Vlan1
1
100
Standby
active
Interface Vlan1
ip address 1.1.1.253/24
hsrp 1
ip 1.1.1.254
HSRP
Active addr
1.1.1.253
Standby addr Group addr

local
1.1.1.254
Left# sh mac address-table address 0000.0c07.ac01

VLAN
MAC Address
Type
age Secure NTFY
Ports
---------+-----------------+--------+-----+------+------+----------G 1
0000.0c07.ac01
static
False False sup-eth1(R)
Right# sh hsrp brief
Interface
Grp Prio P State
Vlan1
1
100
Active
Active addr
local
Standby addr Group addr

1.1.1.252
1.1.1.254
Right# sh mac address-table address 0000.0c07.ac01

VLAN
MAC Address
Type
age Secure NTFY
Ports
---------+-----------------+--------+-----+------+------+----------G 1
0000.0c07.ac01
static
False False sup-eth1(R)
BRKCRS-1930
Cisco Public
Both peers will L3

forward packets destined
to vMac address as long as
either peer in VPC domain
is in active or standby
state for corresponding
group
Virtual mac address (vMac)
will be installed in both
peers
G (gateway) flag must be
present on any MAC
address for which the
nexus is expected to L3
forward packets
Only active will respond
to ARP for VIP
114
1st hop issue with some devices

MAC_B Router MAC1
IP B IP A
Server B
Router MAC1 MAC_B
MAC_B Router MAC1
IP A IP B
IP B IP A
4
2
Router MAC1
0001.0002.0003
Virtual MAC
0000.0c07.ac00
Router MAC2
0005.0006.0007
Virtual MAC
0000.0c07.ac00
PC A sends a packet to Server B
Left VPC switch will receive the packet and

forward it to Server B, note Source MAC of
outgoing packet will be that of Router1
Server B responding to PC A will populate

destination MAC from source MAC of received
frame (this is wrong, it should use ARP)
If frame from BA will be load-balanced to right

switch the MAC address of Router1 will point to
Peer-Link and this is where the frame will be sent
Left switch will receive the frame from Peer-Link

and drop it
5
PC A
MAC_A vMAC
IP A IP B
BRKCRS-1930
1
Why? Frames received from Peer-Link are never

sent out of VPC except those without operational
ports on ingress switch
(egress port ASICs will drop the frame)
Cisco Public
115
Peer-Gateway : the workaround

MAC_B Router MAC1
IP B IP A
With peer-gateway both peers will install router

MACs of each other in L2 table which will allow
them to L3 forward traffic destined to either
Router MAC
Server B
MAC_B Router MAC1
IP B IP A
2
Router MAC1
0001.0002.0003
Virtual MAC
Router
MAC2
0000.0c07.ac00
0005.0006.0007
Virtual MAC
0000.0c07.ac00
Router MAC2
0005.0006.0007
Virtual MAC
Router
MAC1
0000.0c07.ac00
0001.0002.0003
Virtual MAC
0000.0c07.ac00
Server B responding to PC A will populate

destination MAC from source MAC of received
frame (this is wrong, it should use ARP)
Right switch will forward packet towards

destination
PC A
BRKCRS-1930
Cisco Public
116
Peer-Gateway : the implications
MAC_B Router MAC1

IP TOP IP LEFT, TTL 1
Router MAC1
0001.0002.0003
Router MAC2
0005.0006.0007
Virtual MAC
0000.0c07.ac00
Router MAC2
0005.0006.0007
Router MAC1
0001.0002.0003
Virtual MAC
0000.0c07.ac00
Top device attempts to establish OSPF adjacency

with the left switch
If peer-gateway is enabled in VPC domain and

OSPF unicast packet will be load-balanced to the
right switch, this packet will be dropped
Why? Right switch will try to L3-switch the
unicast packet (because RouterMAC1 is marked
as gateway MAC and destination IP is not local)
As packet has TTL==1 it will be dropped
Same applies to any other protocol that uses
unicast packets with TTL==1 entering right switch
but destined to left switch (or vise versa)
Routing protocol peering with devices attached to

VPC domain via SVI interface is not supported
Routed interface should be used in this case
BRKCRS-1930
Cisco Public
117
More information
sh mac address-table <address>
(L2 entry for given MAC )
sh hardware mac address-table <mod> address <address>

(hardware L2 entry for given MAC should be consistent with
above)
sh system internal l2fm l2dbg macdb address <addr>
(history of changes for given mac address)
sh tech hsrp
(from both sides of VPC)
BRKCRS-1930
Cisco Public
118
VPC Agenda
Initialization
Spanning Tree
Traffic forwarding
1st hop redundancy
BRKCRS-1930
Cisco Public
119
IP Multicast with VPC

Receiver sends IGMP report (join)
RP
Access switch sends join to right VPC peer

Right VPC peer creates (*,G) adds VPC to OIF (as
proxy-DR)
Source S1
IGMP is encapsulated in CFS and sent to left peer

Left peer (DR) creates (*,G) adding VPC to OIF
(*,G)VPC
(*,G)VPC
Primary
2ndary
CFS:IGMP
(S1,G)VPC
DR
(S1,G)null
Proxy-DR
DR (left peer) sends PIM Join to RP

Once (S1,G) traffic starts arriving, VPC peers will
resolve which one will be forwarder for that (S,G):
peer with best metric to source or primary in a tie
(this mechanism is specific to PIM in VPC mode,
normally PIM would use assert)
Only forwarder will have OIFs populated in (S,G)
the non-forwarder wont have VPC SVIs in OIF list
Receiver
Forwarder will send a copy of frame to the peerlink for receivers single-connected to other peer
IGMP join
Goal is to allow peer that 1st sees source traffic to forward it to receivers behind VPC
BRKCRS-1930
Cisco Public
120

Prebuilt-SPT
RP
In case of DR failure proxy-DR becomes DR and

posts OIF-list from (*,G) to (S,G), but it will also
need to pull traffic from RP/source which delays
recovery
Source S1
(*,G)VPC
(*,G)VPC
Primary
(S1,G)VPC
2ndary
With ip pim pre-build-spt proxy-DR will also send

a PIM Join to source/RP to draw the traffic
(S1,G)null
Traffic pulled by proxy-DR will be dropped until it

becomes DR provision uplink accordingly (if
pre-build-spt is used)
DR
Receiver
BRKCRS-1930
Cisco Public
121

source behind VPC
When Source is behind VPC both DR and ProxyDR will add OIFs for the group to (S,G)
RP
This is because either peer can receive source

traffic and need to be able to send it to receivers
behind VPCs without crossing peer-link (to keep
traffic locality and to avoid dropping the traffic by
VPC check)
(*,G)VPC2
(*,G)VPC2
Primary
Going to Left switch from Source
2ndary
(S1,G)VPC2
(S1,G)VPC2
Or going to Right switch from Source
Source S1
BRKCRS-1930
VPC1
VPC2
Receiver
Cisco Public
122
Which of VPC peers will be forwarder

Peers do metrics exchange over CFS for each new source
Peer that has better metric to source or primary will be forwarder
VPC1# sh ip pim internal vpc rpf
Source: 10.0.1.1
Pref/Metric: 110/21
Source role: primary
Forwarding state: Win (forwarding)
For sources behind VPC both peers will forward as they have no control on which
one will get the traffic
VPC1# sh ip pim internal vpc rpf
Source: 1.1.1.1
Pref/Metric: 0/0
Source role: primary
Forwarding state: Win-force (forwarding)
BRKCRS-1930
Cisco Public
123
VPC mcast: following packet flow

Nexus# show ip mroute 239.1.2.3
(*, 239.1.2.3/32), uptime: 06:46:05, igmp pim ip static
Incoming interface: Vlan36, RPF nbr: 36.0.0.3
Outgoing interface list: (count: 2)
Ethernet2/43, uptime: 03:01:36, static
Vlan37, uptime: 06:46:05, igmp
control plane state for this group

where information came from
stable?
RPF interface
where are receivers on this vlan?
Is traffic being switched for this group?

counters updated once ~1 minute
packets forwarded in software
average packet size
(33.0.0.33/32, 239.1.2.3/32), uptime: 06:46:05, ip pim mrib

Incoming interface: Vlan36, RPF nbr: 36.0.0.3
Outgoing interface list: (count: 2)
Ethernet2/43, uptime: 03:01:36, mrib
Vlan37, uptime: 06:46:04, mrib
Nexus# show ip igmp snooping groups vlan 37
Type: S - Static, D - Dynamic, R - Router port
Vlan
37
37
Group Address
*/*
239.1.2.3
Ver
v2
Type
R
D
Port list
Vlan37
Eth2/8
Are packets being switched by this entry?

Nexus# show ip mroute 239.1.2.3 summary software-forwarded
Total
Total
Total
Total
Group
number
number
number
number
count:
of
of
of
of
1,
routes: 3
(*,G) routes: 1
(S,G) routes: 1
(*,G-prefix) routes: 1
rough average sources per group: 1.0
Group: 239.1.2.3/32, Source count: 1

Source
packets
bytes
(*,G)
0
0
sw-pkts: 0
33.0.0.33
5046908
252345396
sw-pkts: 1
BRKCRS-1930
aps
0
pps
0
bit-rate
0.000
bps
49
200
80.053
Cisco Public
oifs
2
kbps 2
125
Following the flow: forwarding information

Nexus# show forwarding multicast route group 239.1.2.3
slot 1
=======
(*, 239.1.2.3/32), RPF Interface: Vlan36, flags: G
Received Packets: 0 Bytes: 0
Number of Outgoing Interfaces: 2
Outgoing Interface List Index: 4
Vlan37 Outgoing Packets:0 Bytes:0
Ethernet2/43 Outgoing Packets:N/A Bytes:N/A
(33.0.0.33/32, 239.1.2.3/32), RPF Interface: Vlan36, flags:

This is platform independent forwarding
information
Ethernet2/43 Outgoing Packets:N/A Bytes:N/A
Ingress linecard entry

Egress linecard entry
Counters are updated once per ~1minute
Counters between ingress/egress do not have to
match, as information is collected not at the same
exact time, receiver might join after the entry was
created etc
Ethernet2/43 Outgoing Packets:3032294 Bytes:194066816
(*, 239.1.2.3/32), RPF Interface: Vlan36, flags: G
slot 2
=======
(33.0.0.33/32, 239.1.2.3/32), RPF Interface: Vlan36, flags:

Ethernet2/43 Outgoing Packets:3032294 Bytes:194066816
BRKCRS-1930
Cisco Public
126
When traffic arrives via VPC

How to find which slot receives the
S,G flow when ingress interface is
port-channel scattered across
several modules?
VPC domain 100
VPC
show forwarding multicast route

group <g> source <s>
Nexus# show forwarding multicast route group 239.1.1.1 source 1.0.1.2 | i Received|slot
slot 1
slot 2
BRKCRS-1930
Cisco Public
127
Following the flow: hardware entries

Nexus# show system internal forwarding ipv4 multicast route group 239.1.2.3 source 33.0.0.33 detail
slot
(33.0.0.33/32, 239.1.2.3/32), Flags: *S

Lamira: 1, HWIndex: 0x2200, VPN: 1
RPF Interface: Vlan36, LIF: 0x45, PD oiflist index: 0x2
ML3 Adj Idx: 0xa016, MD: 0x2003, MET0: 0x2004, MET1: 0x2004, MTU Idx: 0x1
Metro Instance: 0
Dev: 1 Index: 0xa019
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
recirc-dti: 0xe20000
Ingress forwarding engine (FE)
Metro Instance: 1
replicates packets to receivers on that
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
linecard and creates distribution copy
Metro Instance: 2
of the packet for other linecards
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
MET pointers (MD + MET0)
Metro Instance: 3
RPF interface read from entry
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
TCAM Entry
Decoded MET chain (on ingress there

is only MD copy created)
(33.0.0.33/32, 239.1.2.3/32), Flags: *S
Egress linecard will receive distribution
Lamira: 1, HWIndex: 0x2200, VPN: 1
copy and replicate it to receivers (using
RPF Interface: Vlan36, LIF: 0x45, PD oiflist index: 0x2
ML3 Adj Idx: 0xa026, MD: 0x2003, MET0: 0x2004, MET1: 0x2004,
MTUpointer)
Idx: connected
0x1
MET1
to the card
Metro Instance: 0
MET1 on egress linecard points to
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
receivers on vlan37 and e2/43
slot
Dev: 1 Index: 0x6046

Metro Instance: 1

BRKCRS-1930
Type: OIF
dest idx: 0x0
elif: 0x80046
Vlan37
smac: 001b.54c2.4241
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
Type: OIF
elif: 0x84029
Ethernet2/43
dest idx: 0x44c
smac: 001b.54c2.4241
Cisco Public
128
Are there drops in forwarding path?

Start looking from Ingress module
Nexus# show hardware internal errors module 1
---------------------------------------Hardware errors as reported in module 1
---------------------------------------...
|------------------------------------------------------------------------|
| Device:Lamira
Role:L3
Mod: 1
|
| Last cleared @ Thu Apr 8 12:57:37 2010
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:0
ID
Name
Value
Ports
------------259 L3 Fib Miss Pkt ctr
0000000000000007
1-32 I1
262 L3 Non-Rpf Drop Pkt ctr
0000000000125617
1-32 I1
319 NF2 V4 IPMAC Lkup Error
0000000000272277
1-32 I1
455 Exception cause: DROP (Unicast)
0000000000025510
1-32 I1
465 Exception cause: DROP (Multicast)
0000000000226148
1-32 I1
Always take several snapshots and look for drops that grow coherently with
[suspected] multicast traffic drops
There are always some drops shown by above command this doesnt always
mean the actual network packets are dropped. Some of these are diag packets,
some are packets that are dropped on blocked ports, extra floods etc
BRKCRS-1930
Cisco Public
129
Wrapping UP
BRKCRS-1930
Cisco Public
VPC compared to VSS

VPC
VSS
Control Plane
SSO
HSRP/VRRP
Distributed
InTRAchassis (w/2 sups)
2 routers, each forwards
traffic
Traffic locality
Failover time
Configuration
synchronization
Yes
Subsecond
Separate configs, key
parameters checked via
CFS
via the Peer-Keepalive link via L2 hellos and
PAgP+
Dual active
detection
BRKCRS-1930
Cisco Public
Redundant Centralized
InTERchassis
Inherent 1st hop
redundnancy, no need
for HSRP
Yes
Subsecond
Using IOS redundancy
framework
132
VPC/VSS: summary
Remember about the implications of 2 control planes
and 2 data planes active at the same time
Pay special attention to configuration and operational

consistency, not only to what is enforced, but also L3
interfaces including their operational state, FHRP
config, ACL config, queueing config
Troubleshoot like a standalone switch 1st, then dive into
VPC/VSS specifics: main one being traffic locality
Both VPC and VSS

simplify logical Layer 2 topology
use Traffic Locality for efficient shortest path
forwarding
BRKCRS-1930
Cisco Public
133
BRKCRS-1930
Recommended Reading
Also browse on-site Cisco Store for suitable reading
BRKCRS-1930
Cisco Public
134
Please complete your Session Survey
We value your feedback - don't forget to complete your online

session evaluations after each session. Complete 4 session
evaluations & the Overall Conference Evaluation (available from
Thursday) to receive your Cisco Networkers 20th Anniversary t-shirt.
All surveys can be found on our onsite portal and mobile website:
www.ciscoliveeurope.com/connect/mobi/login.ww
You can also access our mobile site and

complete your evaluation from your mobile
phone:
1. Scan the Access Code
(See http://tinyurl.com/qrmelist for software,
alternatively type in the access URL)
2. Login
3. Complete and Submit the evaluation
BRKCRS-1930
Cisco Public
135

VSS VS VPC PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

VSS VS VPC PDF

Uploaded by

Copyright:

Available Formats

VPC & VSS: Operation and Troubleshooting

VSS and VPC

No blocked ports, More usable bandwidth, Load-sharing

2011 Cisco and/or its affiliates. All rights reserved.

2011 Cisco and/or its affiliates. All rights reserved.

Spirit of this session

Simple description on how things work

More on the topic

2011 Cisco and/or its affiliates. All rights reserved.

2011 Cisco and/or its affiliates. All rights reserved.

1st hop redundancy

2011 Cisco and/or its affiliates. All rights reserved.

2 active data planes

Standby switch is essentially a

(can be seen as backplane

Special encapsulation on VSL

2011 Cisco and/or its affiliates. All rights reserved.

Link Bringup to establish connectivity with remote chassis

Role Resolution Protocol (RRP) used to determine compatible Hardware and

2011 Cisco and/or its affiliates. All rights reserved.

Troubleshooting VSS: quick sanity check

vss# sh switch virtual link

vss# sh switch virtual link port

2011 Cisco and/or its affiliates. All rights reserved.

Aside from packet/bit rate this is

vss# sh switch virtual link counters

Complete information about LMP

Troubleshooting VSL: LMP

sm(vslp_lmp 6/4), running yes, state operational

Troubleshooting VSL: LMP

Compared to previous command

v - Valid flag set

Timers: T4 - Hello Tx Timer

f - Bi-directional flag set

2011 Cisco and/or its affiliates. All rights reserved.

One of the switches must be

vss# sh switch virtual role detail

Switch Status Preempt

2011 Cisco and/or its affiliates. All rights reserved.

All ports on both sides of VSL

vss# sh switch virtual link port-channel

2011 Cisco and/or its affiliates. All rights reserved.

2011 Cisco and/or its affiliates. All rights reserved.

1st hop redundancy

2011 Cisco and/or its affiliates. All rights reserved.

2011 Cisco and/or its affiliates. All rights reserved.

2011 Cisco and/or its affiliates. All rights reserved.

VSS with 4 supervisors

Before switching to linecard mode

If supervisor fails completely

12.2(33)SXI4 and later

2011 Cisco and/or its affiliates. All rights reserved.

For faster failovers assumption is

L2 MEC will be error-disabled

Dual-active, if not detected will cause severe network outage

2011 Cisco and/or its affiliates. All rights reserved.

Dual-Active Detection options

Requires PAGP+ capable neighbor with

VSLP Fast Hello

L2 Heart Beat Link

L3 Heart Beat Link

Enhanced subsecond detection in