Troubleshooting Cisco routers and switches

Advanced feature troubleshooting
and best practices on Cisco

Routers and Switches
Ivan Shirshin, CSE at High Touch Technical Services
Jaroslaw Gawron, CSE at Global Technical Assistance Centre
Date: November, 4 2016

Agenda
1. Introduction.
2. Overview of customer network.
3. Problem 1: Voice quality issues.
4. Problem 2: Statistics collection failure.
5. Problem 3: IPv6 connectivity loss.
6. Problem 4: Wireless deployment in office B.
7. Conclusion.
Introduction
Goal of this Session
Create awareness of the troubleshooting tools and techniques

available on Cisco switching and routing platforms,
Show how this tools apply in troubleshooting of real world issues,
Show a DEMO of tools implementation in the lab setup.

Define the problem, impact & scope
Define the problem based on the facts and user reports,
considering recent changes (software / hardware / ):
E.g., Application slowness after recent migration
Understand the impact of the problem based on the information

gathered:
E.g., Does that application have slow performance throughout
the day? Or is it intermittent?
Determine the scope of the impact:

E.g., Do all the users experience this issue? Or is it only a
specific floor/building?
Our focus
1) Gain knowledge of troubleshooting capabilities on Cisco switches

and routers.
2) Improve ability to choose the right troubleshooting tool for timely

problem resolution.
Scope/devices
This sessions covers:
Routers running IOS-XE ISR4000, ASR1000

6800, IA
Catalyst switches 3650, 4500, 3850
Overview of customer network
Customer network topology
Customer network topology (cont.)
Problem #1.
Users connected on financial department
have voice quality issues.
Problem #1
Problem definition:
Users from the financial department, connected to Fex101 in HQ are
complaining of voice quality issues when communicating over phones
with Office B
Impact:
Medium users are not able to perfrom day-to-day conversations
Scope :
Issue limited to HQ and Office B communication
System health check
Complex commands, require architecture knowledge:
show platform software process list RP active summary
show process
show platform software process list RP active summary
show platform software process list RP active | inc fman
show platform hardware qfp active infrastructure exmem statistics
show platform hardware qfp active infrastructure exmem statistics user
New, simplified syntax:

show platform resources
Use show <something> platform to show processes from the underlying operating system.
QFP Memory Utilization ASR1K# show platform hardware qfp active infrastructure exmem
statistics
Type: Name: DRAM, QFP: 0

omplex commands Total: 3758096384
InUse: 95705088
Free: 3662391296
Lowest free water mark: 3662391296
Type: Name: IRAM, QFP: 0
Total: 134217728
InUse: 7980032
Free: 126237696
Lowest free water mark: 126237696
Type: Name: SRAM, QFP: 0
...
ASR1K# show platform hardware qfp active infrastructure exmem statistics user
...
10 279092 284672 CEF
40 36441494 36458496 NAT
ESP FECP Chassis Manager
Drivers Forwarding Manager

Drivers
ASR1K# show platform hardware qfp active tcam resource-manager usage Drivers
Linux Kernel
QFP TCAM Usage Information
<snip>
QFP
Crypto
Total TCAM Cell Usage Information
BQS
Assist.
----------------------------------
Name : TCAM #0 on CPP #0
Total number of regions : 3
Total tcam used cell entries : 28
Total tcam free cell entries
Threshold status
:
:
524260
below critical limit
TCAM DRAM DRAM
Resources: simplified view IOS 3.14
RP
CPU
Chassis Manager
IOS
Forwarding Manager
ASR1K#show platform resources Linux Kernel

**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical
State ESP FECP
-------------------------------------------------------------------------------------------------- Chassis Manager
--
RP0 (ok, active) H Drivers Forwarding Manager
Drivers
Control Processor 1.01% 100% 90% 95% H Drivers
Linux Kernel
DRAM 2330MB(23%) 9833MB 90% 95% H
ESP0(ok, active) H
QFP
QFP H BQ Crypto
DRAM 22140KB(2%) 1048576KB 80% 90% H S Assist.

IRAM 0KB(0%) 0KB 80% 90% H
SIP
IOCP SPA
SPA
Driver Chassis
SPA
Driver Manager
Driver
Linux Kernel
SPA SPA SPA

Packet trace
Debugging strategies
Traffic did not reach its target !

What happened to that packet ?
Why did that happen ?
Everyday situations
IPsec ZBF NAT
WAAS SNMP
OTV
Routing
Which feature went wrong ?
What went wrong in the feature ?

Memory
Config Performance Ordering
Bug
Traffic Ambiguity
issue
The Embedded Packet Capture IOS 3.7
One way of capturing packets

Device# monitor capture mycap start
Device# monitor capture mycap access-list v4acl
Device# monitor capture mycap limit duration 1000
Device# monitor capture mycap interface GigabitEthernet 0/0/1 both
Device# monitor capture mycap buffer circular size 10
Device# monitor capture mycap start
Device# monitor capture mycap export tftp://10.1.88.9/mycap.pcap
Device# monitor capture mycap stop Shows whether packets have been received or sent
Shows what packets look like
Device# show monitor capture mycap buffer dump Requires hex dump analysis or export to decoder (sniffer)
Does not tell us what happened to the packet
0
0000: 01005E00 00020000 0C07AC1D 080045C0 ..^...........E.
0010: 00300000 00000111 CFDC091D 0002E000 .0..............
0020: 000207C1 07C1001C 802A0000 10030AFA .........*......
0030: 1D006369 73636F00 0000091D 0001 ..example.......
1
0000: 01005E00 0002001B 2BF69280 080046C0 ..^.....+.....F. Excellent tool but insufficient in many cases
0010: 00200000 00000102 44170000 0000E000 . ......D.......
0020: 00019404 00001700 E8FF0000 0000 ..............
2 http://www.cisco.com/en/US/docs/ios-
0000: 01005E00 0002001B 2BF68680 080045C0 ..^.....+.....E.
0010: 00300000 00000111 CFDB091D 0003E000 .0.............. xml/ios/epc/configuration/xe-3s/asr1000/nm-packet-capture-
0020: 000207C1 07C1001C 88B50000 08030A6E ...............n xe.html
0030: 1D006369 73636F00 0000091D 0001 ..example.......
IOS 3.10
The Packet Tracer and FIA Debugger

X-ConnectReset / Pwr Ctrl
L2 Switch IPv4 IPv6 MPLS
TCAM Resource DRAM
Packet Buffer Part Len / BW Packet # 16
Condition determines
Temp Sensor
DRAM SRAM
packets to be traced Output FIA Input ACL

Input FIA
EEPROM Optionally match
QFP
DDRAM
Pak Match ?
PPE2
Packet Processor EngineComplex
on the egress FIA
BQS
MQC Classify
Output ACL NAT

Boot Flash Input ACL
(OBFL,) FECP PPE1 PPE2 PPE3 PPE4 PPE5
PBR
Thread 1
Thread 2
Thread 3
Thread 4
MQC Classify NAT
JTAG Ctrl PPE6 PPE7 PPE8 PPEN
NAT Encaps Output ACL
IP Unicast
PBR Statistics and final action will be NAT
Dispatcher Crypto
collected (matched packets dropped,
Packet Buffer punted to RP, forwarded to output Encaps
interface )PPE2
Crypto Thread 3 Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
Optionally, FIA actions can logged per packet
System can capture several packets flows
RPs RPs ESP RPs SIPs Packet flows can be reviewed in show commands
Packet-Trace: Details
Designed to address the challenge with troubleshooting datapath
issues in live high-scale environment
Packet-trace provides visibility into the treatment of packets of an IOS-XE
platform to troubleshoot, diagnose, or gain a deeper understanding of the
actions taken on a packet during packet processing.
Integrated platform condition debugging (debug platform condition), making it
a viable option even under heavy traffic situations seen in production
environments.
Three specific levels of inspection are provided by packet-trace. Each level
adds a deeper look into the packet processing at the expense of some packet
processing overhead. Packet Trace is supported on the
ASR1000, ISR4000, and CSR1000V,
introduced in XE3.10
Packet-Trace: Accounting
Accounting keeps track of all interesting packets that enter
and leave the packet processor. There are three count groups:
Summary counts
Packets Matched packets that matched conditions
Packets Traced packets that were traced
Arrival counts
Ingress packets entering via external interfaces
Inject* number of packets seen as injected from control plane
Departure counts
Forward number of packets scheduled/queued for delivery
Punt* number of packets punted to control plane
Drop* number of packets specifically dropped by packet processing
Consume number of packets consumed (e.g. ping request)
Packet-Trace: Summary Data
When enabled, summary data is collected for a specified
number of packets and includes:
Packet number (pactrac specific packet number)
Input interface
Output interface
Final packet state and any punt/drop/inject codes
Summary data collection incurs minimal overhead over normal packet

processing
Often used to isolate specific drop conditions so more detailed inspection can
be used after applying specific conditions
Packet-Trace: Path Data
Path data may be collected per packet for a limited number of
packets and is made up of different types of data as follows:
Common path data (e.g. IP tuple)
Feature specific data (e.g. NAT)
Feature Invocation Array (FIA) trace optionally enabled
Copy of all or part of the incoming and/or outgoing packet optionally enabled
Capturing path data potentially has significant impact on packet processing

capability specifically FIA trace and packet copy.
FIA tracing creates many path data entries costing instructions and DRAM writes
Packet copy creates many DRAM read/writes
Packet-trace will only affect the performance of packets traced (i.e. those
matched by the user provided conditions)
Packet Trace: Memory impact
Packet trace buffers consume QFP DRAM, so be mindful of the amount of memory that a
configuration requires and the amount of memory that is available.
The QFP DRAM usage can be estimated with this formula:
memory needed = (stats overhead) + num of pkts * (summary size + path data size
+ copy size)
You can check the current data-plane DRAM memory consumption by using the show
platform hardware qfp active infrastructure exmem statistics command.
Note: While the stats overhead and summary size are fixed at 2 KB and 128 B, respectively, the path
data size and copy size are user-configurable.
Configuring Packet Tracing (Part 1)
debug platform packet-trace copy packet {in | out | both} [L2 | L3 | L4] [size num-bytes ]
Needs to be done first to enable packet tracing
debug platform packet-trace packet pkt-num [fia-trace | summary-only] [circular] [data-size

data-size ]
pkt-num Specifies the maximum number of packets maintained at a given time.
fia-trace Provides detailed level of data capture, including summary data, feature-spec
summary-only Enables the capture of summary data with minimal details.
circular Saves the data of the most recently traced packets.
data-size Specifies the size of data buffers for storing feature
debug platform packet-trace copy packet {in | out | both} [L2 | L3 | L4] [size num-bytes ]
Enables copying of the packet
Used to specify ingress, egress or both
Optionally, allows specifying the layer of the packet in which the packet copy should start (default L2)
debug platform packet-trace drop [code code-num ]

Limits the packet trace to only dropped packets
Optionally code specifies limiting to a specific drop code
debug platform condition start
Enables the specified matching criteria and starts packet tracing
debug platform condition stop

Deactivates the condition and stops packet tracing.
show platform packet-trace {configuration | statistics | summary | packet {all | pkt-num }
[decode] }
Displays packet-trace data
configuration - Displays packet trace configuration, including any defaults.
statistics - Displays accounting data for all the traced packets.
summary - Displays summary data for the number of packets specified.
{all | pkt-num } [decode] - Displays the path data for all the packets or the packet specified. The
decode option attempts to decode the binary packet into a more human- readable form.
clear platform condition all

Removes the configurations provided by the debug platform condition and debug platform
packet-trace command
Packet Trace
Workflow
Enable and
Define Define Start/Stop,
Buffer Condition View
Criteria
ASR1000 CSR1000V ISR4000

Packet Trace usage summary
Configuration Steps
IOS-XE 15.3(3)S / 3.10(0)S release and later:
debug platform packet-trace enable
debug platform packet-trace packet 8192 circular fia-trace data-size 2048
debug platform packet-trace copy packet both L3 size 64
debug platform condition ipv4 access-list 101 both
debug platform condition start
debug platform condition stop
Review Data
show platform packet-trace summary
show platform packet-trace packet all
show platform packet-trace packet 5
Verify/Clear Configuration
show platform packet-trace configuration
clear platform condition all
Cisco ASR1000 Packet Flow
Reset / Pwr Ctrl Packet Buffer Part Len / BW
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP
DDRAM Packet Processor EngineComplex BQS
Boot Flash
(OBFL,) FECP PPE1 PPE2 PPE3 PPE4 PPE5
JTAG Ctrl PPE6 PPE7 PPE8 PPEN
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
RPs RPs ESP RPs SIPs

Cisco ISR 4400 Packet Flow
Control Plane Data Plane, QFP (6 4xSGMII
DRAM (1 core) and Services or 10 cores)
Plane (3 cores) FPGE
PPE
PPE PPE PPE
4xPCIe
PPE PPE PPE HQF
PPE PPE PPE DRAM
Mgt Eth 4xPCIe System 10G XAUI

FPGA 1 Gb SGMII
Cons/Aux ISC
Platform 10 Gb/slot
Multigigabit
Controller
Fabric
USB Hub SM-X
2 Gb/slot
Flash
NIM
Cisco ISR 4300 Series Architecture
CP/SP Cores Data Plane Cores
IOS
FPGE
Service Container
Multigigabit ISC
Fabric
ISR-WAAS SM-X
KVM - Hypervisor
Service Plane
(control plane CPU) NIM Note:4321 uses 2DP, 1CP & 1SC cores
ISR4K# debug platform packet-trace enable
Please remember to turn on 'debug platform condition start' for packet-trace to work
ISR4K# debug platform packet-trace packet 16 fia-trace data-size 2048

ISR4K# debug platform packet-trace copy packet output size 64 L2
ISR4K# debug platform condition ipv4 30.30.30.30/32 egress
ISR4K# debug platform condition start

ISR4k#show platform packet-trace summary
Pkt Input Output State Reason
0 Gi0/0/2 Gi0/0/3 FWD
1 Gi0/0/2 Gi0/0/3 FWD
2 Gi0/0/2 Gi0/0/3 FWD
3 Gi0/0/2 Gi0/0/3 FWD
4 Gi0/0/2 Gi0/0/3 FWD
5 Gi0/0/2 Gi0/0/3 FWD
6 Gi0/0/2 Gi0/0/3 FWD
7 Gi0/0/2 Gi0/0/3 FWD
8 Gi0/0/2 Gi0/0/3 FWD
Packet Trace HQ packet FIA trace
ISR4k#show platform packet-trace packet 0
Packet: 0 CBUG ID: 7591099
Summary
Input : GigabitEthernet0/0/2
Output : GigabitEthernet0/0/3
State : FWD
Timestamp
Start : 666522942675396 ns (11/03/2016 09:57:31.291323 UTC)
Stop : 666522942684566 ns (11/03/2016 09:57:31.291333 UTC)
Path Trace
Feature: IPV4
Source : 30.30.30.30
Destination : 40.40.40.40
Protocol : 17 (UDP)
SrcPort : 800
DstPort : 1678
Feature: FIA_TRACE
Entry : 0x11081e8c - IPV4_VFR_REFRAG
Lapsed time: 3680 ns
Feature: FIA_TRACE
Entry : 0x11081f4c - IPV4_OUTPUT_L2_REWRITE
Lapsed time: 3380 ns
<...>
Packet Trace HQ packet decode
ISR4K# show platform packet-trace packet 3 decode
...
Packet Copy Out

0062ecda c20454a2 7420e593 080045b8 00b20000 00003e11 eef71e1e 1e1e2828
28280320 068e009e a4690001 02030405 06070809 0a0b0c0d 0e0f1011 12131415
ARPA
Destination MAC : 0062.ecda.c204
Source MAC : 54a2.7420.e593
Type : 0x0800 (IPV4)
IPv4
Version
Header Length
: 4
: 5 ef
ToS : 0xb8
Total Length : 178
Identifier : 0x0000
IP Flags : 0x0
Frag Offset : 0
TTL : 62
Protocol : 17 (UDP)
Header Checksum : 0xeef7
Source Address : 30.30.30.30
Destination Address : 40.40.40.40
UDP
Source Port : 800
Destination Port : 1678
Length : 158
Checksum : 0xa469
ASR1K-2# debug platform packet-trace enable
Please remember to turn on 'debug platform condition start' for packet-trace to work
ASR1K-2# debug platform packet-trace packet 16 fia-trace data-size 2048

ASR1K-2# debug platform packet-trace copy packet output size 64 L2
ASR1K-2# debug platform condition interface gi0/0/0 ingress
ASR1K-2# debug platform condition start

ASR1K-2# show platform packet-trace summary
0 Gi0/0/0 Gi0/0/2 DROP 20 (QosPolicing)
Packet Trace Office B packet FIA trace
ASR1K-2# show platform packet-trace packet 3 <continued>
Packet: 3 CBUG ID: 15071590486 Feature: FIA_TRACE
Summary Entry : 0x4068ed34 - GOTO_OUTPUT_FEATURE
Input : GigabitEthernet0/0/0 Lapsed time: 360 ns
Output : GigabitEthernet0/0/2 Feature: FIA_TRACE
State : DROP 20 (QosPolicing) Entry : 0x403e44a0 - IPV4_MC_INPUT_VFR_REFRAG
... Lapsed time: 53 ns
Path Trace Feature: FIA_TRACE
Feature: IPV4 Entry : 0x40740534 - MPLS_OUTPUT_L2_REWRITE
Source : 30.30.30.30 Lapsed time: 333 ns
Destination : 40.40.40.40 Feature: FIA_TRACE
Protocol : 17 (UDP) Entry : 0x4073640c - OUTPUT_DROP
SrcPort : 800 Lapsed time: 66 ns
DstPort : 1678 Feature: FIA_TRACE
Feature: FIA_TRACE Entry : 0x4026f940 - IPV4_OUTPUT_QOS
Entry : 0x4043e894 - IPV4_INPUT_DST_LOOKUP_CONSUME Lapsed time: 10466 ns
Lapsed time: 173 ns
Feature: FIA_TRACE
Entry : 0x40437a00 - IPV4_INPUT_IPOPTIONS_PROCESS
Lapsed time: 66 ns
<...>
Packet Trace Office B packet decode
ASR1K-2# show platform packet-trace packet 3 decode
...
454800b2 00000000 3c11f167 1e1e1e1e 28282828 0320068e 009ea469 00010203
04050607 08090a0b 0c0d0e0f 10111213 14151617 18191a1b 1c1d1e1f 20212223
IPv4
Version
Header Length
: 4
: 5
af21
ToS : 0x48
Total Length : 178
Identifier : 0x0000
IP Flags : 0x0
Frag Offset : 0
TTL : 60
Protocol : 17 (UDP)
Header Checksum : 0xf167
Source Address : 30.30.30.30
Destination Address : 40.40.40.40
UDP
Source Port : 800
Destination Port : 1678
Length : 158
Checksum : 0xa469
Office B QoS configuration
Configure a workaround
ASR1K-2# show policy-map interface Gi0/0/3
GigabitEthernet0/0/3
Service-policy output: DROP_af21
Class-map: class_af21 (match-all)

1247330 packets, 239487360 bytes
5 minute offered rate 2228000 bps, drop rate 2224000 bps
Match: dscp af21 (18)
police:
cir 200000 bps, bc 6250 bytes, be 6250 bytes
conformed 2038 packets, 391296 bytes; actions:
transmit
exceeded 32 packets, 6144 bytes; actions:
drop
violated 1245260 packets, 239089920 bytes; actions:
drop
conformed 4000 bps, exceeded 1000 bps, violated 2224000 bps
ASR1K-2#conf t
Enter configuration commands, one per line. End with CNTL/Z.
ASR1K-2(config)#interface Gig0/0/3
ASR1K-2(config-if)#no service-policy output DROP_af21
ASR1K-2(config-if)#end
Packet Trace Office B summary, no drops
ASR1K-2#debug platform condition stop
ASR1K-2#clear platform packet-trace statistics
ASR1K-2#debug platform condition start
ASR1K-2#show platform packet-trace summary
0 Gi0/0/0 Gi0/0/2 FWD

1 Gi0/0/0 Gi0/0/2 FWD
2 Gi0/0/0 Gi0/0/2 FWD
3 Gi0/0/0 Gi0/0/2 FWD
4 Gi0/0/0 Gi0/0/2 FWD
5 Gi0/0/0 Gi0/0/2 FWD
6 Gi0/0/0 Gi0/0/2 FWD
7 Gi0/0/0 Gi0/0/2 FWD
Packet Trace Office B packet after the fix
ASR1K-2# show platform packet-trace packet 0 <continued>
Packet: 0 CBUG ID: 15266286095 Feature: FIA_TRACE
Summary Entry : 0x4068ed34 - GOTO_OUTPUT_FEATURE
Input : GigabitEthernet0/0/0 Lapsed time: 360 ns
Output : GigabitEthernet0/0/2 Feature: FIA_TRACE
State : FWD Entry : 0x403e44a0 - IPV4_MC_INPUT_VFR_REFRAG
Lapsed time: 66 ns
Path Trace Feature: FIA_TRACE
Feature: IPV4 Entry : 0x40740534 - MPLS_OUTPUT_L2_REWRITE
Source : 30.30.30.30 Lapsed time: 333 ns
Destination : 40.40.40.40 Feature: FIA_TRACE
Protocol : 17 (UDP) Entry : 0x4002d670 - MARMOT_SPA_D_TRANSMIT_PKT
SrcPort : 800 Lapsed time: 3093 ns
DstPort : 1678
Feature: FIA_TRACE
Entry : 0x40722130 - DEBUG_COND_INPUT_PKT
Lapsed time: 440 ns
Feature: FIA_TRACE
Entry : 0x4043e894 - IPV4_INPUT_DST_LOOKUP_CONSUME
Lapsed time: 160 ns
Feature: FIA_TRACE
Entry : 0x4043e414 - IPV4_INPUT_FOR_US_MARTIAN
Lapsed time: 106 ns
<...>
46
Packet trace new features IOSd trace
Details of a packet that has been traced.
asr1000# show platform packet-trace packet 0
Packet: 0 CBUG ID: 14231008

Destination:
Summary PUNT to RP
Input : GigabitEthernet2
Output : internal0/0/rp:0
State : PUNT 11 (For-us data)
<removed output>
Path Trace Feature: IPV4
Output : <unknown>
Source : 172.16.1.1
GRE tunnel source
Protocol : 47 (GRE)
Packet Copy In <removed output>
and destination
addresses.
Packet trace new features IOSd trace
Details of a packet that has been traced. (cont.)
Inbound tracing
IOSd Path Flow: Packet: 0 CBUG ID: 14231008 reported by IOSd
Feature: INFRA
Pkt Direction: IN
Packet Rcvd From CPP IP input on IOSd
Feature: IP
Pkt Direction: IN
received the packet
Packet Enqueued in IP layer
Source : 172.16.1.1 GRE tunnel source
Interface : GigabitEthernet2 and dest addresses
Feature: IP
Pkt Direction: IN
Source : 172.16.10.10
Traffic through the
Destination : 172.16.101.10 box post GRE
decapsulation
GRE encapsulated traffic transporting traffic through the box
Traffic punted instead of being processed by QFP
Excess of traffic dropped by QFP policers protecting the path to the RP
Packet trace new features - Interface
Starting 16.3, source/destination interfaces are now shown as Unknown or exact interface in FIA
TRACE. This helps to track where this packet is in its path at each step.
Feature: FIA_TRACE
Output : <unknown>
Entry : 0x813a5554 - IPV4_INPUT_DST_LOOKUP_CONSUME
Feature: FIA_TRACE
Output : <unknown>
Entry : 0x813a5558 - IPV4_INPUT_FOR_US_MARTIAN
Feature: IPV4_INPUT_LOOKUP_PROCESS_EXT
Output : internal0/0/rp:0
Entry : Input - 0x812fcd58
New features Trace logs
Trace logs are now generated and saved in binary instead of text
(much faster, compressed message using errors codes instead of plain text)
Use more /compressed <filename> command to decode:
F340.04.19-4400-11# more /compressed bootflash:/tracelogs/btrace_rotate_sh_R0-0$

==> /bootflash//tracelogs/btrace_rotate_sh_R0-0.17730_10.20161025045439.bin.gzXXXXXa <==
2016/10/25 04:54:39.852 [btrace_rotate_sh] [17730]: UUID: 0, ra: 0, TID: 0 (note): Purged reflector_R0-
0.3413_54.20161025012333.bin.gz from /harddisk/tracelogs
2016/10/25 04:54:39.855 [btrace_rotate_sh] [17730]: UUID: 0, ra: 0, TID: 0 (note): Purged reflector_R0-
0.3413_55.20161025013833.bin.gz from /harddisk/tracelogs
Work in progress to make trace logs more simple

SPAN on IA
Where we are ?
SPAN on IA
We need to confirm that our test client is actually working

fine and sending all packets as expected to access layer.
Tools: SPAN on IA
{SPAN on IA} vs {SPAN on VSS}
FEX-101#configure terminal
FEX-101(config)#monitor session 1 source interface gi1/0/1
FEX-101(config)#monitor session 1 destination interface gi1/0/2
VSS PARENT VSS PARENT

VSS PARENT VSS PARENT
VS
FEX 101
FEX 101
Gig 101/0/1 Gig 101/0/2

Gig 101/0/1 Gig 101/0/2
Device Under SPAN Destination

Device Under SPAN Destination Test
Test
6880#configure terminal
6880(config)#monitor session 1 source interface gi101/1/0/1
6880(config)#monitor session 1 destination interface gi101/1/0/2
SPAN on IA
6880#configure terminal
1 6880(config)#service internal
2 6880#test platform software console fex <fex-id> enable timeout <minutes>
6880#attach fex 101

3 Attach FEX:101 ip:192.1.1.101 required to enable confg
Trying 192.1.1.101 ... Open changes directly via fex console
User Access Verification
Password: cisco
FEX-101>enable
Password: cisco
FEX-101#conf t
FEX-101(config)#monitor session 1 source interface gi1/0/1
FEX-101(config)#monitor session 1 destination interface gi1/0/2
IA SPAN Results
All RTP SEQ numbers comming

from the phone are OK,
no packet loss
CONCLUSION:
no issue found related to the source
Checking resources and ASIC drops on IA
3 Main verification points

VSS FEX Downlink where congestion/drops
3 can occur
2 FEX Uplink
1
Ingress interface
of FEX Client
6880VSS#show interface Gig 101/1/0/1
GigabitEthernet101/1/0/1 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is c472.9585.b903 (bia c472.9585.b903)
MTU 9216 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 232/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseT
input flow-control is off, output flow-control is unsupported
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00 Egress direction
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 912282000 bits/sec, 532883 packets/sec
5 minute output rate 4524545 bits/sec, 2232 packets/sec
951705714 packets input, 203665020228 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
Ingress 517057 packets output, 823232202 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
Ingress direction
interface 0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
of FEX Client 0 output buffer failures, 0 output buffers swapped out
Gig 101/1/0/1 <-> (on FEX) Gig 1/0/1
6880VSS#rem com fex 101 show controllers ethernet-controller GigabitEthernet 1/0/1

Transmit GigabitEthernet1/0/1 Receive
196616 Bytes 2814880 Bytes
1398 Unicast frames 1590 Unicast frames
221 Multicast frames 9307 Multicast frames
2 Broadcast frames 7213 Broadcast frames
0 Too old frames 103617 Unicast bytes
0 Deferred frames 1257312 Multicast bytes
0 MTU exceeded frames 1453951 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 0 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 11169 Minimum size frames
0 8 collision frames 690 65 to 127 byte frames
Ingress 0 13 collision frames
0 14 collision frames
0
0
Overrun frames
Pause frames
interface 0 15 collision frames
0 Excessive collisions 0 Symbol error frames
of FEX Client ......

0 Late collisions 0 Invalid frames, too large
6880#rem com fex 101 test etherchannel load-balance interface port-channel 1 ip 1.1.1.1 2.2.2.2
Would select Te1/0/2 of Po1
PO1 is the FEX uplink

PO101 is the same link from VSS perspective
2 6880#show fex 101

FEX: 101 Description: FEX0101 state: online
FEX version: 15.2(3m)E6
Extender Model: C6800IA-48FPD, Extender Serial: FCW1851A2P8
FCP ready: yes
Image Version Check: enforced
FEX Fabric Portchannel Ports: 2
Uplink Fabric port for control traffic: Te1/5/6
Fabric interface state:
Po101 - Interface Up.
Te1/5/6 - Interface Up. state: bound
Te2/5/6 - Interface Up. state: bound
6880#rem com fex 101 show platform port-asic stats enqueue tenGigabitEthernet 1/0/2
Interface Te1/0/1 TxQueue Enqueue Statistics

Queue 0
Weight 0 Frames 2273845694
Weight 1 Frames 69
Weight 2 Frames 4821
Queue 1
Weight 0 Frames 0 4 queues 3 thresholds based
Weight 1 Frames 0 on QOS capabilities of IA Switch
2 Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
FEX Weight 0 Frames 0
Weight 1 Frames 0
Uplink Weight 2 Frames 0
6880#rem com fex 101 show platform port-asic stats drop tenGigabitEthernet 1/0/2
Interface Te1/0/1 TxQueue Enqueue Statistics

Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
2 Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
FEX Weight 1 Frames 0
Uplink Weight 2 Frames 0
Non zero counters are
indications of drops.
Checking ingress interfaces on VSS
6880VSS#show interface port-channel 101
Port-channel101 is up, line protocol is up (connected)
VSS FEX Hardware is EtherChannel, address is fc5b.3998.6607 (bia fc5b.3998.6607)
MTU 1500 bytes, BW 20000000 Kbit/sec, DLY 10 usec,
Downlink reliability 255/255, txload 12/255, rxload 12/255

Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is unknown
3 input flow-control is on, output flow-control is off

Members in this channel: Te1/5/6 Te2/5/6
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 957306000 bits/sec, 534209 packets/sec
5 minute output rate 957305000 bits/sec, 534210 packets/sec
58462001262 packets input, 13095518844174 bytes, 0 no buffer
Received 146718 broadcasts (145036 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
58462021296 packets output, 13095534439831 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Checking ingress interfaces on VSS
6880VSS#show queueing interface TenGigabitEthernet 1/5/5
Packets dropped on Transmit:
VSS FEX BPDU packets: 0
Downlink queue thresh dropped [cos-map]

------------------------------------------------------------------
1 1 0 [0 1 2 3 4 5 6 7 ] Separate verification
3 Packets dropped on Receive:
for both VSS
BPDU packets: 0 physical
Egress interfaces
direction
queue thresh dropped [cos-map]
------------------------------------------------------------------
1 1 0 [0 1 2 3 4 5 6 7 ]
6880VSS#show queueing interface TenGigabitEthernet 2/5/5
Packets dropped on Transmit:

BPDU packets: 0

------------------------------------------------------------------
1 1 0 [0 1 2 3 4 5 6 7 ]
Packets dropped on Receive:

BPDU packets: 0

------------------------------------------------------------------
1 1 0 [0 1 2 3 4 5 6 7 ]
VSS forwarding of problem
traffic
Check VSS forwarding of client voice traffic
Is the traffic reaching the VSS

parent ?
What the VSS parent will do with
the traffic ?
Is the traffic forwarded via the
VSS directly connected device ?
Is the traffic forwarded via the VSL
link to peer chasis ?
Is the traffic dropped / CPU
FEX processed ?
ELAM is the answer !

ELAM
Embedded Logic Analyzer Module (ELAM) is an engineering tool that is used
to look inside Cisco ASICs.
ELAM is architecture-specific and therefore will have different capabilities and
different CLI syntax across different forwarding engines (FE).
Identifying the appropriate FE, creating triggers, and interpreting ELAM data
for complex flows requires full architectural and forwarding knowledge
ELAM is NOT a supported feature. It is a diagnostic tool

designed for internal use. Anything and everything about it
may change from version to version without notice
ELAM - Basics to Know Before Performing an ELAM
Data Bus (DBUS) and Result Bus (RBUS)

The DBUS contains several platform specific internal fields along with the header information from a frame required to
make the forwarding decision. We use the DBUS information to validate where the frame was received and basic
data about the frame.
The RBUS will contain information about the forwarding decision to help determine if the frame was altered and where
it was sent.
Local Target Logic (LTL)

The LTL is an index used to represent a port or group of ports. The source LTL index and the destination LTL index
tell us on which port the frame was received and where it was sent.
ELAM - Workflow
Identify the expected Configure an ELAM After the ELAM

ingress Forwarding trigger to capture Start the ELAM triggers, display and
Engine (FE) specific frame analyze the data
ELAM Syntax DFC4 Sup2T/PFC4/DFC4
eureka ASIC
Sup2T(config)# service internal

Sup2T# show platform capture elam asic eureka slot 5
Assigned asic_desc=eu50
Sup2T# show platform capture elam trigger master eu50 dbus dbi ingress <trigger>
Sup2T# show platform capture elam start
cap_commands: Default ELAM RBI PB1 added to list
Sup2T# show platform capture elam status
ID# Role ASIC Slot Inst Ver ELAM Status
----- ---- ------- ---- ---- --- --------- ------
eu50 M EUREKA 5 0 1.3 DBI_ING Capture Completed
eu50 s EUREKA 5 0 1.3 RBI_PB1 Capture Completed
ID# ELAM Trigger

----- --------- ----------
eu50 DBI_ING <trigger displayed here> Use the following command to map source
eu50 RBI_PB1 TRIG=1 index/destination index to physical ports on
Sup2T# show platform capture elam data
DBUS data: PFC4/DFC4:
(output omitted)
RBUS data: show platform hardware ltl index <index>
(output omitted)
ELAM - Triggers
The trigger must align to the frame type. The majority of all traffic will fall into one of three
categories:
IPv4 IPv6 Other
Common Triggers for each frame type are shown below

IPv4 IPv6 Other All Frame Types
SMAC SMAC DATA VLAN
DMAC DMAC SRC_INDEX
IP_SA IP6_SA DST_INDEX
IP_DA IP6_DA
IP_TTL IP6_TTL
IP_TOS IP6_CLASS
L3_PT (ICMP,IGMP,TCP,UDP) L3_PT (ICMP, IGMP, TCP,
TCP_SPORT, TCP_DPORT UDP)
UDP_DPORT, UDP_SPORT IP6_L4DATA
ICMP_TYPE
ELAM Example IPv4
10.1.117.231/25
Sup2T# show mod 5
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
5 5 Supervisor Engine 2T 10GE w/ CTS (Acti VS-SUP2T-10G SAL15056BKR
Gi5/3 (VLAN 10)
Traffic ingresses on module 5 which is the active supervisor and will

therefore be the ingress FE.
The traffic flow is from host 10.1.117.231 toward host 10.1.117.1 so the
trigger will be:
Gi5/2 (VLAN 20)
IPv4 if IP_SA=10.1.117.231 IP_DA=10.1.117.1
10.1.117.1/25
ELAM IPv4 Example
Sup2T(config)# service internal
Sup2T# show platform capture elam asic eureka slot 5 10.1.117.231/25
Sup2T# show platform capture elam trigger master eu50 dbus dbi ingress
ipv4 if ip_sa=10.1.117.231 ip_da=10.1.117.1
Sup2T# show platform capture elam start

cap_commands: Default ELAM RBI PB1 added to list Gi5/3 (VLAN 10)
Sup2T# show platform capture elam status
----- ---- ------- ---- ---- --- --------- ------
ID# ELAM Trigger

----- --------- ---------- Gi5/2 (VLAN 20)
eu50 DBI_ING FORMAT=IP L3_PROTOCOL=IPV4 IP_SA=10.1.117.231
IP_DA=10.1.117.1
eu50 RBI_PB1 TRIG=1
10.1.117.1/25
ELAM IPv4 Example
Sup2T# show platform capture elam data 10.1.117.231/25
Sup2T# show platform hardware ltl index 0x102
(some output omitted)
DBUS data: LTL index 0x102 contain ports :
VLAN ............................ [12] = 10 =========================================
SRC_INDEX ....................... [19] = 0x102 Gi5/3 <----- packet received on Gi5/3
L3_PROTOCOL ..................... [4] = 0 [IPV4]
L3_PT ........................... [8] = 1 [ICMP] Gi5/3 (VLAN 10)
IP_TTL .......................... [8] = 255
Sup2T# show platform hardware ltl index 0x101
IP_SA ........................... = 10.1.117.231
IP_DA ........................... = 10.1.117.1 LTL index 0x101 contain ports :
=========================================
RBUS data: Gi5/2 <----- packet sent out Gi5/2
FLOOD ........................... [1] = 0
DEST_INDEX ...................... [19] = 0x101
VLAN ............................ [12] = 20 Packet received on VLAN 10 with a TTL of Gi5/2
255 (VLAN 20)
IP_TTL .......................... [8] = 254 and routed out VLAN 20 with a TTL of 254
REWRITE_INFO
i0 - replace bytes from ofs 0 to ofs 11 with seq '00 00 0C 07 AC CA B4
14 89 61 37 80'.
Rewrite information on the packet contains destination MAC 10.1.117.1/25

(0000.0c07.acca) and source MAC (b414.8961.3780)
ELAM VSS Example In case of first ELAM capture on the first arrival point
always select <ingress>
10.10.10.10
6880#show platform capture elam asic eureka slot 5
6880#show platform capture elam trigger master eu50 dbus dbi ingress ipv4 if
ip_sa=20.20.20.20 ip_da=10.10.10.10 Vlan 10
Te 1/5/2
6880VSS#show platform capture elam start
cap_commands: Default ELAM RBI PB1 added to list
Te 1/5/4
6880#show platform capture elam status
VSS
ELAM Mode: local
Te 2/5/4
1
----- ---- ------- ---- ---- --- --------- ------ Te 1/5/6 Te 2/5/6

eu50 s EUREKA 5 0 1.3 RBI_PB1 Capture Completed FEX
ID# ELAM Trigger Vlan 20 Gig 101/1/0/3

----- --------- ----------
20.20.20.20
eu50 DBI_ING FORMAT=IP L3_PROTOCOL=IPV4 IP_SA=20.20.20.20 IP_DA=10.10.10.10
eu50 RBI_PB1 TRIG=1
ELAM VSS Example
6880#show platform capture elam data
DBUS data:
10.10.10.10
VLAN ............................ [12] = 20 [HW BD: 20]
SRC_INDEX ....................... [19] = 0x2032
[Po101[Te1/5/6,Te2/5/6],Gi101/1/0/3]
LEN ............................. [16] = 214 Vlan 10
Te 1/5/2
FORMAT .......................... [2] = 0 [IP]
PACKET_TYPE ..................... [3] = 0 [ETHERNET]
Te 1/5/4
L3_PROTOCOL ..................... [4] = 0 [IPV4]
VSS
L3_PT ........................... [8] = 1 [ICMP]
DMAC ............................ = 84b8.02e1.e5c0
Te 2/5/4
1
SMAC ............................ = 0000.0060.dd19 Te 1/5/6 Te 2/5/6
IP_TTL .......................... [8] = 64

IP_SA ........................... = 20.20.20.20 FEX
IP_DA ........................... = 10.10.10.10
Vlan 20 Gig 101/1/0/3
RBUS data:
20.20.20.20
SEQ_NUM ......................... [5] = 0xE
DEST_INDEX ...................... [19] = 0x101 [Te1/5/2,Po2[Te2/5/4]]
VLAN ............................ [12] = 1012
IP_TTL .......................... [8] = 63
IP_CSUM_VALID ................... [1] = 1
Egress interface + VSL link
REWRITE_INFO
i0 - replace bytes from ofs 0 to ofs 11 with seq '54 A2 74 20 E5 92 84 B8 02 E1
E5 C0'.
FCS2 ............................ [8] = 0x61
ELAM VSS Standby Example Need to login into standby
chassis
6880VSS#remote login switch 1 modu 5
6880VSS-sdby#show platform capture elam asic eureka slot 5
6880VSS-sdby#show platform capture elam trigger master eu50 dbus dbi egress ipv4 When capturing packets
if ip_sa=20.20.20.20 ip_da=10.10.10.10 after passing via VSL link,
egress direction is
There are two different Packet Buffers (PB) in which the RBUS data can reside. required.
Determination of the correct PB instance is dependent upon the exact module type
10.10.10.10
and ingress port. Typically, it is recommended that you configure PB1, and if the
RBUS does not trigger, then repeat the configuration with PB2. If no RBUS trigger is
provided, Cisco IOS automatically creates a trigger on PB1 Vlan 10
Te 1/5/2
6880VSS-sdby#sh pla cap elam tri s eu50 r r pb2

6880VSS-sdby#show platform capture elam start Te 1/5/4
6880VSS-sdby#show platform capture elam status 2VSS
Te 2/5/4
ELAM Mode: local
ID# Role ASIC Slot Inst Ver ELAM Status Te 1/5/6 Te 2/5/6
----- ---- ------- ---- ---- --- --------- ------
eu50 M EUREKA 5 0 1.3 DBI_EGR Capture Completed FEX
Vlan 20 Gig 101/1/0/3
ID# ELAM Trigger
----- --------- ---------- 20.20.20.20
eu50 DBI_EGR FORMAT=IP L3_PROTOCOL=IPV4 IP_SA=20.20.20.20 IP_DA=10.10.10.10
eu50 RBI_PB2 TRIG=1
ELAM VSS Standby Example
6880VSS-sdby#show platform capture elam data
DBUS data: 10.10.10.10
SEQ_NUM ......................... [5] = 0x8
VLAN ............................ [12] = 1012
SRC_FLOOD ....................... [1] = 0 Vlan 10
SRC_INDEX ....................... [19] = 0x2032 Te 1/5/2
[Po101[Te1/5/6,Te2/5/6],Gi101/1/0/3]
LEN ............................. [16] = 214 Te 1/5/4
FORMAT .......................... [2] = 0 [IP]
L3_PROTOCOL ..................... [4] = 0 [IPV4] 2
VSS
Te 2/5/4
DMAC ............................ = 54a2.7420.e592

Te 1/5/6 Te 2/5/6
SMAC ............................ = 84b8.02e1.e5c0
IP_TTL .......................... [8] = 63 FEX
IP_CHKSUM ....................... [16] = 0x3E46
IP_SA ........................... = 20.20.20.20
IP_DA ........................... = 10.10.10.10 Vlan 20 Gig 101/1/0/3
20.20.20.20
RBUS data:
SEQ_NUM ......................... [5] = 0x8
DEST_INDEX ...................... [19] = 0x1101 [Te1/5/2]
VLAN ............................ [12] = 1012
RBH ............................. [3] = b000 Only Egress interface
REWRITE_INFO
i0 - no rewrite.
ELAM problematic traffic
66880VSS#show platform capture elam data
DBUS data:
SEQ_NUM ......................... [5] = 0x15
VLAN ............................ [12] = 30 [HW BD: 30]
SRC_INDEX ....................... [19] = 0x2032
[Po101[Te1/5/6,Te2/5/6],Gi101/1/0/3] Conclusion:
LEN ............................. [16] = 196
FORMAT .......................... [2] = 0 [IP]
Voice trafic from the
L3_PT ........................... [8] = 17 [UDP] problematic transmision is
DMAC ............................ = 84b8.02e1.e5c0
SMAC ............................ = 0000.0060.dd19 software forwarded
IP_SA ........................... = 30.30.30.30
IP_DA ........................... = 40.40.40.40
RBUS data:
SEQ_NUM ......................... [5] = 0x15 What does the index
FLOOD ........................... [1] = 0 380 mean ?
DEST_INDEX ...................... [19] = 0x380
6880VSS#show platfor software ltl well-known-index | i 380

LTL_EXPORT_PRIMARY_ROUTER_PORT 0 0x380 Index of CPU RP
LTL_EXPORT_PRIMARY_BRIDGE_PORT 1 0x380
LTL_EXPORT_PRIMARY_SWITCH_PORT 2 0x380
CPU Packet capture 6880VSS#show netdr captured-packets
l2idb Gi101/1/0/3, l3idb Vl30, routine
| i timestamp
inband_process_rx_packet, timestamp 08:25:47.010
Confirmation
l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
6880VSS#debug netdr capture rx l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
6880VSS#show netdr captured-packets
A total of 4096 packets have been captured l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
The capture buffer wrapped 0 times l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
Total capture capacity: 4096 packets l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
------- dump of incoming inband packet ------- l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, l2idb
timestamp 08:25:47.010
Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
dbus info: src_vlan 0x1E(30), src_indx 0x2032(8242), len 0xD0(208)
bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x1380(4992),
l2idbCoS 5
Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
cap1 0, cap2 0 l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
05820800 001E0000 20320000 D0080000 1E000404 20000400 00000010 l2idb Gi101/1/0/3, l3idb Vl30,
1380B989 routine inband_process_rx_packet, timestamp 08:25:47.010
destmac 84.B8.02.E1.E5.C0, srcmac 00.00.00.60.DD.19, shim ethertype CCF0
earl 8 shim header IS present: l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
version 0, control 64(0x40), lif 30(0x1E), mark_enable 1, l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
feature_index 0, group_id 0(0x0), acos 46(0x2E), l2idb Gi101/1/0/3, l3idb Vl30, routine inband_process_rx_packet, timestamp 08:25:47.010
ttl 14, dti 4, dti_value 508(0x1FC)
100003C8 00BB8080 01FC
ethertype 0800 Not just a single packet but
protocol ip: version 0x04, hlen 0x05, tos 0xB8, totlen 178, identifier 0
df 0, mf 0, fo 0, ttl 64, src 30.30.30.30, dst 40.40.40.40 entire stream is CPU
udp src 800, dst 1678 len 158 checksum 0xA469 Source and Destination processed
IPs match our voice
- traffic
Feature Manager
Feature Manager
- Features are enabled by the ASICs advanced logic
- Feature Manager handles configuration of features, can efficiently allocate available hardware resources to them
- Keeps track of all features that have impact on hardware forwarding
- Resolve conflicts
- Share hardware resources among multiple interfaces
- Display information about programming

Check features on 6800 SVI
ISR4451-1 ISR4451-2
6880VSS#show fm fie interface vlan 30
HQ
6880X-1 6880X-2
6880VSS#show fm interface vlan 30
NMS
System
3850-1 3850-2 3850-3

IP Phone
FEX102
FEX101
3850-4
SVI features list
6880VSS# show fm fie interface vlan 30
No. of features admitted = 2
<...>
>>> Label state for protocol: FM_PROTOCOL_IP <<<

Number of passes: 1
<...>
Information for bank: 0

Adj mapped: 1 result type: FIE_RESULT_TYPE_ACL precedence: 3
Feature list in the bank:
IPv4 Regular L2 NDE Mon1 Ingress Feature IPv4 Regular MCAST NDE Mon1 Ingress Feature
<...>
PBR
Information for bank: 1
Adj mapped: 1 result type: FIE_RESULT_TYPE_ACL precedence: 2
Feature list in the bank:
IPv4 Regular L3 NDE Mon1 Ingress Feature IP_ACCESS_INGRESS PBR - Policy Based Routing
SVI feature manager All features
6880VSS# show fm interface vlan 30 are correctly
Interface: Vlan30 IP is enabled; admin_state is up programmed
hw_state[INGRESS] = not reduced, hw_state[EGRESS] = not reduced
<...>
inbound label: 3
Feature PBR - Policy Based Routing:
-----------------------------------------------------------------------------
FM_FEATURE_PBR i/f: Vl30 rmap: TEST
Punt to CPU
=============================================================================
----------------------------------------------------
Seq. No: 10 Seq. Result : FM_RESULT_BRIDGE
----------------------------------------------------
1 V 0.0.0.0 30.30.30.0 0 0 0 ----- 0 -----
M 0.0.0.0 255.255.255.0 0 0 0 00000 0 0
PERMIT_RESULT
Route-map misconfiguration
6880VSS# show running-config interface vlan 30
interface Vlan30
ip address 30.30.30.1 255.255.255.0
ip access-group 103 in
ip flow monitor netflow-original input
ip flow monitor netflow-original output
ip policy route-map TEST
Set clause is
route-map TEST, permit, sequence 10
Match clauses: missing
ip address (access-lists): 130
Set clauses:
Policy routing matches: 5735647324 packets, 1101244286208 bytes
Set the next
6880VSS#conf term hop
route-map TEST permit 10
set ip next-hop 10.10.10.10
SVI feature manager, after fix
6880VSS# show fm interface vlan 30
Interface: Vlan20 IP is enabled; admin_state is up

hw_state[INGRESS] = not reduced, hw_state[EGRESS] = not reduced
<...>
inbound label: 51
Feature PBR - Policy Based Routing:
FM_FEATURE_PBR i/f: Vl20 rmap: PBR Hardware

-----------------------------------------------------------------------------
redirect
=============================================================================
----------------------------------------------------
Seq. No: 10 Seq. Result : FM_RESULT_ADJREDIRECT
----------------------------------------------------
1 V 0.0.0.0 30.30.30.0 0 0 0 ----- 0 -----
M 0.0.0.0 255.255.255.0 0 0 0 00000 0 0
PERMIT_RESULT Adjacency: 0x20346354
Problem #2.
Monitoring systems have issues with
statistics collection from 3850 switches.
Problem #2
Problem definition:
Centralized monitoring system has problems with statistic collection
from 3850 switches located in HQ
Impact:
Low
Scope :
Only NMS systems seems to be effected
Login or ping to 3850 switch is not working:
6880VSS# ping 10.10.55.8 repeat 10
escape sequence to abort.

Sending 10, 100-byte ICMP Echos to 10.10.55.8, timeout
is 2 seconds:
..!...!...
Success rate is 20 percent (2/10), round-trip
min/avg/max = 1/4/24 ms
High CPU
3850-1# show processes cpu sorted | e 0.00
Core 0: CPU utilization for five seconds: 43%; one minute: 50%; five minutes: 49%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12622 3158805 60970503 192 24.35 24.71 24.51 34818 iosd
5701 1741495 13224121 78 24.20 23.82 23.68 1088 fed
6239 676120 15556850 43 2.13 2.11 2.08 0 pdsd
<...>
3850-1# show processes cpu detailed process iosd sort | e 0.00

PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
12622 L 3272305 6101876 194 25.08 24.69 24.57 34818 iosd
12622 L 3 12622 588495 4883600 0 22.66 22.67 22.77 34818 iosd
12622 L 2 14137 2113440 7457443 0 2.38 1.93 1.73 34816 iosd.fastpath
12622 L 0 14138 567040 4574544 0 0.05 0.08 0.06 34818 CMI Thread
162 I 1873910 58609 0 91.88 91.66 91.99 0 NGWC L2M
Find CPU queue
show platform punt statistics port-asic <port-asic> cpuq <queue> direction <rx|tx>
3850-1# show platform punt statistics port-asic 0 cpuq -1 direction rx

snip>
RX (ASIC2CPU) Stats (asic 0 qn 16 lqn 16):
RXQ 16: CPU_Q_PROTO_SNOOPING
----------------------------------------
Packets received from ASIC : 79099152 -1 argument lists
Send to IOSd total attempts
Send to IOSd failed count
: 79099152
: 1240331 all of the queues
RX suspend count : 1240331
RX unsuspend count : 1240330
RX unsuspend send count : 1240330
RX unsuspend send failed count : 0
RX dropped count : 0
RX conversion failure dropped : 0
RX pkt_hdr allocation failure : 0
RX INTACK count : 0
RX packets dq'd after intack : 0
Active RxQ event : 9906280
RX spurious interrupt : 0
<snip>
Catalyst 3650/3850: FED tracing
Forwarding Engine Driver (FED) is the heart of Cisco Unified Access
switching platforms (like Catalyst 3650/3850), and is responsible for
hardware programming and forwarding.
FED Tracing
IOSd
FED
Kernel
data plane traffic mgmt / control plane traffic

Catalyst 3650/3850: FED tracing
FED tracing is a mechanism to capture packets sent by the FED towards

the IOSd.
3850-1# set trace control fed-punject-detail [ enable | disable | clear ]
3850-1# set trace fed-punject-detail direction [ rx | tx ] <filters>
3850-1# set trace control fed-punject-detail buffer-size <bytes>

Catalyst 3650/3850: FED tracing, Sample
3850-1# set trace fed-punject-detail direction rx filter_add cpu-queue 16 16
3850-1# set trace fed-punject-detail direction rx match_all
3850-1# set trace fed-punject-detail direction rx filter_enable
3850-1# set trace control fed-punject-detail buffer-size 32000
3850-1# set trace control fed-punject-detail enable
3850-1# show mgmt-infra trace messages fed-punject-detail
[10/25/14 11:47:37.809 CST 1619f0ac 5694]
33 33 00 00 00 16 d4 c9 ef f1 d4 65 81 00 03 44
86 dd 60 00 00 00 00 24 00 01 fe 80 00 00 00 00
00 00 c4 ed d7 d4 e8 7c 36 3c ff 02 00 00 00 00 Source and Destination MAC

00 00 00 00 00 00 00 00 00 16 3a 00 05 02 00 00 Address, and Ethertype
Root cause
IPv6 is not enabled in HQ.
Solutions:
1. Find this MAC by checking MAC address table and block it.
2. Enable MLD snooping. When MLD snooping is enabled, a per-VLAN IPv6 multicast
address table is constructed in software and hardware. The switch then performs IPv6
multicast-address based bridging in hardware, avoiding to be processed by software.
3850-1#config terminal
3850-1(config)# ipv6 mld snooping
3850-1(config)# end
Normal CPU utilization
3850-1# sh processes cpu sorted | e 0.00
6106 1547872 35097160 166 1.59 1.54 1.51 1088 fed
8979 3282850 34676914 94 0.60 0.74 0.72 0 iosd
6108 1792550 2978488 601 0.36 0.37 0.37 0 stack-mgr
<...>
3850-1# show processes cpu detailed process iosd sort | e 0.00

PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
8979 L 3286510 3471163 94 0.78 0.74 0.73 0 iosd
8979 L 0 8979 1771980 2895816 0 0.45 0.36 0.36 0 iosd
8979 L 0 10056 1494750 5395638 0 0.32 0.36 0.36 0 iosd.fastpath
<...>
Login or ping to 3850 switch is working now:
6880VSS# ping 10.10.55.8 repeat 10
Type escape sequence to abort.

Sending 10, 100-byte ICMP Echos to 10.10.55.8, timeout
is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip
min/avg/max = 1/4/24 ms
Problem #3.
Intermediate IPv6 connectivity problem
in office A.
Problem #3
Problem definition:
Since 3 days Developers connected in Office A using IPv6 are having
intermediate issues with connectivity and with ability to access public
IPv6 resources
Impact:
High software developers are unable to perform basic operations
Scope :
Problems are related to Office A, as only there IPv6 protocol is
enabled.
Ipv6 connectivity problem
Working fine
Not working
Verify IPv6 forwarding path
Investigate IPv6 forwarding path from switch/router and

working and non-working stations.
Tools: ping, traceroute

IPv6 forwarding path
3650-1#ping ipv6 2001:420:1101:1::a
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:420:1101:1::a, timeout is 2 seconds
!!!!! ipv6.cisco.com
2001:420:1201:2::a
Office A
Success rate is 100 percent (5/5), round-trip min/avg/max = 25/26/30 ms
SVI 1
2001:dead:beef::/64 Default Gatway
2001:dead:beef:2::2
2001:dead:beef:2::/64
WAN
ISR420-1#ping
ipv62001:420:1101:1::a
ISR4420-1 Type escape sequence to abort.
3650-1
Sending 5, 100-byte ICMP Echos to 2001:420:1101:1::a,
timeout is 2 seconds
C:\Users\admin>ping ipv6.cisco.com !!!!!
Pinging ipv6.cisco.com [2001:420:1101:1::a] with 32 Success rate is 100 percent (5/5), round-trip min/avg/max
bytes of data: Request timed out. = 25/26/31 ms
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 2001:420:1101:1::a

Packets: Sent = 4, Received = 0, Lost = 4 (100%
loss),
End Station configuration verification
End stations are using

dual stack ipv4 & ipv6.
Ipv6 address assigment
is based on SLAC
Router Discovery protocol
Discover default/first hop routers
Discover on-link prefixes
A R
ICMP Type = 133 (Router Solicitation)

Source = Host link-local address
RS
Destination = ALL-ROUTERS multicast address (FF02::2)
Query = please send RA
RA ICMP Type = 134 (Router Advertisement)

Source = Router Link-Local address LLR
Destination = All-nodes multicast address (FF02::1)
Routing Data = router lifetime, preference=medium,
Table RIB ::0/0 LLR Option = Prefix X,Y,Z, lifetime
Use R as default gateway
The LINK-LOCAL address is the router identity

End station verification
Where the ipv6 prefix

bad::/64
is coming from ?
Use packet capture to find rouge RA
Capture on end station
End station is receiving

expected RA with prefix
2001:dead:beef::/64
Capture on end station
However it is also
receiving
unexpected RA
with
prefix
bad::/64
and with route
information for
::/0
2002::/3
FC00::/7
Router Theft: role (and session hijacking!)
Attacker tricks victim into accepting itself as default router
Based on rogue Router Advertisements
Many variants: preference, timing, final RA, etc.
R X
A C
X
RIB
RA Source = LLR, preference=medium
::0/0 LLR
Session via R
RA Source = LLC, Destination=ALL-NODES,
preference=high
::0/0 LLC
Session
Most frequent issue seen via link
on the C
DoS attack: denial of address resolution
(one packet)
Attacker responds to all Resolution Requests
A B
X
MAC B
ICMP type = 135 (Neighbor Solicitation) NS-lookup

Dst = Solicited-node multicast address of B
target = B
Query = what is Bs Link-Layer Address?
Neighbor B - INCMPL Src = B

cache NA Dst = A Src = B
Options = TLLA (MACFAKE)
Dst = A
Options = TLLA (MACB)
B MAC FAKE REACH
MACFAKE
Solution ? IPv6 First Hop Security
FHS features can be classified in three feature categories as follows:
Core:
RA Guard blocks unauthorized Router Advertisements (RAs)
DHCP Guard blocks unauthorized DHCP servers
IPv6 Snooping analyzes control/data switch traffic, detects IP address, and stores/updates them in a binding table.
Advanced:
Source/Prefix Guard validates source address or a prefix of IPv6 traffic sourced from the link
Destination Guard validates the destination address of IPv6 traffic reaching the link
Performance and scalability:

RA Throttler facilitates scale by converting multicast RA traffic into unicast
ND Multicast Suppress controls Neighbor Discovery (ND) traffic necessary for proper link operations and improves
performance
General principles on FH command interface
Each FH feature provides a configuration mode to create and populate policies (+ one
implicit default policy)
ipv6 nd raguard policy host
device-role host
Each FH feature provides commands to attach policies to targets: box, vlan, port
vlan configuration 100
ipv6 nd raguard attach-policy host
ipv6 snooping
interface e 0/0
ipv6 nd raguard attach-policy router
Packets are processed by the lowest-level matching policy for each feature
Packets received on e0/0 are processed by policy ra-guard router AND policy snooping default
Packets received on any other port of vlan 100 are processed by policy ra-guard host AND policy
snooping default
How to block rouge RA
Port ACL: blocks all ICMPv6 RA from hosts
interface FastEthernet0/2
ipv6 traffic-filter ACCESS_PORT in
RA
access-group mode prefer port
RA-guard lite: pre-programmed ACL

Authorized Port
interface FastEthernet0/2 RA
ipv6 nd raguard
access-group mode prefer port
RA-guard: deep RA packet inspection
RA
ipv6 nd raguard policy HOST Port Not
device-role host Authorized RA
ipv6 nd raguard policy ROUTER - hop-limit

device-role router - M & O flag
vlan configuration 100 - Router preference RA
ipv6 nd raguard attach-policy HOST - Source

interface FastEthernet0/0 - Prefix list
ipv6 nd raguard attach-policy ROUTER - CGA credentials
After implementing RA Guard
ipv6 nd raguard policy HOST
device-role host
ipv6 nd raguard policy ROUTER
device-role router
vlan configuration 1
ipv6 nd raguard attach-policy HOST
!
3560-1#show ipv6 nd raguard policy HOST
Policy HOST configuration:
device-role host
Policy HOST is applied on the following targets:
Target Type Policy Feature Target range
vlan 1 VLAN HOST RA guard vlan all
Logging RA Guard activity
!
If we want to know which
port is sending the rouge
ipv6 snooping logging packet drop
RA we can follow the src
!
mac or enable additional
logging
3560-1#
*Oct 31 14:35:08.077: %SISF-4-PAK_DROP: Message dropped A=FE80::250:56FF:FE84:C17A G=- V=1 I=Gi1/0/4 P=NDP::RA
Reason=Message unauthorized on port
Reason=Message unauthorized
The toolbox
Vulnerability Attack tool Mitigation Where Security level Deployability
thc, si6,..
Router Role theft fake_router6 Increase legal router preference Router Weak Low
flood_router6 Manual default gateway configuration Host Very Strong Medium-Low
redir6 SeND Router Authorization Host Very Strong Low
Host isolation (PVLAN) Switch Very Strong Medium
Port Access Lists (PACL) Switch Medium Medium-High
RA guard Switch Medium-Strong Medium-High
Router Identity theft/ Address Theft parasite6 Static ND cache entry Host Very Strong Low
SeND CGA Host Very Strong Low
Binding Guard (IPv6 snooping) Switch Strong High
DoS: denial of address initialization dos-new-IPv6 Binding Guard (IPv6 snooping) Switch Strong High
SeND CGA Very Strong Medium
DoS: denial of address assignment denial6 DHCP guard Switch Strong High
fake_advertiser6 DHCP authentication Host Strong Low
DoS: denial of address configuration thcping6 RA guard Switch Medium-Strong Medium-High
dos-new-IPv6 PACL Switch Medium Medium
DoS: denial of Address Resolution (1pkt) frag6 Binding Guard Switch Medium-Strong Medium-High
DoS: denial of Address Resolution (flood) scan6 Destination Guard Router Strong Medium
dos-new-IPv6 RACL Router Medium Medium-Low
DoS: denial of Link Operations (flood) dos-new-IPv6 ND control Router Weak Low
flood_advertise6 Binding Guard control Switch Very Strong Very High
Misdirecting responses syn6_flood Source Guard, Prefix Guard Switch Very Strong Very High
ACL Router Very Strong Low
uRPF Router Weak Low
IPv6 First Hop Security Platform Support
Wireless
LAN
Catalyst 6500 Catalyst Catalyst ASR1000 Catalyst Controller Nexus
Feature/Platform Series 4500 Series 2K/3K Series Router
7600 Router
3850 (Flex 7500, 3k/5k/6k/7k
5508, 2500,
WISM-2)
RA Guard 15.0(1)SY 15.1(2)SG 15.0.(2)SE 15.2(4)S 15.0(1)EX 7.2 NX-OS 7.2
NX-OS 7.2
IPv6 Snooping 15.0(1)SY1 15.1(2)SG 15.0.(2)SE XE 3.9.0S 15.2(4)S 15.0(1)EX 7.2
NX-OS 7.2
DHCPv6 Guard 15.2(1)SY 15.1(2)SG 15.0.(2)SE 15.2(4)S 15.0(1)EX 7.2
Source/Prefix 15.2(1)SY 15.2(1)E 15.0.(2)SE2 XE 3.9.0S 15.3(1)S 7.2

NX-OS 7.2
Guard
NX-OS 7.2
Destination Guard 15.2(1)SY 15.1(2)SG 15.2(1)E XE 3.9.0S 15.2(4)S
RA Throttler 15.2(1)SY 15.2(1)E 15.2(1)E 15.0(1)EX 7.2
ND Multicast 15.2(1)SY 15.1(2)SG 15.2(1)E XE 3.9.0S 15.0(1)EX 7.2

Suppress
Note 1: IPv6 Snooping support in 15.0(1)SY does not extend to DHCP or data packets; only ND packets are snooped
Note 2: Only IPv6 Source Guard is supported in 15.0(2)SE; no support for Prefix Guard in that release
Problem #4.
Wireless deployment in office B.
Problem #4
Problem definition:
End customer is planning to deploy converged access in office B,
however the wireless functionality is not available
Impact:
High not able to proceed with POC
Scope :
Limited to Office B
Wireless configuration is not accepted
Unable to apply
wireless configuration
Seems that switch does

not accept any basic
wireless configuration
Official requirements
4507-1#show mod
Chassis Type : WS-C4507R+E
Power consumed by backplane : 40 Watts

---+-----+--------------------------------------+------------------+-----------
1 24 1000BaseX (SFP) WS-X4624-SFP-E JAE1221IT3Q
3 8 Sup 8-E 10GE (SFP+), 1000BaseX (SFP) WS-X45-SUP8-E CAT1720L1UU
M MAC addresses Hw Fw Sw Status

--+--------------------------------+---+------------+----------------+---------
1 0021.1c7c.07d7 to 0021.1c7c.07ee 1.0 Ok
3 503d.e583.3740 to 503d.e583.3747 0.4 15.1(1r)SG5 03.07.02.E Ok
Sanity check - CPU
4507-1#show processes cpu sorted
5594 2455962 20531785 221 1.68 0.20 0.05 0 iosd
4890 30689 123327 248 0.02 0.01 0.01 0 sh
5526 88226 624348 141 0.02 0.01 0.01 0 mem_mgmt
5527 51025 2023720 25 0.02 0.01 0.01 0 mgmte_tap
5537 193236 655386 294 0.02 0.02 0.02 0 cpumemd
1 1539 1131 1360 0.00 0.00 0.00 0 init
2 9 333 27 0.00 0.00 0.00 0 kthreadd
3 105 21600 4 0.00 0.00 0.00 0 migration/0
4 0 3 2 0.00 0.00 0.00 0 sirq-high/0
5 1638 217120 7 0.00 0.00 0.00 0 sirq-timer/0
6 0 5 0 0.00 0.00 0.00 0 sirq-net-tx/0
Sanity check- memory, diagnostic tests
4507-1#show diagnostic result module all
4507-1#show processes memory sorted
System memory : 3870592K total, 892521K Current bootup
used, 2978071K free,diagnostic level: reserved
323712K kernel minimal
Lowest(b) : 2575726796
module 1: SerialNo : JAE1221IT3Q
PID Text Data Stack Heap RSS Total Process
5594 152352 942972 100 264 1252920 1349052
Overall Diagnostic Resultiosd
for module 1 : PASS
5532 4528 93756 92 5320 Diagnostic level at card ffm
42852 229960 bootup: minimal
5560 112 69736 88 3468 24808 110932 cli_agent
5529 1288 116520 88 1348 24264
Test results:161664
(. = Pass, licensed
F = Fail, U = Untested)
5534 1308 170116 88 4652 19772 220136 eicored
5523 760 42348 88 2812 1) linecard-online-diag
13044 80368 ------------> .
osinfo-provider
5562 764 108568 88 1488 2) stub-rx-errors
12000 152104 ------------------>
snmp_subagent .
5592 284 62308 88 428 9688 106236 licenseagentd
5531 304 121056 88 556 9540
module 3:
161180 installer
SerialNo : CAT1720L1UU
5595 212 51676 88 872 9532 90476 ha_mgr
5512 984 50624 88 720 9200 93796
Overall Diagnostic Resultobfld
for module 3 : PASS
Diagnostic level at card bootup: minimal
Test results: (. = Pass, F = Fail, U = Untested)
1) supervisor-bootup ---------------> .
2) linecard-online-diag ------------> .
3) stub-rx-errors ------------------> .
4) supervisor-rx-errors ------------> .
CLI Analyzer - Tool that
uses collective knowledge
of Cisco TAC
CLI Analyzer
What is it?
The Cisco CLI Analyzer is TAC powered SSH client with Advanced tools supporting ASA, IOS, IOS-XE, IOS-XR and
NX-OS software.
Who can use it?

Anyone with a Cisco.com account and valid support contract.
How does it work?

Diagnostic information is captured via the Command Line Interface (CLI) and sent to Cisco for processing. Findings and
recommendations are sent back to the CLI Analyzer in a results window.
What is the value?

CLI Analyzer reduces customer effort and creates a simplified support experience through thoughtful
integrations/workflows. This translates into saved time and money.
Where can I get it?

http://cway.cisco.com/go/sa
Support Experience Today
Engineer Customer Tools Troubleshoot
Open SSH client > SSH show tech
Software Bugs
HTTP search,upload, etc.
Docs TAC
Errors
Support Experience Simplified
Engineer Integrated Business
Workflows Outcomes
Cisco
CLI Analyzer
Open CLI Analyzer

Save Time
Bug Analysis Case

Tools Automation
Save Money
TAC Tools
Software
Integrated TAC
Knowledge
Main Features
Telnet/SSH client
Real-time Contextual Help and Highlighting (CHH)
Real-time Tools:
System Diagnostics: ASA, IOS (including XE and XR), NX-OS
ASA: Firewall Top Talkers, Traceback Analyzer, Packet Tracer
IOS-XR: BGP Top Talkers, L2VPN Top Talkers, LPTS Top Talkers
File Analysis: Possibility to run system diagnostics and CHH offline on copy-pasted
show commands or file with show commands when box is not reachable
Telnet/SHH Client
Download
www.cisco.com -> Support -> Tools
Telnet/SHH Client
Quick connect
Recent activity
Search devices sort
Store accessed
devices
Telnet/SHH Client Settings
User settings
Logs
Proxy
Master password
Theme
Telnet/SHH Client Auto-check system
Telnet/SHH Client inside the client window
Search for
Case insensitive
Use regular expression in
search
Disable highlight
regex example
\s10.\d.\d\d\d.\d
Searches for IPv4 addresses
of the form 10.x.xxx.x
Logging
Logging session to file
Start/stop provide file
name
Telnet/SHH Client
Contextual Help and
Highlighting (CHH)
Contextual Help &
Highlighting
Highlighting
Show version
Show running-config
Show log
Any show command!
Regular expressions make

sure that you can shorten the
show command
How does it work?
TAC Engineers constantly creating new CHH rules.

No need to upgrade CLI Analyzer client to benefit from new rules as
they are downloaded automatically when starting CLI Analyzer.
Tools require CCO username/password to access rules (and run
tools).
Replacing Output Interpreter
Output Interpreter was not
updated anymore
Limited supported commands
Limited OS and platform support
No need to upload commands
to cisco.com anymore
commands highlighted inside
CLI Analyzer.
CLI Analyzer Tools
Tools
First tool is the System Diagnostics

Available across platforms (ASA, IOS, IOS-XE, IOS-XR)
Other tools are platform specific
Specific commands
Correlate information from multiple commands or same command taken
multiple times
System Diagnostics
Reports known problems found
with the devices configuration
and operating state
Input is show tech-support
enable mode needed!

System Diagnostics collect show tech
File Analysis
File Analysis
Requires show version !
Upload file with

commands. Or paste
commands.
Returns list of alerts found

in the commands.
File Analysis
Many show commands

supported
List is growing with new

rules being added
File Analysis
results
List of notifications and
alerts, marked with risk
level
Sorted into categories
HTTP Links to
corresponding guides and
documents
Wireless issue:
Software mode
CLI Analyzer and wireless issue
Software mode change
4507-1#software expand file bootflash:cat4500es8-universalk9.SPA.03.07.02.E.152-3.E2.bin
Preparing install operation ...
[3]: Starting install operation
[3]: Expanding bundle bootflash:cat4500es8-universalk9.SPA.03.07.02.E.152-3.E2.bin
[3]: Copying package files
[3]: Package files copied
[3]: Finished expanding bundle bootflash:cat4500es8-universalk9.SPA.03.07.02.E.152-3.E2.bin
[3]: Verifying and copying expanded package files to bootflash:
4507-1#dir bootflash: | i conf
54328 -rw- 1726 Oct 24 2016 11:33:56 +00:00 packages.conf
4507-1#conf t
4507-1(config)#boot system flash bootlfash:packages.conf
4507-1#wr
Building configuration...
4507-1#reload
Proceed with reload? [confirm]
After the software mode change
4507-1#show mod
Chassis Type : WS-C4507R+E
Power consumed by backplane : 40 Watts

---+-----+--------------------------------------+------------------+-----------
1 24 1000BaseX (SFP) WS-X4624-SFP-E JAE1221IT3Q
3 12 Sup 8-E 10GE (SFP+), 1000BaseX (SFP) WS-X45-SUP8-E CAT1720L1UU
M MAC addresses Hw Fw Sw Status

--+--------------------------------+---+------------+----------------+---------
1 0021.1c7c.07d7 to 0021.1c7c.07ee 1.0 Ok
3 503d.e583.3740 to 503d.e583.374b 0.4 15.1(1r)SG5 03.07.02E Ok
Mod Redundancy role Operating mode Redundancy status We start to see wireless
----+-------------------+-------------------+----------------------------------
3 Active Supervisor SSO Active
Daughter Card
Mod Submodule Model Serial No. Hw Status
----+-----------------------+-----------------+------------+----+---------
3 Daughter Card WS-UA-SUP8E CAT1720L5N0 0.4 Ok
After the software mode change
4507-1(config)#wireless ?
assisted-roaming Configure dot11k assisted-roaming feature parameters
broadcast Enable Ethernet Broadcast Support
client Configure client parameters
dot11-padding Configure over-the-air frame padding
exclusionlist Manage exclusion list entries
ipv6 Global wireless IPv6 configurations
linktest Configure linktest frame size and number of frames to send
load-balancing Configure Aggressive Load Balancing
management Configure wireless management parameters
mdns-bridging Enable Ethernet mDNS Support
media-stream Config media stream groups
mgmt-via-wireless Enable management access from wireless clients
mobility Configure the Inter-Switch Mobility Manager
multicast Enable Ethernet Multicast Support
peer-blocking Configure p2p peer blocking Full support for wireless
probe Configure rate-limiting of probe requests.
qos Enable qos Services configuration
rf-network Sets the RF-Network Name
security Configure wireless security features
sip SIP preferred call numbers.
wgb Configure WGB client
wlancc Enable WLANCC on the controller
wps Global WPS settings
CLI Analyzer System
specific tools
ASA Tools
ASA Firewall Top Talkers
Helps determine which connections passing traffic through
an ASA have the highest bit rate during a certain period of
time.
ASA Packet Tracer

Allows administrators to simulate sending packets through
the ASA. If the packet is dropped, the ASA configuration
setting or feature that could have contributed to the packet
drop is identified.
ASA System Diagnostics

Utilizes Cisco TAC knowledge to analyze the ASA and
detect some known issues such as system problems,
configuration mistakes, and best practice violations.
ASA Traceback Analyzer

Attempts to match the root cause of a crash to a known bug
if the ASA has experienced a system traceback. If a
matching bug is found, the ASA version(s) in which the bug
is fixed are provided.
IOS-XR Tools
IOS-XR System Diagnostics
Utilizes Cisco TAC knowledge to analyze the router and
detect some known issues such as system problems,
configuration mistakes, and best practice violations.
IOS-XR BGP Top Talkers

Allows administrators to determine the BGP peers with the
rates rates of messages sent or received over a period of
time for the various VRFs and address-families.
IOS-XR L2VPN Top Talkers

Allows administrators to determine the L2VPN Point-To-
Point circuit or bridge-domains with the highest traffic rates.
IOS-XR LPTS Top Talkers

Helps administrators determine the type of traffic being
handed off from hardware to software and at which
rate.
Coming back to customer
network issues
Voice
quality
problem
Statistics
collection
problem
IPv6
connectivity
issue
CLI Analyzer - New tools
Packet capture
tool
Does not require to know
platform commands
GUI interface
Automatic packet capture

analysis
Packet
capture interface
CPU packet
capture filters
Platform filters:
direction
interfaces
Vlans
etc.
CPU packet
capture results
Flow analysis
Decodes
PCAP
Platform specific
CPU packet
capture - Top talkers
Conclusion
Summary
To troubleshoot, use step by step
approach, define the problem and
narrow it down
There is a variety of tools
available on Cisco Platforms
Tools depend on the platform and
require sufficient expertise
We are making effort to simplify
and add new troubleshooting
features and new tools, such as
Packet Trace and CLI Analyzer
Additional resources
Packet trace
http://www.cisco.com/c/en/us/support/docs/content-networking/adaptive-session-redundancy-asr/117858-technote-asr-00.html
ELAM overview
https://www.ciscolive365.com/connect/sessionDetail.ww?SESSION_ID=7608
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/116644-technote-product-00.html
FED tracing
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html
IPv6 First Hop Security

http://www.cisco.com/c/en/us/about/security-center/ipv6-first-hop.html
http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipv6_fhsec/configuration/xe-16/ip6f-xe-16-book.html
6500 High CPU troubelshooting

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/63992-6k-high-cpu.html
https://supportforums.cisco.com/document/59926/troubleshooting-high-cpu-6500-sup720
Wireless config guide on 4500 Sup8

http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-4500-series-switches/guide-c07-733704.html
3850 high cpu troubleshooting

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html
CLI Analyzer
https://cway.cisco.com/docs/cisco-cli-analyzer/2.0/About_the_Cisco_CLI_Analyzer.htm

Troubleshooting Cisco routers and switches

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Troubleshooting Cisco routers and switches

Uploaded by

Copyright:

Available Formats

Advanced feature troubleshooting

and best practices on Cisco

Date: November, 4 2016

Create awareness of the troubleshooting tools and techniques

Show how this tools apply in troubleshooting of real world issues,

Show a DEMO of tools implementation in the lab setup.

Understand the impact of the problem based on the information

Determine the scope of the impact:

1) Gain knowledge of troubleshooting capabilities on Cisco switches

2) Improve ability to choose the right troubleshooting tool for timely

This sessions covers:

Routers running IOS-XE ISR4000, ASR1000

show platform software process list RP active summary

show platform software process list RP active | inc fman

show platform hardware qfp active infrastructure exmem statistics

show platform hardware qfp active infrastructure exmem statistics user

New, simplified syntax:

Type: Name: DRAM, QFP: 0

Drivers Forwarding Manager

ASR1K#show platform resources Linux Kernel

SPA SPA SPA

Traffic did not reach its target !

Which feature went wrong ?

What went wrong in the feature ?

One way of capturing packets

The Packet Tracer and FIA Debugger

packets to be traced Output FIA Input ACL

Output ACL NAT

Summary data collection incurs minimal overhead over normal packet

Capturing path data potentially has significant impact on packet processing

debug platform packet-trace packet pkt-num [fia-trace | summary-only] [circular] [data-size

debug platform packet-trace drop [code code-num ]

debug platform condition stop

clear platform condition all

ASR1000 CSR1000V ISR4000

JTAG Ctrl PPE6 PPE7 PPE8 PPEN

RPs RPs ESP RPs SIPs

PPE PPE PPE DRAM

Mgt Eth 4xPCIe System 10G XAUI

ISR4K# debug platform packet-trace packet 16 fia-trace data-size 2048

ISR4K# debug platform condition start

Packet Copy Out

ASR1K-2# debug platform packet-trace packet 16 fia-trace data-size 2048

ASR1K-2# debug platform condition start

Service-policy output: DROP_af21

Class-map: class_af21 (match-all)

ASR1K-2#clear platform packet-trace statistics

ASR1K-2#debug platform condition start

ASR1K-2#show platform packet-trace summary

Pkt Input Output State Reason

0 Gi0/0/0 Gi0/0/2 FWD

asr1000# show platform packet-trace packet 0

Packet: 0 CBUG ID: 14231008

Use more /compressed <filename> command to decode:

F340.04.19-4400-11# more /compressed bootflash:/tracelogs/btrace_rotate_sh_R0-0$

Work in progress to make trace logs more simple

We need to confirm that our test client is actually working

VSS PARENT VSS PARENT

Gig 101/0/1 Gig 101/0/2

Device Under SPAN Destination

2 6880#test platform software console fex <fex-id> enable timeout <minutes>

6880#attach fex 101

All RTP SEQ numbers comming