Professional Documents
Culture Documents
BRKEWN-3011
Wesley Terry
BRKEWN-3011
Cisco Public
Additional Troubleshooting
BRKEWN-3011
Cisco Public
Supportability
BRKEWN-3011
Cisco Public
Supportability
WLC Supportability
Methods of Management Using the GUI Important Show Commands (CLI)
Important Debugs (CLI) Best Practices
AP Supportability
Methods of Accessing the AP Important Show Commands
BRKEWN-3011
Cisco Public
WLC Supportability
Methods of Management
GUI
HTTPS (E) / HTTP (D)
CLI
Console
SNMP
V1 (D) / V2 (E) Change me! V3 (E) Change me Note: Management Via Wireless Clients (D)
BRKEWN-3011
Cisco Public
WLC Supportability
Using the GUI
Monitor
AP/Radio Statistics
BRKEWN-3011
Cisco Public
WLC Supportability
Using the GUI
APs are sorted by Controller Associated Time Check bottom of AP list for any recent AP disruptions Select AP to see Controller Associated Time (duration)
BRKEWN-3011
Cisco Public
WLC Supportability
Using the GUI
Management
SNMP Config
BRKEWN-3011
Cisco Public
WLC Supportability
Important Show Commands (CLI)
Show run-config
Must have! No exceptions!
show run-config commands (like IOS show running-config) show run-config no-ap (no AP information added)
BRKEWN-3011
Cisco Public
WLC Supportability
Important Debugs (CLI)
Config session timeout 60, sets 60 minute idle timeout Debug mac addr <mac address> Used to filter debugs on specific Mac Address Debug disable-all (Disables all debugs)
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
10
WLC Supportability
Best Practices
BRKEWN-3011
Cisco Public
11
AP Supportability
Methods of Accessing the AP
Console Telnet (D) / SSH (D) No GUI support
AP Remote Commands
Default Mode (E)=Enabled (D)=Disabled
Enabling Telnet/SSH
WLC CLI: config ap [telnet/ssh] enable <ap name> WLC GUI: Wireless > All APs > Select AP > Advanced Select [telnet/ssh] > Apply
BRKEWN-3011
Cisco Public
12
AP Supportability
AP Remote Commands (WLC CLI)
BRKEWN-3011
Cisco Public
13
AP Supportability
Show Commands (AP CLI or WLC Remote Cmd)
Show log
WLC: show ap eventlog <ap name> Show capwap client <?>
CLI Tips
Debug capwap console client
Debug capwap client no-reload
BRKEWN-3011
Cisco Public
14
BRKEWN-3011
Cisco Public
15
BRKEWN-3011
Cisco Public
16
BRKEWN-3011
Cisco Public
17
The problem cannot be reproduced in the Cisco Technical Assistance Center lab
Access to the affected equipment for debugging purposes is not available
BRKEWN-3011
Cisco Public
18
BRKEWN-3011
Cisco Public
19
BRKEWN-3011
Cisco Public
20
BRKEWN-3011
Cisco Public
21
AssureWave
AW is no longer tagged on CCO, but AW validation results are available at: http://www.cisco.com/go/assurewave
Results available 4 weeks after CCO
MD
MD tag represents stable releases for mass adoption MD tag will be considered on CCO after AW release validation, 10 weeks in field and TAC/Escalation signoff
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
22
Escalation Code
Escalation is a post-CCO maintenance release with specific/minimal customer impacting SW fixes
Fix must be fully committed to the next CCO MR
Sanity + focus tested Fully TAC+BU supported Running-Master so each release builds upon the previous
BRKEWN-3011
Cisco Public
23
Troubleshooting Basics
BRKEWN-3011
Cisco Public
24
Radio
Driver
Supp.
Supplicant Logs
WLC EOIP
RADIUS
ACS
IP
DHCP
802.11 Management
WLC
Wired Sniff
Wired Sniff
DHCP Logs
Wireless Sniff
Spectrum Analysis
WLC Debugs
ACS Logs
NTP
BRKEWN-3011
Cisco Public
25
Troubleshooting Basics
Troubleshooting 101
Clearly define the problem Understand any possible triggers Know the expected behavior
Reproducibility
Questions
Problem Definition
Recommended Tools
Spectrum Analyzer Wireless Sniffer and Wired Captures
Tests
Analysis
Solution(s)
BRKEWN-3011
Cisco Public
26
Troubleshooting 101
Troubleshooting is an art with no right or wrong procedure, but best with a logical methodology.
Good description: Client associations are rejected with Status17 several times before they associate successfully.
BRKEWN-3011
Cisco Public
27
Troubleshooting 101
Step 2: Understand any possible triggers
If something previously worked but no longer works, there should be an identifiable trigger
Understanding any and all configuration or environmental changes could help pinpoint a trigger
BRKEWN-3011
Cisco Public
28
Troubleshooting 101
Step 4: Reproducibility
Any problem that has a known procedure to reproduce (or frequently randomly occurs) should be easy to diagnose
Being able to easily validate or disprove a potential solution saves time by being able to quickly move on to the next theory
If a problem is reproducible in other environments with a known procedure, TAC/BU can facilitate internal testing and proposed fix/workaround verification
Debugs and Captures of working scenarios can help pin point where exactly the difference is
BRKEWN-3011
Cisco Public
29
Recommended Tools
Wireless Sniffer
Example: Linksys USB600N with Omnipeek
TAC can publish Omnipeek-RA if you have compatible HW
Spectrum Analyzer
Spectrum Expert with Card or Clean-Air AP
BRKEWN-3011
Cisco Public
30
BRKEWN-3011
Cisco Public
31
8. (Optional: EAPOL Authentication) 9. (Optional: Encrypt Data) 10. Move User Data
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
32
BRKEWN-3011
Cisco Public
33
Description
802.1x (L2) Authentication Pending
DHCP_REQD WEBAUTH_REQD
RUN
(Cisco Controller) >show client detail 00:16:ea:b2:04:36 Client MAC Address............................... 00:16:ea:b2:04:36 .. Policy Manager State............................. WEBAUTH_REQD 00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
BRKEWN-3011
Cisco Public
34
BRKEWN-3011
Cisco Public
35
BRKEWN-3011
Cisco Public
36
Association
(Cisco Controller) >debug client 00:16:EA:B2:04:36 (Cisco Controller) > (Cisco Controller) > Association received from mobile on AP 00:26:cb:94:44:c0 0.0.0.0 START (0) Changing ACL 'none' (ACL ID 0) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:1621) Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site 'default-group', interface '3' Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface '3
STA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0 Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36
0.0.0.0 START (0) Initializing policy 0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2) 0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3) 0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client 0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1 apfMsAssoStateInc apfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to Associated Scheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0
BRKEWN-3011
Cisco Public
37
Association
Association received from mobile on AP 00:26:cb:94:44:c0 0.0.0.0 START (0) Changing ACL 'none' (ACL ID 0) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:1621) Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site 'default-group', interface '3' Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface '3'
Association received
Association Request, client did not Roam (Reassociate)
AP Base Radio = 00:26:cb:94:44:c0
vlan 3
Vlan = Vlan # of Dynamic Interface
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
38
Association
STA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0 Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36
STA - rates
Madatory Rates (>128) = (#-128)/2
BRKEWN-3011
Cisco Public
39
Association
0.0.0.0 START (0) Initializing policy 0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2) 0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3) 0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client 0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1 apfMsAssoStateInc apfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to Associated Scheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds
0.0.0.0 START
0.0.0.0 = IP we know for client (In this case nothing)
Scheduling deletion
Session Time on WLAN (1800 seconds in this case)
BRKEWN-3011
Cisco Public
40
Association
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0
BRKEWN-3011
Cisco Public
41
Association - FSR
Processing WPA IE type 221, length 22 for mobile 00:16:ea:b2:04:36 CCKM: Mobile is using CCKM CCKM: Processing REASSOC REQ IE Including CCKM Response IE (length 62) in Assoc Resp to mobile Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) Vap Id 6 Slot 1
OR
Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36 Received RSN IE with 1 PMKIDs from mobile 00:16:ea:b2:04:36 Received PMKID: (16) [0000] cb bc 27 82 88 14 92 fd 3b 88 de 6a eb 49 be c8 Found an entry in the global PMK cache for station Computed a valid PMKID from global PMK cache for mobile FSR aIOS CUWN
yes yes no
yes
Cisco Public
Association - Takeaway
Association vs. Reassociation Debug shows
AP, Slot, AP-Group, WLAN ID, Interface, Data Rates, Encryption type
Association Response
Confirms if Client is associated Defines reason if denied
Further troubleshooting
May require Wireless Sniffer or capture at AP Switchport If not sending Assoc Request, must know why from Client
BRKEWN-3011
Cisco Public
43
BRKEWN-3011
Cisco Public
44
802.1X Authentication
Supplicant Authenticator
Server
Radius-Access-Accept
(Key) Session Key
45
Cisco Public
WPA2-AES-802.1X
Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0 Station 00:16:ea:b2:04:36 setting dot1x reauth timeout = 1800 dot1x - moving mobile 00:16:ea:b2:04:36 into Connecting state
Processing Access-Accept for mobile 00:16:ea:b2:04:36 ***OR*** Processing Access-Reject for mobile 00:16:ea:b2:04:36
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
46
2 Notification
3 NAK 4 MD5 5 OTP 6 Generic Token 13 EAP TLS 17 LEAP
Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3) Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36 Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EAP Type 25)
BRKEWN-3011
Cisco Public
47
Cisco Public
48
WPA2-AES-PSK - Failed
Starting key exchange to mobile 00:1e:8c:0f:a4:57, data packets will be dropped Sending EAPOL-Key Message to mobile 00:1e:8c:0f:a4:57 state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00 Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57 Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57 Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57 Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57 802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57 Retransmit 1 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57 Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57 Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57 Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57 Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57 802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57 Retransmit 2 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57 802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57 Retransmit failure for EAPOL-Key M1 to mobile 00:1e:8c:0f:a4:57, retransmit count 3, mscb deauth count 3 Blacklisting (if enabled) mobile 00:1e:8c:0f:a4:57 apfBlacklistMobileStationEntry2 (apf_ms.c:4192) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:16:9c:4b:c4:c0 from Associated to Exclusion-list (1)
BRKEWN-3011
Cisco Public
49
L2 Authentication - Takeaway
8021X_REQD means L2 Authentication pending
Authentication/Encryption has not be established
PSK is 802.1X, key is derived from PSK not AAA If Processing Access-Reject
AAA/RADIUS Rejected the user (not the WLC)
If Processing Access-Accept
AAA/Radius Accepted the user M1-M4 should follow
Further Troubleshooting
Debug aaa [all/event/detail/packet] enable Debug dot1x [aaa/packet] enable
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
50
BRKEWN-3011
Cisco Public
51
Client DHCP
00:16:ea:b2:04:36 Received EAPOL-key in PTKINITNEGOTIATING state 00:16:ea:b2:04:36 apfMs1xStateInc
*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0 ................... 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 308,vlan 0, port 29, encap 0xec03) ................... 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00) ................... 00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20) 00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0
BRKEWN-3011
Cisco Public
52
Client DHCP
Client is in DHCP_REQD state
Client State = DHCP_REQD
Proxy Enabled:
DHCP Relay/Proxy
Proxy Disabled:
Between Client and Server DHCP is broadcast out VLAN IP helper or other means required
IP Address Learned
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
53
BRKEWN-3011
Cisco Public
54
BRKEWN-3011
Cisco Public
55
BRKEWN-3011
Cisco Public
56
38.172: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00) 38.173: 00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20) 38.173: 00:16:ea:b2:04:36 10.10.1.103 RUN (20) Reached PLUMBFASTPATH: from line 5273 38.173: 00:16:ea:b2:04:36 10.10.1.103 RUN (20) Replacing Fast Path rule 38.173: 00:16:ea:b2:04:36 Assigning Address 10.10.1.103 to mobile 38.173: 00:16:ea:b2:04:36 DHCP success event for client. Clearing dhcp failure count for interface management 38.174: 00:16:ea:b2:04:36 DHCP sending REPLY to STA (len 414, port 29, vlan 0) 38.174: 00:16:ea:b2:04:36 DHCP transmitting DHCP ACK (5) 38.174: 00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0 38.174: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 0 38.174: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36 38.174: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.1.103 38.174: 00:16:ea:b2:04:36 DHCP siaddr: 10.10.1.30, giaddr: 0.0.0.0 38.174: 00:16:ea:b2:04:36 DHCP server id: 1.1.1.1 rcvd server id: 10.10.1.3 38.179: 00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0
BRKEWN-3011
Cisco Public
57
*00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 3, port 29, encap 0xec00) *00:16:ea:b2:04:36 DHCP processing DHCP OFFER (2) *00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0 *00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 0, flags: 0 *00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36 *00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.3.86 *00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0 *00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3 *00:16:ea:b2:04:36 DHCP successfully bridged packet to STA
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
58
BRKEWN-3011
Cisco Public
59
Client sends IP packet (Orphan Packet), we learn IP DS sends packet to client, we learn IP from DS
Seen with mobile devices that talk before validating DHCP Up to client to realize their address is not valid for the subnet DHCP Required on WLAN for prevent this
BRKEWN-3011
Cisco Public
60
If Proxy is enabled
Confirm DHCP Server on Interface (or Wlan) is correct
DHCP Server may not respond to WLC Proxy (Firewalls?)
Further Troubleshooting
Check DHCP Server for what it believes is happening
If WLC does not show a BOOTREQUEST, confirm the client request arrives to the WLC and leaves in the configured way
BRKEWN-3011
Cisco Public
61
BRKEWN-3011
Cisco Public
62
Webauth
*apfReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)
*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state to WEBAUTH_REQD (8) last state WEBAUTH_REQD (8)
*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) pemAdvanceState2 5170, Adding TMP rule *DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Successfully plumbed mobile rule (ACL ID 255) *DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 Assigning Address 10.10.3.86 to mobile
*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state RUN (20)
*emWeb: 00:16:ea:b2:04:36 Session Timeout is 1800 - starting session timer for the mobile *emWeb: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063 *emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, TokenID = 5006 IPv6 Vlan = 3, IPv6 intf id = 8 *emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Successfully plumbed mobile rule (ACL ID 255)
*pemReceiveTask: May 17 22:25:16.578: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1, dtlFlags 0x0
BRKEWN-3011
Cisco Public
63
Webauth Redirect
Client in WEBAUTH_REQD state
ARP and DNS must be functional Client attempts to browse internet WLC Hijacks the handshake
Webauth
Client State = WEBAUTH_REQD
Client redirects to Virtual Interface Certificate negotiation if applicable Webauth page is displayed Client authenticates
Successful Authentication
Cisco Public
64
BRKEWN-3011
Cisco Public
65
3-Way Handshake HTTP GET 200 Response 3-Way Handshake HTTP(S) GET Webauth Page Displayed
WLC Responding with SYN, ACK Redirect to Virtual Interface Comes from Here
WLC Responding with SYN, ACK
Cisco Public
66
Webauth - Takeaway
If WEBAUTH_REQD, then not authenticated
Only traffic allowed is DHCP, ARP, DNS, Pre-Auth ACL, IPv6*
If not redirected, can client browse to virtual IP? Cert issue? Consider disabling HTTPS for HTTP webauth Most common scenario involves ARP/DNS failure
Must confirm that client actually sends TCP SYN (http) to IP
If proven that TCP SYN is sent and WLC does not SYN ACK, then there may be a WLC side problem
Debug webauth enable <client ip address> debug client <MAC Address>
debug pm ssh-appgw enable debug pm ssh-tcp enable
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
67
BRKEWN-3011
Cisco Public
68
Run State
10.10.3.82 DHCP_REQD (7) Change state to RUN (20) last state RUN (20) 10.10.3.82 RUN (20) Reached PLUMBFASTPATH: from line 5273 10.10.3.82 Added NPU entry of type 1, dtlFlags 0x0 OR 10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14) 10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state RUN (20) Session Timeout is 1800 - starting session timer for the mobile 10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063 10.10.3.86 Added NPU entry of type 1, dtlFlags 0x0
RUN State is the Client Traffic Forwarding State Client is Connected and should be functional
BRKEWN-3011
Cisco Public
69
BRKEWN-3011
Cisco Public
70
Deauthenticated Client
Idle Timeout
Occurs after no traffic received from Client
Session Timeout
Occurs at scheduled duration (default 1800 seconds)
Will force WEBAUTH user to WEBAUTH again
apfMsExpireCallback (apf_ms.c:608) Expiring Mobile! apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated Scheduling deletion of Mobile Station: (callerId: 45) in 10 seconds apfMsExpireCallback (apf_ms.c:608) Expiring Mobile! Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
71
Deauthenticated Client
WLAN Change
Modifying a WLAN in anyway Disables and Renables WLAN
apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983) Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)
Manual Deauth
From GUI: Remove Client From CLI: config client deauthenticate <mac address>
apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 6, reasonCode 1 Scheduling deletion of Mobile Station: (callerId: 30) in 1 seconds apfMsExpireCallback (apf_ms.c:608) Expiring Mobile! apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)
BRKEWN-3011
Cisco Public
72
Deauthenticated Client
Authentication Timeout
Auth or Key Exchange max-retransmissions reached
Retransmit failure for EAPOL-Key M3 to mobile 00:1e:8c:0f:a4:57, retransmit count 3, mscb deauth count 0 Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller 1x_ptsm.c:534)
BRKEWN-3011
Cisco Public
73
Deauthentication - Takeaway
Client can be removed for numerous reasons
WLAN change, AP change, configured interval
Start with Client Debug to see if there is a reason for a clients deauthentication Further Troubleshooting
Client debug should give some indication of what kind of deauth is happening
Packet capture or client logs may be require to see exact reason
BRKEWN-3011
Cisco Public
74
BRKEWN-3011
Cisco Public
75
Find All
Association Received (will also pull reassociations) Assoc Resp Access-Reject timeoutEvt
BRKEWN-3011
Cisco Public
76
BRKEWN-3011
Cisco Public
77
BRKEWN-3011
Cisco Public
78
BRKEWN-3011
Cisco Public
79
Client Connectivity
Unified Wireless Network: Troubleshoot Client Issues Document ID: 107585
Configuration Issues
SSID Mismatch Security Mismatch Disabled WLAN
Unsupported Data-Rates Disabled Clients Radio Preambles
MFP
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
80
802.11n Speeds
Troubleshoot 802.11n Speeds Document ID: 112055
Configuration Issues
11n Support Enabled
BRKEWN-3011
Cisco Public
81
802.11n A-MPDU/A-MSDU
Aggregation methods used could impact interop or performance
802.11n Status: A-MPDU Tx: Priority 0............................... Enabled Priority 1............................... Disabled Priority 2............................... Disabled Priority 3............................... Disabled Priority 4............................... Enabled Priority 5............................... Enabled Priority 6............................... Disabled Priority 7............................... Disabled A-MSDU Tx: Priority 0............................... Enabled Priority 1............................... Enabled Priority 2............................... Enabled Priority 3............................... Enabled Priority 4............................... Enabled Priority 5............................... Enabled Priority 6............................... Disabled Priority 7............................... Disabled
Cisco Public
BRKEWN-3011
82
BRKEWN-3011
Cisco Public
83
Secondary objective: Carry out RF analysis It is NOT a management or monitoring tool Focused to work off-line to the WLC
Not TAC supported Development: wlc-conf-app-dev@cisco.com General internal alias:wlc-conf-app@cisco.com Pet project: no official Cisco product.
BRKEWN-3011
Cisco Public
84
Where?
Support Forums DOC-1373
BRKEWN-3011
Cisco Public
85
Input Needed
Complete config output from WLC
Show run-config
It does not work with old show running-config or with TFTP backup, or with show tech
The show run-config acts as snapshot of current config + RF state Likely best to obtain config from SSH with
config paging disable
BRKEWN-3011
Cisco Public
86
BRKEWN-3011
Cisco Public
87
Functionality Overview
Audit Checks
BRKEWN-3011
Cisco Public
88
Functionality Overview
Config View
BRKEWN-3011
Cisco Public
89
BRKEWN-3011
Cisco Public
90
Reducing CCI
Turn off excess 2.4 radios. May want to do this gradually, e.g. turn off 20% of radios per attempt
After turning off excess radios, could set DCA sensitivity to high Let DCA/power settings settle down overnight. See how things look in the morning Repeat till you see the desired coverage in 2.4GHz
BRKEWN-3011
Cisco Public
91
BRKEWN-3011
Cisco Public
92
8 12 channels in use (20 seem to be too many for the 792x to scan) In all locations, seek this:
Hottest channel's AP is at least -67dBm
Next hottest AP on that channel is at least 19 dB below the hottest
BRKEWN-3011
Cisco Public
93
Additional Troubleshooting
BRKEWN-3011
Cisco Public
94
Additional Troubleshooting
Wireshark Tutorial
Mobility VoWiFi
BRKEWN-3011
Cisco Public
95
Wireshark Tutorial
BRKEWN-3011
Cisco Public
96
Wireshark Tutorial
Default Wireshark view might look like this:
BRKEWN-3011
Cisco Public
97
Wireshark Tutorial
Newer versions of Wireshark have a feature for Apply as Column
This will take any decodable parameter and make a column
BRKEWN-3011
Cisco Public
98
Wireshark Tutorial
Within seconds your wireshark can also have:
BRKEWN-3011
Cisco Public
99
Wireshark Tutorial
Filtering data is just as easy
BRKEWN-3011
Cisco Public
100
BRKEWN-3011
Cisco Public
101
Wireshark Tutorial
Wireshark can also de-encapsulate CAPWAP DATA
Edit > Preference > Protocols > CAPWAP
BRKEWN-3011
Cisco Public
102
Wireshark Tutorial
With CAPWAP de-encapsulated you can see all the packets to/from client (between AP and WLC)
BRKEWN-3011
Cisco Public
103
BRKEWN-3011
Cisco Public
104
BRKEWN-3011
Cisco Public
105
BRKEWN-3011
Cisco Public
106
BRKEWN-3011
Cisco Public
107
Sniffer Mode AP
Select channel to Sniff Select destination for traffic
BRKEWN-3011
Cisco Public
108
Sniffer Mode AP
Omnipeek has a Remote Adapter to capture this data Wireshark, just capture network adapter
NOTE: Wireshark does not open the port UDP 5000 PC will send ICMP Unreachables
BRKEWN-3011
Cisco Public
109
Sniffer Mode AP
With wireshark, filter !icmp.type == 3 Data (UDP 5000) still not intelligible yet
Decode as Airopeek
BRKEWN-3011
Cisco Public
110
Sniffer Mode AP
BRKEWN-3011
Cisco Public
111
AP Discover/Join
BRKEWN-3011
Cisco Public
112
AP Discover/Join
AP Runs Hunting Algorithm to Find Candidate Controllers to Join
BRKEWN-3011
Cisco Public
113
AP - Discover Process
AP Discovery Req to known and learned WLCs Broadcast
Reaches WLCs with MGMT Interface in local subnet of AP Use ip helper-address <ip> with ip forward-protocol udp
Dynamic
DNS: cisco-capwap-controller
DHCP: Option 43
Configured (nvram)
High Availability WLCs Pri/Sec/Ter/Backup Last WLC All WLCs in same mobility group as last WLC Manual from AP - capwap ap controller ip address <ip>
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
114
AP - Discover Process
X
Discover Request sent to all methods the AP knows Discover Response sent from all WLCs that received the Discovery Request
broadcast
115
BRKEWN-3011
Cisco Public
AP WLC Selection/Join
WLCs send Discovery Response back to AP
Name, Capacity, AP Count, Master?, AP-MGR, Load per APMGR
Master Controller Greatest available capacity Ratio of total capacity to available capacity
BRKEWN-3011
Cisco Public
116
Troubleshooting AP Discovery/Join
Lightweight AP (LAP) Registration to a Wireless LAN Controller (WLC), Document ID 70333 Make sure time on WLC is accurate! From AP:
Debug ip udp Debug capwap client events
From WLC
Debug mac addr <AP ethernet mac> Debug capwap [event/error/packet] enable
Debug pm pki enable
BRKEWN-3011
Cisco Public
117
RRM
BRKEWN-3011
Cisco Public
118
RRM
There are usually only two common scenarios or issues involving RRM
BRKEWN-3011
Cisco Public
119
RRM Debugs
WLC debug airewave-director <?>
AP
debug capwap rm mesurements debug capwap rm rogue
BRKEWN-3011
Cisco Public
120
Load Information
Receive Utilization.. 0 % Rx load to Radio
Nearby APs
AP 00:16:9c:4b:c4:c0 slot 0.. -28 dBm on 11 (10.10.1.5) AP 00:26:cb:94:44:c0 slot 0.. -32 dBm on 11 (10.10.1.4)
BRKEWN-3011
Cisco Public
121
Broadcast/Multicast
BRKEWN-3011
Cisco Public
122
Broadcast/Multicast
BRKEWN-3011
Cisco Public
123
Broadcast/Multicast
AP Multicast Mode Multicast
Address must be unique among WLCs
Broadcast Traffic is delivered via the Multicast Mode AP/WLC/Client Subnets must be Multicast enabled
For Multicast Mode - Multicast
BRKEWN-3011
Cisco Public
124
Broadcast/Multicast
AP Show Commands
Show capwap mcast Show capwap mcast mgid all
BRKEWN-3011
Cisco Public
125
Client Mobility
BRKEWN-3011
Cisco Public
126
MobilityIntra-Controller
Client roams between two APs on the same controller
BRKEWN-3011
Cisco Public
127
MobilityInter-Controller (Layer 2)
BRKEWN-3011
Cisco Public
128
MobilityLayer 3
Layer 3 roaming (a.k.a. anchor/foreign)
New WLC does not have an interface on the subnet the client is on New WLC will tell the old WLC to forward all client traffic to the new WLC
BRKEWN-3011
Cisco Public
129
MobilityMessaging Flow
When a client connects to a WLC for the first time, the following happens:
New WLC sends MOBILE_ANNOUNCE to all controllers in the mobility group when client connects
Old WLC sends HANDOFF_REQUEST New WLC sends HANDOFF_REPLY
BRKEWN-3011
Cisco Public
130
BRKEWN-3011
Cisco Public
131
BRKEWN-3011
Cisco Public
132
BRKEWN-3011
Cisco Public
133
Cisco Public
134
Auto Anchoring Fast Secure Roaming APs get all of these as a Discover candidate
Auto Anchoring
BRKEWN-3011
Cisco Public
135
BRKEWN-3011
Cisco Public
136
BRKEWN-3011
Cisco Public
137
VoWiFi
Wireless IP Phone Deployment Guide
http://www.cisco.com/en/US/docs/voice_ip_comm/cuipph/79 25g/7_0/english/deployment/guide/7925dply.pdf
Best Practices
-67 dBm signal with 20-30% cell overlap
802.11A CCKM for Fastest Roaming Avoid designs where AP is seen at superb signal, but drops off instantly
BRKEWN-3011
Cisco Public
138
VoWiFi - Troubleshooting
Must know if problem occurs during roaming events or when no association change takes place
If no change in connection
Interference Coverage loss with no other candidate End to End QOS missing/problem
Does the client associate to the same AP again? Is the phone roaming to the designed next candidate?
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
139
VoWiFi - Troubleshooting
Define a reproducible area where you believe you have perfect voice coverage but have problems
Place phone in Neighbor List Mode (On a call)
Real Time current AP RSSI and candidate list Confirm AP as next best candidate is realistically a good candidate
Confirm devices roams to correct candidate where the intended design specifies
BRKEWN-3011
Cisco Public
140
VoWiFi - Debugs
Phone can Trace (debug) to file or syslog
Recommend USB Connection and SYSLOG
Configured via GUI Enable Debug level for Kernel, WLAN MGR, WLAN Driver
WLC Debugs
Debug client <mac> Debug cac all enable
BRKEWN-3011
Cisco Public
141
Summary
BRKEWN-3011
Cisco Public
142
Summary
Client
WLC - show run-config, debug client <mac>, debug dhcp message enable, debug dot1x <?> enable, debug aaa <?> enable, AP - Show tech, show controller D<0/1> Data - Driver/Supplicant Logs, Wireless Capture, AAA Logs, DHCP Logs
Webauth
WLC - (Client debugs), debug webauth enable <IP>, debug pm ssh-appgw enable, debug pm ssh-tcp enable Client - local capture
Mobility
WLC - debug mobility handoff enable, debug mobility keepalive enable <IP> Data - Wired capture
AP Join
WLC - debug capwap [events/error/packet] enable AP - debug capwap client events, debug ip udp Data - Wired capture
RRM
WLC - show run-config, debug airewave-director <?> AP - debug capwap rm measurements, debug capwap rm rogue
Multicast/Broadcast
AP - show capwap mcast, show capwap mcast mgid all Data - Infrastructure Configuration
Voice
WLC - (Client debugs), debug cac all enable Data Wireless capture, Phone traces
BRKEWN-3011
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
143
Summary
Links:
Understanding Debug Client on Wireless LAN Controllers (WLCs) Document ID: 100260
Unified Wireless Network: Troubleshoot Client Issues Document ID: 107585 Troubleshoot 802.11n Speeds Document ID: 112055 Troubleshoot a Lightweight Access Point Not Joining a Wireless LAN Controller Document ID: 99948
BRKEWN-3011
Cisco Public
144
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Dont forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.
BRKEWN-3011
Cisco Public
145
BRKEWN-3011
Cisco Public
146
BRKEWN-3011
Cisco Public
147
Thank you.
BRKEWN-3011
Cisco Public
148