Troubleshooting the Cisco Application Control Engine BRKAPP-3003 (c) 2011 Cisco and / or its affiliates. All rights reserved. Understanding the architecture and flow management will help troubleshoot the Application control Engine.
Troubleshooting the Cisco Application Control Engine BRKAPP-3003 (c) 2011 Cisco and / or its affiliates. All rights reserved. Understanding the architecture and flow management will help troubleshoot the Application control Engine.
Troubleshooting the Cisco Application Control Engine BRKAPP-3003 (c) 2011 Cisco and / or its affiliates. All rights reserved. Understanding the architecture and flow management will help troubleshoot the Application control Engine.
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public BRKAPP-3003
1 Advanced Troubleshooting the Cisco Application Control Engine BRKAPP-3003 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 2 Core Message Understanding the architecture and flow management will help troubleshoot the Application Control Engine 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 3 Session Objective ACE Architecture Understand the ACE architecture and connectivity through ACE Verify software images, licenses and image recovery Use the real-time TCP-DUMP command Understand access list and ACL merge on ACE Flow Management Understand the difference between L4 and L7 processing Check for possible asymmetric flows Provide layer 7 troubleshooting Ability to monitor performance and troubleshoot resources Understand high availability At the End of the Session, You Will Be Able To: 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 4 Session Agenda ACE Architecture Discuss the Architecture Functions of control plane and data plane Common debugging commands Packet Capturing and logging Traffic Forwarding on ACE Admin Context and ACL Merge Flow Management Connection Handling on ACE Layer 4/7 Troubleshooting and Performance Health Monitoring on ACE High Availability on ACE 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 55 ACE Architecture 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 6 ACE20 Module Hardware Architecture Switch Fabric Interface 16G Daughter Card 1 Daughter Card 2 8G 8G SSL Crypto 10G 2G Console port Sup Connect 100M Control Plane Network Processor 1 Network Processor 2 10G 10G Classification Distribution Engine (CDE) 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 7 ACE30 Module Hardware Architecture Switch Fabric Interface 16G 2G Console port Sup Connect 100M Control Plane 8G Daughter Card 1 NP1 NP2 8G Daughter Card 2 NP1 NP2 Classification Distribution Engine (CDE) 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 8 2x 700MHz MIPS 1 GB Memory Control Plane Software Supervisor Connection DBUS 16 Gbps Bus RBUS EOBC Cisco ASIC 100 Mbps 8 Gbps 8 Gbps 1 Gbps ACSW OS 60Gbps switching Capacity IPv4, IPv6 Classifications TCP Checksum Generation/Verification Variable Load Distribution Daughter Card 1 16 Gbps CEF720 Linecard 20 Gbps 20 Gbps Switch Fabric ACE30 Detailed Hardware Architecture CPU Classification Distribution Engine (CDE) Network Processor 1 Verni FPGA DRAM 4 GB DRAM 4 GB Network Processor 2 shared memory Daughter Card 2 Network Processor 3 Verni FPGA DRAM 4 GB DRAM 4 GB Network Processor 4 shared memory Cavium Octeon CN5860 (OcteonPlus) 16 core, 600 MHz CPUs with 4G DRAM 32k iCache, 16k dCache, 2MB L2 cache On chip support for Encryption/Decryption Coprocessors for Compression/Decompression 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 9 Data Traffic vs. Management Traffic ACE30 Control plane architecture is very similar to ACE20 Device control Configuration manager (CLI, XML API, SSH, ) Server health monitoring (native probes, TCL scripts) Syslog's, SNMP, ARP, DHCP relay High-Availability ACL Compilation ACE30 data plane architecture is very similar to ACE 4710 Connection management TCP termination Access lists NAT SSL Offload Regular expression matching Load Balancing & forwarding 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 10 10 Common Debugging 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 11 Common Debugging 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 12 Common Debugging Show commands on the Catalyst 6500 Supervisor show version show clock show module show power show asic slot <n> show interface TenGigabitEthernet <n>/1 show interface TenGigabitEthernet <n>/1 trunk show svclc vlan-group [no] power enable <module> show svclc module <n> traffic Make sure the module status is OK VLANs used by ACE must be configured in the MSFC 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 13 Common Debugging Show commands available on ACE show version show cde health show ft group status show ip int br show int vlan <n> show arp show service-policy show serverfarm show rserver show probe show conn show stat show ip traffic show resource usage show np 1 me-stats -s norm show np 1 me-stats -s norm M1 System Information L2, L3 Performance, Resources Debugging Flows L4, L7 This provides the DELTA If incorrect version, check bootparameter 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 14 Show Module from the Catalyst 6500 Supervisor cat6k#show mod Mod Ports Card Type Model Serial No. --- ----- -------------------------------------- ------------------ ----------- 1 1 Application Control Engine 10G Module ACE20-MOD-K9 SAD12345678 2 48 48 port 10/100 mb RJ45 WS-X6348-RJ-45 SAD04450L44 5 2 Supervisor Engine 720 (Active) WS-SUP720-3BXL SAD08300D5L Mod MAC addresses Hw Fw Sw Status --- ---------------------------------- ------ ------------ ------------ ------- 1 0001.0002.0003 to 0001.0002.000a 2.4 8.7(0.22)ACE A2(2.3a) Ok 2 00d0.d32e.1b42 to 00d0.d32e.1b71 1.5 5.4(2) 8.5(0.46)RFW Ok 5 000f.f7be.b17c to 000f.f7be.b17f 4.0 8.1(3) 12.2(PP_R31_ Ok Mod Sub-Module Model Serial Hw Status ---- --------------------------- ------------------ ----------- ------- ------- 5 Policy Feature Card 3 WS-F6K-PFC3BXL SAD083006N2 1.3 Ok 5 MSFC3 Daughterboard WS-SUP720 SAD082905VE 2.1 Ok Mod Online Diag Status ---- ------------------- 1 Pass 2 Pass 5 Pass Module status shows OK 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 15 Verifying Version and Licenses ACE/Admin# show version Cisco Application Control Software (ACSW) <snip> Software loader: Version 12.2[121] system: Version A2(2.3a) [build 3.0(0)A2(2.3a) system image file: [LCP] disk0:c6ace-t1k9-mz.A2_2_3a.bin installed license: ACE-08G-LIC ACE-VIRT-250 ACE-SSL-15K-K9 Hardware Cisco ACE (slot: 1) cpu info: number of cpu(s): 2 cpu type: SiByte cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz Installed Licenses 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 16 Available System Memory and Uptime ACE/Admin# show version Continuation of output [...] memory info: total: 827128 kB, free: 335372 kB shared: 0 kB, buffers: 3540 kB, cached 0 kB cf info: filesystem: /dev/cf total: 1014624 kB, used: 529472 kB, available: 485152 kB last boot reason: NP 2 Failed: NP ME Hung configuration register: 0x1 ACE kernel uptime is 7 days 23 hours 42 minute(s) 25 second(s) Displays ACE module uptime Useful information in case of system reload 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 17 ACE File System Use the dir command to view directory listing for files ACE/Admin# dir ? core: Directory or filename disk0: Directory or filename image: Directory or filename probe: Directory or filename volatile: Directory or filename The internal File system is mapped as below /mnt/cf - Image: Also the following compressed file systems are used /TN-HOME = disk0: /TN-CONFIG = Startup config /TN-LOGFILE = Internal Storage for audit logs /TN-CERTKEY-STORAGE : internal storage for Cert and Keys /TN-COREFILE = core: 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 18 ACE File System Load debug plug-in to access ACE file system Startup configuration located at /mnt/cf/TN-CONFIG ACE will generate / fix any missing or corrupted file systems during boot When to use the format command? If you receive the following error Warning!! This will erase everything in the compact flash including startup configs for all the contexts and reboot the system!! ACE/Admin# write memory ERROR!config filesystem is not mounted on compact flash 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 19 Working with Core Files If ACE creates a core file you can locate the files in the core directory All cores files are stored in dir core: (core names are self explanatory) ACE/Admin# dir core: 99756 Apr 5 17:57:05 2007 ixp2_crash.txt 13047 Apr 5 17:56:59 2007 loadBalance_core_log.tar.g Ixpx_crash.txt will have some details on the core dump If it is a kernel crash , then a file named crash info will be available in core Show version will show last reload reason 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 20 20 System Logging 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 21 Logging Features Each virtual context generates logs independently and sends to specified destinations Syslog server, console, buffer, SNMP station, etc.. Rate limiting of syslog messages is recommended. Never log to the console using level 7 ACE can log connection setup/teardown at the connection speed Access-List deny entries are logged Use the terminal monitor command to display log message when not using console Useful commands to troubleshoot syslog issues: show logging statistics show logging history show logging queue Make sure logging queue size is set properly 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 22 Basic Configuration to Enable Logging Enable logging on the ACE logging enable logging timestamp logging monitor 4 logging trap 4 logging buffer 4 logging history 4 logging queue 1024 no logging message 111008 It is recommended to disable or change the severity level of some syslog messages. Use logging message syslog_id [level severity_level] command To enable the logging of connection setup and teardown messages, use the logging fastpath command. Use the logging rate-limit to limit the rate at which the ACE generates messages to the syslog server 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 23 23 Real-Time TCP Dump 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 24 Real-Time TCP Dump Supportability and analysis of load balanced traffic is a major requirement in today's load balanced environment ACE can capture real-time packet information for the network traffic that passes through it The attributes of the packet capture are defined by an ACL The ACE buffers the captured packets, and you can copy the buffered contents to a file in flash memory on the ACE or to a remote server User can also display the captured packet information on your console or terminal; capture can also be exported and viewed using Ethereal or Wireshark 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 25 Real-Time TCP Dump To enable the packet capture on ACE use the capture command capture c1 interface vlan 211 access-list FILTER bufsize 64 Buffer in Kbytes (can be circular) Pre-defined ACL to identify relevant traffic Interface to apply capture One capture session per context Capture triggered at flow setup Capture configured on client interface where flow is received 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 26 Real-Time TCP Dump ACE can capture traffic based on a configured access-list and interface Follow the following procedure to capture traffic on ACE: 1. Specify an ACL 2. Capture on an interface or globally access-list FILTER line 10 extended permit tcp any any eq www capture c1 interface vlan 211 access-list FILTER Show capture status show status and buffer size ACE/Admin# show capture c1 status Capture session : c1 Buffer size : 64 K Circular : no Buffer usage : 1.00% Status : stopped 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 27 Real-Time TCP Dump Start the capture on the ACE ACE/Admin# capture c1 start 23:40:37.236868 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 58: 172.16.11.190.443 > 209.165.201.11.1180: S 1389739009:1389739009(0) ack 617249474 win 17408 <mss 1460> (ttl 255, id 2401, len 44, bad cksum 0!) 23:40:37.239102 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 54: 172.16.11.190.443 > 209.165.201.11.1180: . ack 71 win 17408 (ttl 255, id 2402, len 40, bad cksum 0!) ACE/Admin# capture c1 stop To copy the packet capture to disk0: use the copy capture ACE/Admin# copy capture c1 disk0: c1 Maximum buffer size is 5MB of data 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 28 28 Traffic Forwarding on ACE 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 29 ACE Load Balancer Policy Lookup Order There can be many features applied on a given interface, so feature lookup ordering is important The feature lookup order followed by data path in ACE is as follows: 1. Access-control (permit or deny a packet) 2. Management traffic 3. TCP normalization/connection parameters 4. Server load balancing 5. Fix-ups/application inspection 6. Source NAT 7. Destination NAT The policy lookup order is implicit, irrespective of the order in which the user configures policies on the interface 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 30 ACE in Routed Mode IP subnets cannot overlap within a context but can across two contexts Non-Load balanced traffic is routed. ACE needs to ARP for destination before forwarding packet Client MAC ACE MAC Client IP VIP Random Port VIP Port ACE MAC Selected Server MAC Client IP Server IP Random Port Server Port 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 31 ACE in Bridge Mode Non-Load balanced connections are bridged from client to server vlan Client MAC ACE MAC Client IP VIP Random Port VIP Port Client MAC Selected Server MAC Client IP Server IP Random Port Server Port 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 32 Checking VLAN Configuration Show interface provides you with valuable information ACE/Admin# show interface vlan 211 vlan210 is up Hardware type is VLAN MAC address is 00:16:36:fc:b3:36 Virtual MAC address is 00:0b:fc:fe:1b:02 Mode : routed IP address is 172.16.10.21 netmask is 255.255.255.0 FT status is active Description:WAN Side MTU: 1500 bytes Last cleared: never Alias IP address is 172.16.10.23 netmask is 255.255.255.0 Peer IP address is 172.16.10.22 Peer IP netmask is 255.255.255.0 Assigned on the physical port, up on the physical port 499707 unicast packets input, 155702918 bytes 1485258 multicast, 5407 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 497610 unicast packets output, 46804782 bytes 6 multicast, 8201 broadcast 0 output errors, 0 ignored 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 33 MAC Addresses Virtual MAC (VMAC) is used for the alias IP, VIP address Alias IP and Virtual IP (VIP) are associated with a VMAC only if high availability is configured Active context responds to ARPs for alias IP with VMAC One unique VMAC per FT Group 00:0b:fc:fe:1b:XX (XX=FT group number in hex) Packets destined to the VMAC are blocked on standby context 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 34 MAC Addresses The VMAC is a function of ft-group-id. Therefore different cards must have different ft-group-ids Use the show interface internal iftable to locate the VMAC Each ACE supports 1,024 shared VLANs, and uses only one bank of MAC addresses randomly selected at boot time ACEs may select the same address bank so avoid this conflict use the shared-vlan-hostid command 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 35 Key Things to Know About ARP on ACE For unicast packets, if the destination MAC is unknown ACE will drop the packet, instead of flooding it So IP-address-to-MAC mapping and outgoing interface needs to happen first ARP entries are populated as follows: With ARP requests Learning through incoming ARP requests Gratuitous ARP packets Layer 2 mode: ARP is the only way to learn IP to MAC and interface mapping 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 36 36 Admin Context Resource Reservation 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 37 Admin Context Resource Reservation If Admin context is not configured correctly, Admin could be starved of all resources When configuring resource allocations in ACE, it is possible to allocate 100% of resources to non-Admin contexts, so that the Admin context is no longer reachable via ICMP, telnet, SNMP, etc In some cases, this could cause FT between a pair of HA ACE modules to fail, and create an active/active situation Highly recommended to put some safeguard in place to ensure that the Admin context always receives at least a small percentage of resources 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 38 Admin Context Resource Reservation Shows starved resources and drops for throughput ACE/Admin# show resource usage context Admin Allocation Resource Current Peak Min Max Denied ------------------------------------------------------------------------------- Context: Admin conc-connections 9 9 0 0 0 mgmt-connections 2 12 0 0 0 proxy-connections 0 0 0 0 0 xlates 0 0 0 0 0 bandwidth 0 4715 0 0 3704068 throughput 0 4247 0 0 3704068 mgmt-traffic rate 0 468 0 125000000 0 connection rate 0 7 0 0 8 ssl-connections rate 0 0 0 0 0 mac-miss rate 0 1 0 0 0 inspect-conn rate 0 0 0 0 0 acl-memory 26816 26880 0 0 0 sticky 0 0 0 0 0 regexp 0 0 0 0 0 syslog buffer 1024 4096 0 1024 0 syslog rate 0 7 0 0 118 No resources reserved 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 39 Admin Context Resource Reservation Shows heartbeats missed increasing. Heartbeats are not reaching the peer. Possibility for both ACEs to go Active/Active ACE/Admin# sh ft stats HA Heartbeat Statistics ------------------------ Number of Heartbeats Sent : 1095573 Number of Heartbeats Received : 1092586 Number of Heartbeats Missed : 2987 Number of Unidirectional HB's Received : 2640 Number of HB Timeout Mismatches : 0 Num of Peer Up Events Sent : 1 Num of Peer Down Events Sent : 1 Successive HB's miss Intervals counter : 0 Successive Uni HB's recv counter : 0 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 40 Admin Context Resource Reservation Below shows the problem why ACE is starved of all resources resource-class admin limit-resource all minimum 0.10 maximum equal-to-min Suggest the following reserved resources for Admin resource-class Admin limit-resource conc-connections min 5.00 max equal-to-min limit-resource mgmt-connections min 5.00 max equal-to-min limit-resource rate bandwidth min 5.00 max equal-to-min limit-resource rate ssl-connections min 5.00 max equal-to-min limit-resource rate mgmt-traffic min 5.00 max equal-to-min limit-resource rate conc-connections min 5.00 max equal-to-min 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 41 41 Access-Control Lists and ACL Merge 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 42 ACL Merge Process and Enhancements New ACL merge enhancements added to ACE ACL merge is responsible for merging all the features and generating a single merged list for an given interface. ACL compiler is responsible for programming the merged list into MTrie data structure Fast retrieval of data ACL memory usage has been optimized to better support incremental changes The new implementation provides a consistent ACL memory usage during system bootup time and during incremental changes after the system comes up This feature also provides an early detection of failure if the configuration needs more ACL resources than available Also, note ACL masks are in 255.255.x.x format (not 0.0.y.y) 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 43 View Total Action Nodes Use the show np 1 access-list resource to view action nodes ACE/Admin# show np 1 access-list resource ACL Tree Statistics for Context ID: 3 ======================================= ACL memory max-limit: None ACL memory guarantee: 0.00 % MTrie nodes(used/guaranteed/max-limit): 6 / 0 / 262143 (compressed) 2 / 0 / 21999 (uncompressed) Leaf Head nodes (used/guaranteed/max-limit): 3 / 0 / 262143 Leaf Parameter nodes (used/guaranteed/max-limit): 7 / 0 / 524288 Policy action nodes used: 4 memory consumed: 4696 bytes resource-limited 128 bytes other 4824 bytes total. min-guarantee: 0 bytes total. max-limit: 78610432 bytes total, 0 % consumed 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 44 44 Connection Handling in ACE 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 45 Flow Management Level of Flow Processing Type of Processing Feature of Function Layer 3 and Layer 4 Balance on first packet Basic Load Balancing Applies to TCP/UDP for layer 4 rules Source IP Sticky Applies to all other IP protocols TCP/IP Normalization Layer 7 TCP Splicing Terminate TCP Connection HTTP Layer 7 rules based on first request (URL LB) Buffer request, inspect, LB Cookie Sticky (Persistence) Create Hardware Shortcut Generic TCP Payload Parsing Layer 7 Re-proxy TCP Splicing + ability to parse subsequent HTTP requests within the same TCP HTTP Layer 7 rules with HTTP 1.1 connections keepalive (persistence rebalance) Layer 7 Full-Proxy Fully terminate clients connection SSL Offload TCP re-use HTTP 1.1 Pipelining Protocol Inspection (FTP,SIP) 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 46 Internal Mapping of TCP/UDP Flows TCP and UDP Flows = 2 X Internal Half Flows ACE/Admin# show conn conn-id np dir proto vlan source destination stat -------------+--+----+--------+-----+--------------------------+-------------------------------+---------+ 9 1 In TCP 211 209.165.201.11:1867 172.16.11.190:80 ESTAB 6 1 Out TCP 411 192.168.1.11:80 209.165.201.11:1867 ESTAB Client IP:port VIP Address Server IP Returning half flow automatically created for both TCP and UDP flows INIT, SYNACK, ESTAB, CLOSED SYN_SEEN, SYN_SEEN, ESTAB, CLOSED Non TCP shows as -- Use conn-id to track flow through ACE Check the Network Processor 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 47 Troubleshooting Connections Use the show stats connection command to show connections statistics Use the clear stats connection command to clear these counters ACE/Admin# show stats connection +------------------------------------------+ +------- Connection statistics ------------+ +------------------------------------------+ Total Connections Created : 288232 Total Connections Current : 2 Total Connections Destroyed: 283404 Total Connections Timed-out: 892 Total Connections Failed : 3934 Note: ACE does not destroy connection. These are connections closed correctly!!! 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 48 Troubleshooting Connections Use the show stats loadbalance command to view the load balance statistics To clear the load balance statistical information stored in the ACE buffer, use the clear stats loadbalance command ACE/Admin# show stats loadbalance +------------------------------------------------------------+ +------- Loadbalance statistics ----------------------+ +------------------------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0 Total Layer4 rejections : 0 Total Layer7 decisions : 24 Total Layer7 rejections : 0 Total Layer4 LB policy misses : 0 Total Layer7 LB policy misses : 0 Total times rserver was unavailable : 0 Total ACL denied : 0 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 49 Troubleshooting Individual Connections Use the NP and connection ID from show conn command to view the front-end and back-end connection statistics using show np <#> me-stats -c <connection ID> -v ACE/Admin# show np 1 me-stats -c 4096 v +------------------------------------------------------------+ +------- Individual connection statistics -------------------+ +------------------------------------------------------------+ Connection ID:seq: 4096[0x1000].2 Other ConnID : 8194[0x2002].14 Proxy ConnID : 0[0x0].0 Next Q : 0[0x0] 10.1.1.22:23 -> 12.2.2.14:8739 [RX-NextHop: TX] [TX-NextHop: CP] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 6 Inbound Flag : 0 Interface Match : Yes Interface MatchID:24 <snip> 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 50 Troubleshooting Individual Connections To further debug and check if the traffic pattern matches the correct rule, the following command can be used: show np 1 access-list trace vlan <inbound vlan> in protocol <IP protocol #> source <source IP> <source port or 0> destination <destination IP> <destination port> ACE/Admin# show np 1 access-list trace vlan 10 in protocol 6 source 10.10.10.1 0 destination 10.20.30.40 80 <snip> <look for NAT pool ID, vserver ID, etc.> src nat 0x0 dst nat 0x0 vserver 0x66 fixup 0x0 <snip> <vserver ID here is 0x66 or 102 decimal> Now, the internal vserver ID 102 can be looked up in the config: ACE/Admin# show cfgmgr internal table l3-rule | inc 102 102 224 249 0 0 DATA_VALID Internal Policy Map # is 224 and Class Map # is 249: ACE/Admin# show cfgmgr internal table policy-map | inc 224 224 MyPolicy9 0 DATA_VALID ACE/Admin# show cfgmgr internal table class-map | inc 249 249 MyClass4 0 DATA_VALID 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 51 Troubleshooting VIP ACE/Admin# show service-policy client-vips detail Status : ACTIVE Description: - ----------------------------------------- Interface: vlan 211 service-policy: client-vips class: VIP-HTTPS VIP Address: Protocol: Port: 172.16.11.190 tcp eq 443 loadbalance: L7 loadbalance policy: HTTPS-POLICY VIP Route Metric : 77 VIP Route Advertise : DISABLED VIP ICMP Reply : ENABLED-WHEN-ACTIVE VIP State: INSERVICE curr conns : 22 , hit count : 22 dropped conns : 0 client pkt count : 0 , client byte count: 0 server pkt count : 0 , server byte count: 0 max-conn-limit : 0 , drop-count : 0 conn-rate-limit : 0 , drop-count : 0 bandwidth-rate-limit : 0 , drop-count : 0 L7 Loadbalance policy : HTTPS-POLICY class/match : class-default LB action : primary serverfarm: backend-ssl backup serverfarm : - hit count : 22 dropped conns : 0 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 52 Troubleshooting Serverfarm Best command for checking server status and load ACE/Admin# show serverfarm HTTPS-FARM detail serverfarm : HTTPS-FARM, type: HOST total rservers : 4 active rservers: 4 description : - state : ACTIVE predictor : ROUNDROBIN failaction : - back-inservice : 0 partial-threshold : 0 num times failover : 0 num times back inservice : 0 total conn-dropcount : 0 ----------connections----------- real weight state current total failures ---+---------------------+--------+---------------------+-----------+------ rserver: linux-1 192.168.1.11:0 8 OPERATIONAL 0 0 0 max-conns : - , out-of-rotation count : - min-conns : - conn-rate-limit : - , out-of-rotation count : - bandwidth-rate-limit : - , out-of-rotation count : - retcode out-of-rotation count : - 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 53 53 Layer 7 Troubleshooting 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 54 Layer 7 Policy Hits Expanding the show service-policy using the detail option to provide hit count for layer 7 matches ACE/Admin# show service-policy client-vips detail Status : ACTIVE Description: - ----------------------------------------- Interface: vlan 211 service-policy: client-vips <snip> L7 Loadbalance policy : pslb class-map : curl1 LB action : serverfarm: s1 hit count : 3 dropped conns : 0 class-map : curl2 LB action : serverfarm: s2 hit count : 0 dropped conns : 0 Shows hit count for layer 7 load balanced policy 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 55 Match URL Hit Count Expanding the show service-policy using the url-summary option to provide visibility on which match http url are getting hit ACE/Admin# show service-policy url-summary Service-Policy: VIRTUAL-HOSTING-01 L3-Class: WEB-SSL L7-Class: VH-01 match http url /ACCOUNTING/.* hit: 42 Service-Policy: VIRTUAL-HOSTING-02 L3-Class: WEB-SSL L7-Class: VH-02 match http url /BUSINESS/.* hit: 93 match http url /SALES/.* hit: 102 match http url /SPECIAL/.* hit: 67 match http url /BUSINESSOBJECTS/.* hit: 78 match http url /CUSTOMERS/.* hit: 84 Use the show service-policy <service-policy-name> class- map <L3-class map-name> url-summary to provide better granularity 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 56 Troubleshooting HTTP To effectively troubleshoot HTTP use the show stat http command ACE/Admin# show stats http +------------------------------------------+ +-------------- HTTP statistics -----------+ +------------------------------------------+ LB parse result msgs sent : 6288 , TCP data msgs sent : 9143 Inspect parse result msgs : 0 , SSL data msgs sent : 6041 TCP fin/rst msgs sent : 135 , Bounced fin/rst msgs sent: 19 SSL fin/rst msgs sent : 13 , Unproxy msgs sent : 0 Drain msgs sent : 3107 , Particles read : 37917 Reuse msgs sent : 1539 , HTTP requests : 3145 Reproxied requests : 0 , Headers removed : 1549 Headers inserted : 1598 , HTTP redirects : 2 HTTP chunks : 0 , Pipelined requests : 0 HTTP unproxy conns : 0 , Pipeline flushes : 0 Whitespace appends : 0 , Second pass parsing : 0 Response entries recycled : 3032 , Analysis errors : 0 Header insert errors : 1509 , Max parselen errors : 0 Static parse errors : 9 , Resource errors : 0 Invalid path errors : 0 , Bad HTTP version errors : 0 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 57 Troubleshooting HTTP Cookies ACE parses HTTP requests for cookies with the name given in the configuration and can skip a certain number of bytes and look for another specific number of bytes. If the cookie is not found, then the ACE looks for a string in the URL, starting with one of the characters /?&#+ and followed by a "=", then parses that value. If no cookie or HTTP URL cookie exists ACE defaults to the predictor for that farm ACE can parse HTTP headers (includes cookies) up to 64kB (default header max parse length is 2048k) Make sure that sticky timeout (note this is more like an idle timeout) matches the session timeout on the application 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 58 Troubleshooting TCP Re-Use When using TCP connection re-use,"Connection: keep- alive" is inserted and "Connection: close" is removed from the clients HTTP request, to avoid closing the server connection early User needs to configure Source NAT in the policy map when using TCP connection re-use Use the show stats http | include Reuse counters to check if see if TCP Re-use is in effect ACE/Admin# show stats http | include Reuse Reuse msgs sent : 1 , HTTP requests : 4 sh conn detail will also show information about server side connection reuse 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 59 59 Troubleshooting HTTP Compression 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 60 HTTP Compression Overview ACE uses Cavium Octeon zip engine Implement deflate block as defined in RFC 1951 Hardware determines fixed or dynamic Huffman encoding History buffer is supported to achieve better compression ratio Support two output file formats. GZIP (RFC1952) or X- GZIP (RFC2616) and ZLIB (aka DEFLATE) RFC1950 Compression is used with HTTP connection only Compression only supports HTTP 1.1 protocol No decompression support Feature Available on ACE 4710 and ACE30 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 61 HTTP Compression Searching for cisco in www.google.com Compressed data 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 62 ACE Compression Traffic Flow Example 2. ACE rewrites Clients request GET / HTTP/1.1 Accept-Encoding: gzip, deflate 1. Request before ACE GET / HTTP/1.1 Accept-Encoding: identity Request after ACE 4. ACE Inspects response HTTP/1.1 200 OK Content-type: text/html Content-Encoding: deflate Transfer-Encoding: chunked 6. Response after ACE Server sends uncompressed HTTP payload of 5963 bytes 7. Client receives compressed HTTP payload 2577 bytes Cisco ACE Client LAN HTTP/1.1 200 OK Content-type: text/html Content-Length: 5963 3. Response before ACE 5. ACE Compresses Response Server WAN 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 63 Default Compression Controls Parameter-map type http compression Minimum content size (512 bytes) to compress compress minimum-size 100 - Compress if content length is 100 bytes or more User-Agent Exclusion (Null) compress user-agent UnknownBrowser - Disallow compression for Unknown Browser Compress only http text/* type compress mimetype image/jpeg - Compress jpeg content 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 64 Debugging HTTP Compression Check the following if there no configuration error From client side: 1.Accept-Encoding is not present or has invalid type 2.User-Agent is being excluded from the configuration 3.HTTP version is not 1.1 or higher From server side 1.Invalid HTTP response header 2.HTTP response code not 200 3.Content type is not allowed 4.Content length is too small 5.Chunk encoding has invalid format Get request from client: GET HTTP/1.1 Host: www.yahoo.com User-Agent: Mozilla/5.0 Windows; U; Windows NT 5.1; Accept: text/html,application/xhtml+xml, Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 65 Debugging HTTP Compression (Cont.) look at the stats from show np x me-stat -s http Analysis errors: 0 0 General HTTP internal error Static parse errors: 0 0 General HTTP parsing error Compression reqs sent: 0 0 Compression rsps rcvd: 0 0 Compression bytes in : 0 0 Compression bytes out: 0 0 Compression rx data in rsp wait: 0 0 Compression no paticles: 0 0 Not enough internal buffer for compressed output Compression no buffers fpa:0 0 Not enough internal buffer for hardware Compression no buffers sglist: 0 0 Not enough internal buffer for hardware Compression no buffers result zip: 0 0 Not enough internal buffer for hardware Compression session gone: 0 0 HTTP session is deleted Compression session cleaned: 0 0 Compresssion rslt non-success: 0 0 Hardware compression error Compression out alloc 0 0 Compression out dealloc 0 0 Compression chunk error 0 0 HTTP input chunk error Compression error reset 0 0 HTTP compression session reset Compression session alloc 0 0 Compression session free 0 0 Compression history set 0 0 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 66 66 Troubleshooting Secure Socket Layer (SSL) 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 67 Troubleshooting SSL Configuration of SSL on ACE is relatively simple. However if you experience an issue, how to troubleshoot? Make sure the certificate and key used in ssl-proxy are valid. Use the crypto verify command ACE/Admin# crypto verify RSA2048.key RSA2048.cert Keypair in RSA2048.key matches certificate in RSA2048.cert Check the size and location of the key. Use the show crypto key command ACE/Admin# show crypt key all Filename Bit Size Type -------- -------- ---- RSA2048.key 2048 RSA 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 68 Troubleshooting SSL Review the certificate details. Use the show crypto certificate command ACE/Admin# show crypto certificate cisco-sample-cert Certificate: Data: Version: 3 (0x2) Serial Number: ad:e4:e2:f1:50:b7:ce:bd Signature Algorithm: sha1WithRSAEncryption Issuer: C=IN, ST=KA, L=BLR, O=CISCO, OU=ADBU, CN=SSL-TEST Validity Not Before: Apr 3 09:50:55 2009 GMT Not After : Apr 1 09:50:55 2019 GMT Subject: C=IN, ST=KA, L=BLR, O=CISCO, OU=ADBU, CN=SSL-TEST Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (1024 bit) Modulus (1024 bit): 00:cf:a2:60:66:5b:ce:b6:38:6f:94:df:0d:1c:61: 26:af:7a:05:49:ed:8d:93:3b Exponent: 65537 (0x10001) 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 69 Troubleshooting SSL: CRL Download Check to make sure you can download the CRL ACE/Admin(config-ssl-proxy)# do show crypto crl test2 detail test2: URL: http://119.60.60.23/test.crl Last Downloaded (Cached): Sat Aug 8 16:14:24 2009 UTC Total Number Of Download Attempts: 1 Failed Download Attempts: 0 Successful Loads: 1 Failed Loads: 0 Hours since Last Load: 0 No IP Addr Resolutions: 0 Host Timeouts: 0 Next Update Invalid: 0 Next Update Expired: 0 Bad Signature: 0 CRL Found-Failed to load: 0 File Not Found: 0 Memory Outage failures: 0 Cache Limit failures: 0 Conn failures: 0 Internal failures: 0 Not Eligible for download: 3 HTTP Read failures: 0 HTTP Write failures: 0 Looking for all best-effort CRLs in the system and their download status. Use the show crypto crl best-effort command 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 70 Advanced SSL Debugging This command provides the current crypto statistics ACE/Admin# sh np 1 me-stats "-s crypto Crypto Statistics: (Current) ------------------ ARC4 operations: 376572 0 TCP msgs received: 285260 0 APP msgs received: 235151 0 Nitrox messages forwarded to XScale: 381041 0 SSL ctx allocated: 47758 0 SSL ctx freed: 47758 0 SSL received bytes: 61070430 0 SSL transmitted bytes: 283256220 0 SSL received application bytes: 7679113 0 SSL transmitted application bytes: 275120867 0 SSL received non-application bytes: 53391317 0 SSL transmitted non-application bytes: 3292887 0 Bulk flush operations: 95037 0 ME records sent to XScale: 285808 0 ME records received from XScale: 47723 0 ME hw responses: 471516 0 First segments received: 47400 0 Handshake failure alert: 94 0 CM close: 446 0 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 71 Advanced SSL Debugging The show stats crypto server command provides statistics of the SSL handshake ACE/Admin# show stats crypto server +---- Crypto server termination statistics -----+ +------- Crypto server alert statistics --------+ +--- Crypto server authentication statistics ---+ +------- Crypto server cipher statistics -------+ +------ Crypto server redirect statistics ------+ +---- Crypto server header insert statistics ---+ These statistics provide details of the SSL packets for example; which version client interacted with ACE, which cipher is used, whether re-handshake happened, whether session id reuse happened and which SSL alerts are received or sent by ACE 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 72 72 Health Monitoring on ACE 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 73 Fundamentals for ACE Probing ACE probes are fundamental to the system. It is key to not oversubscribe the ACE health monitoring system Use the show resource internal socket to determine how many sockets ACE has open. This is an Admin command ACE/Admin# show resource internal socket Application MaxLimit Current Creates Frees -------------------------------------------------------------- SYSTEM 4000 0 0 0 CRITICAL 50 0 0 0 AAA 256 0 0 0 MGMT 256 0 0 0 XINETD 512 1 12 11 HEALTH_MON 2500 532 193494 192962 USER_TCL 200 0 0 0 SYSLOG 256 10 14 4 VSH 256 0 0 0 OverAll - 650 194812 194162 Non Reg App Usage: 107 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 74 Health Monitoring Process If you see probing issues, check the health monitoring process. The show proc cpu command provides very useful information ACE/Admin# show proc cpu CPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35% PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process 972 1072965 613352 1749 35.9 18.5% 21.67% 20.90% arp_mgr HM process is only consuming 1.40%. Why is the control plane CPU running at 30%? Check what process is consuming CPU ACE/Admin# show proc cpu | inc hm CPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35% PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process 987 90257 57805 1561 0.0 1.40% 1.46% 1.43% hm 988 90198 58952 1530 0.0 1.49% 1.49% 1.44% hm 989 851 2947 288 0.0 0.0 % 0.1 % 0.0 % hm 990 0 2 56 0.0 0.0 % 0.0 % 0.0 % hm 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 75 Health Monitoring on ACE Use the show probe detail command to determine the status of the probe or possible last failure ACE/Admin# show probe detail Cut output --------------------- probe results -------------------- probe association probed-address probes failed passed health ------------------- ---------------+----------+----------+----------+------- rserver : CAS1 10.7.53.55 24 24 0 FAILED Socket state : CLOSED No. Passed states : 0 No. Failed states : 1 No. Probes skipped : 0 Last status code : 403 No. Out of Sockets : 0 No. Internal error: 0 Last disconnect err : Received invalid status code Last probe time : Wed Nov 25 18:48:16 2009 Last fail time : Wed Nov 25 18:25:16 2009 Last active time : Never 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 76 76 High Availability on ACE 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 77 High Availability Basic Building Blocks FT PEER Only one FT peer per ACE device 1:1 peer relationship FT GROUP One FT group per ACE virtual context FT VLAN Designated VLAN between the redundant peers All HA related traffic sent over this VLAN FT VLAN can be trunked between two Catalyst 6500 Chassis Should not be used for normal traffic Admin Context Context A Context B Context A Context B ACE2 (FT PEER) FT VLAN FT Group 2 FT Group 3 ACE1 (FT PEER) FT Group 1 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 78 High Availability Control Traffic TCP Connection between FT Peers State Machine (Election, Preempt, Relinquish) Configuration sync State Sync for ARP Heartbeats between FT peers Heartbeats are sent over UDP Monitors the health of the peer Heartbeat interval and count are configurable 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 79 ACE High Availability State Machine Active/Standby Election (assuming both peers are initialized at same time) Based on a priority scheme Member with highest priority becomes ACTIVE Other member enters the STANDBY_CONFIG state If priorities are equal, member with the higher IP address wins STANDBY_CONFIG State Startup Configuration Sync from Active to Standby Running Configuration Sync from Active to Standby Knob to turn on/off 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 80 ACE High Availability State Machine STANDBY_BULK State ARP Sync (knob to turn on/off) Connection Table Sync Sticky Database Sync (knob to turn on/off) STANDBY_HOT State Standby FT group member is ready to take over Incremental Configuration Sync from Active to Standy Incremental State Sync from Active to Standby STANDBY_COLD State Due to error during Config Sync or Incremental Config Sync No Config or State Sync happens from Active to Standby STANDBY_WARM State Major version mismatch between peers (example 2.x and 4.x) 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 81 ACE High Availability State Machine Mismatch in software version FT Peer may become INCOMPATIBLE ACTIVE ACTIVE state on both FT group members Mismatch in Virtual Context Licenses Configuration Sync (all types) for Admin context is disabled State Sync for Admin context will continue to happen For matching user contexts Configuration State Sync will work Mismatch in Other Licenses Configuration and State Sync will work After switchover, new Active will handle traffic as per its licenses 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 82 ACE Redundancy Query VLAN When no heartbeat is received, ACE can use the Query Vlan to check the HA status ACE tries to do a ping to the destination via the Query VLAN If ping fails, the Standby will transition to the ACTIVE state If ping succeeds, the Standby will transition to a STANDBY_COLD state To configure a query interface, enter the following: ACE/Admin(config-ft-peer)# query-interface vlan 110 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 83 83 More Debugging Commands 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 84 Additional Debugging Some more ACE debugging commands show np <#> me-stats -cpu show np <#> me-stats Q show np <#> me-stats -s fp show np <#> me-stats -s tcp show np <#> me-stats -s icm show np <#> me-stats -s ocm show proc cpu show netio stats Show service-policy summary 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 85 Recommended Reading 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 86 BRKAPP-3003 Complete Your Online Session Evaluation Receive 25 Cisco Preferred Access points for each session evaluation you complete. Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Dont forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com. 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 87 BRKAPP-3003 87 Visit the Cisco Store for Related Titles http://theciscostores.com 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 88 BRKAPP-3003 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 89 BRKAPP-3003 Thank you. 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 90 90 Appendix and Additional Troubleshooting Information 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 91 Additional Information Layer 4 flow setup Layer 7 flow setup TCP Connection States 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 92 Layer 4 Flow Setup SYN SYN_ACK Shortcut ACK Shortcut Data Shortcut Data Shortcut Matches Existing Flow Rewrites L2/L3/L4 Matches VIP Selects Server Rewrites L2/L3/L4 Basic Load Balancing Source IP sticky TCP/IP Normalization 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 93 Layer 7 Flow Setup Client Connects to L7 VIP SYN Starts Buffering ACK Data ACKs Client Packets Keeps Buffering Matches VIP w/L7 logic Chooses SEQ # Replies w/SYN_ACK HTTP L7 rules on first request (cookie sticky, URL parsing, ) Generic TCP payload parsing 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 94 Layer 7 Flow Setup (Cont.) ACE Establishes Connection to Server Data SYN_ACK Empties Buffer Sends Data to Server Acts as Client Does Not Forward SYN_ACK Parses the Data Selects Server Initiates TCP 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 95 Layer 7 Flow Setup (Cont.) ACE Splices the Flows (UNPROXY) ACK Data Shortcut ACK Shortcut Data Shortcut Matches Existing Flow Rewrites L2/L3/L4 and SEQ/ACK Does Not Forward ACK Ready to Splice the Flows 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 96 Layer 7 Flow Setup ACE Reproxies the Connection ACK Data ACK Data Shortcut
ACK
Shortcut Shortcut Shortcut Data REPROXY ACKs GET & Buffer
HTTP L7 rules with HTTP 1.1
connection keepalive (persistence rebalance) 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 97 Layer 7 Flow Setup ACE Acts as a Full Proxy F u l l
P r o x y I n d e p e n d e n t
c l i e n t
&
s e r v e r
c o n n e c t i o n s SYN SYN_ACK ACK Data GET/HTTP 1.1 ACK SYN SYN_ACK ACK DataGET ACK ACK Data Data HTTP/1.1 200 OK HTTP/1.1 200 OK Client connection Server connection
SSL offload TCP re-use Protocol inspections HTTP 1.1 pipelining 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKAPP-3003 98 TCP Connection States L4 TCP Connections SYNSEEN (Client SYN received) INIT (Server side half flow initialized) SYNACK (SYN ACK sent by server) ESTAB (Client and Server; TCP Handshake completed) L7 TCP Connections SYNSEEN (Client SYN received) ESTAB (Client side TCP Handshake completed; SYN ACK sent by ACE, Client ACK received) ESTAB (Server side TCP Handshake completed from ACE after L7 data received from the client and parsed) CLOSED (Client or Server FIN ACK followed by ACK)