KEMBAR78
BRKRST-3068 Troubleshooting Catalyst 2K and 3K.pdf
Uploaded byssusercbaa33
PDF, PPTX66 views

BRKRST-3068 Troubleshooting Catalyst 2K and 3K.pdf

This document provides an overview of troubleshooting Catalyst 2K and 3K switches. It discusses monitoring system resources like CPU usage, port ASICs, memory and TCAM. Common issues covered include link problems, hardware failures, and high CPU. A variety of show commands are provided to monitor resources, analyze errors and diagnose potential causes. The goal is to identify and solve access layer incidents with confidence.

Download as PDF, PPTX
1 / 109
2 / 109
3 / 109
4 / 109
5 / 109
6 / 109
7 / 109
8 / 109
9 / 109
10 / 109
11 / 109
12 / 109
13 / 109
14 / 109
15 / 109
16 / 109
17 / 109
18 / 109
19 / 109
20 / 109
21 / 109
22 / 109
23 / 109
24 / 109
25 / 109
26 / 109
27 / 109
28 / 109
29 / 109
30 / 109
31 / 109
32 / 109
33 / 109
34 / 109
35 / 109
36 / 109
37 / 109
38 / 109
39 / 109
40 / 109
41 / 109
42 / 109
43 / 109
44 / 109
45 / 109
46 / 109
47 / 109
48 / 109
49 / 109
50 / 109
51 / 109
52 / 109
53 / 109
54 / 109
55 / 109
56 / 109
57 / 109
58 / 109
59 / 109
60 / 109
61 / 109
62 / 109
63 / 109
64 / 109
65 / 109
66 / 109
67 / 109
68 / 109
69 / 109
70 / 109
71 / 109
72 / 109
73 / 109
74 / 109
75 / 109
76 / 109
77 / 109
78 / 109
79 / 109
80 / 109
81 / 109
82 / 109
83 / 109
84 / 109
85 / 109
86 / 109
87 / 109
88 / 109
89 / 109
90 / 109
91 / 109
92 / 109
93 / 109
94 / 109
95 / 109
96 / 109
97 / 109
98 / 109
99 / 109
100 / 109
101 / 109
102 / 109
103 / 109
104 / 109
105 / 109
106 / 109
107 / 109
108 / 109
109 / 109
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Catalyst 2K and 3K BRKRST-3068
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Session Goals  Identify various system resources and monitor their usage.  Select the right steps to solve common access layer incidents.  Diagnose a former black-box with confidence. 3
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Agenda  Architecture Overview  Areas of Troubleshooting Hardware components Forwarding 3750 3560 2960 4
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Catalyst 2K/3K Architecture Overview:  Port ASICs, CPU, Memory, Stack Phy, TCAM, Switch Fabric  The number of interfaces per Port ASIC varies by platform. Memory CPU Stack PHY Port ASIC Port ASIC Port ASIC Switch Fabric Stack Phy Flash Serial Modular PHY 10/100 10G or 1G 12 Port PHY 12 Port PHY 12 Port PHY 12X1G 12X1G 24X1G POE 12 Port PHY 12X1G 12X1G 24X1G POE Two Stack Cables TCAM SRAM TCAM SRAM TCAM SRAM Stack errors High? Buffers? QoS TCAM usage Running out? Interface Flaps? 5
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Agenda  Architecture Overview  Areas of Troubleshooting Hardware components Forwarding 3750 3560 2960 6
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Link Issues  Is the link coming up as expected?  Are packets being sent and received on the port?  Are there errors on the port?  Link Issue: Failure for the physical interface to remain operational  Can be caused by a Layer 1 or Layer 2 problem  Layer 1: Interface PHY does media conversion (10/100/1000Mbps, 10G)  Layer 2: Interface PHY ensures 802.3 compliance 7
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Port Status and Counters 3700X-1#sh int gig 1/0/1 GigabitEthernet1/0/1 is down, line protocol is down (notconnect) Hardware is Gigabit Ethernet, address is 70ca.9b2d.9f81 (bia 70ca.9b2d.9f81) MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX input flow-control is off, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output never, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec Traditional interface level statistics command 8
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Link Not Coming Up  Verify the configured duplex and speed on both switch and attached host; fixing speed and duplex should be done on both sides  Upgrade the NIC drivers on the host to the latest version available from the vendor  Try a different cable/NIC and switchport to exclude faulty hardware Switch#show interfaces status | inc connected Gi1/0/1 connected trunk a-full 10 10/100/1000BaseTX Gi1/0/2 connected 101 a-full a-100 10/100/1000BaseTX Gi1/0/24 connected 1 a-full a-1000 10/100/1000BaseTX 9
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public SFP Link Issue Prevention EEM Tcl Policy Use Case  1000 Base-T (copper) and 100 Base-FX SFPs have embedded PHYs, allowing speed and duplex to be configured on their respective interface  However, these settings are cleared as soon as the SFP is unplugged  If the same SFP type is re-inserted, its configuration is not recovered LinkUpApplyConfig.tcl can be downloaded at the following hyperlink: https://supportforums.cisco.com/docs/DOC-23267  LinkUpApplyConfig is a Tcl policy that monitors an SFP link-up event  Speed and duplex settings (in startup-config) are automatically re- applied to the SFP interface 1 2 1 2 EEM Tcl Policy 10
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Checking Physical Cabling  Use the TDR feature on the port to determine possible cabling issues: miswiring or cable breaks  Interfaces will be brought down and up when run on active ports Switch# test cable-diagnostics tdr interface GigabitEthernet4/0/1 TDR test started on interface Gi4/0/1 A TDR test can take a few seconds to run on an interface Use 'show cable-diagnostics tdr' to read the TDR results. Switch# show cable-diagnostics tdr interface GigabitEthernet4/0/1 TDR test last run on: March 01 03:11:11 Interface Speed Local pair Pair length Remote pair Pair status --------- ----- ---------- ------------------ ----------- -------------------- Gi4/0/1 1000M Pair A 3 +/- 1 meters Pair A Normal Pair B 2 +/- 1 meters Pair B Normal Pair C 3 +/- 1 meters Pair C Normal Pair D 3 +/- 1 meters Pair D Normal 11 Open?
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public *Mar 1 05:19:41.222: %SFF8472-5-THRESHOLD_VIOLATION: Gi4/1/1: Rx power high warning; Operating value: -4.2 dBm, Threshold value: -7.0 dBm. Link Issues: Checking Physical Cabling  Use the DOM feature on the link ports to determine possible optical transmission issues: dirty connectors, wrong attenuation, transceiver issues 12 Gi4/1/1 Gi1/1/1 Switch# show interface GigabitEthernet4/1/1 transceiver details Optical High Alarm High Warn Low Warn Low Alarm Receive Power Threshold Threshold Threshold Threshold Port (dBm) (dBm) (dBm) (dBm) (dBm) ------- ----------------- ---------- --------- --------- --------- Gi4/1/1 -4.2 + -3.0 -7.0 -27.9 -32.2 Switch# show interface GigabitEthernet1/1/1 transceiver ITU Channel not available (Wavelength not available), Transceiver is internally calibrated. ++ : high alarm, + : high warning, - : low warning, -- : low alarm. NA or N/A: not applicable, Tx: transmit, Rx: receive. mA: milliamperes, dBm: decibels (milliwatts). Optical Optical Temperature Voltage Tx Power Rx Power Port (Celsius) (Volts) (dBm) (dBm) --------- ----------- ------- -------- -------- Gi1/1/1 38.6 3.29 -2.0 -11.3
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: MACsec Encryption Switch2# show macsec GigabitEthernet 2/1/4 MACsec is enabled Replay protect : enabled Replay window : 0 Include SCI : yes Cipher : GCM-AES-128 Confidentiality Offset : 0 Capabilities Max. Rx SA : 16 Max. Tx SA : 16 Validate Frames : strict PN threshold notification support : Yes Ciphers supported : GCM-AES-128 Transmit Secure Channels SCI : D0D0FDA045830000 Elapsed time : 00:43:26 SC Statistics Auth-only (0 / 0) Encrypt (157426 / 0) Receive Secure Channels SCI : EC4476E280950000 Elapsed time : 00:43:26 Current AN: 1 Previous AN: 0 Check Details on Encryption Tunnel negotiation on both link ends Switch1# show macsec GigabitEthernet 1/0/1 MACsec is enabled Replay protect : enabled Replay window : 0 Include SCI : yes Cipher : GCM-AES-128 Confidentiality Offset : 0 Capabilities Max. Rx SA : 16 Max. Tx SA : 16 Validate Frames : strict PN threshold notification support : Yes Ciphers supported : GCM-AES-128 Transmit Secure Channels SCI : EC4476E280950000 Elapsed time : 00:42:36 SC Statistics Auth-only (0 / 0) Encrypt (10207 / 0) Receive Secure Channels SCI : D0D0FDA045830000 Elapsed time : 00:42:36 Current AN: 1 Previous AN: 0 13 Symptom: Link not coming up
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: MACsec Encryption IEEE 802.1ae Statistics computed by the MACsec PHY Switch# show macsec GigabitEthernet 1/0/1 ... SC Statistics Notvalid pkts 0 Invalid pkts 0 Valid pkts 193333 Late pkts 0 Uncheck pkts 0 Delay pkts 0 Port Statistics Ingress untag pkts 0 Ingress notag pkts 343 Ingress badtag pkts 0 Ingress unknownSCI pkts 0 Ingress noSCI pkts 0 Unused pkts 0 Notusing pkts 0 Decrypt bytes 20486316 Ingress miss pkts 79 14
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Link Issues Troubleshooting Steps Commands Cabling issues test cable-diagnostics tdr interface show interface transceiver Interface not coming up show interface status show interface .. counters errors show macsec interface Command Summary 15
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch Hardware Components: CPU Functions  Runs the IOS  Processes Control Plane traffic (LACP / PAgP / VTP / STP / CDP)  Processes packets that are not switched in Hardware Packets with IP options, Packets with expired TTL, ARP, Snooping, Software ACLs and SNMP Memory CPU Stack PHY Port ASIC Switch Fabric 10G or 1G TCAM High? 18
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  CPU Utilisation can become high due to 2 reasons: • Processes taking up resources • Forwarded Network Traffic  Using CPU cycles is not a problem  6-8% is minimum - depending upon IOS Feature set  Normal or Expected CPU Utilisation 10-12% 3700X-1#sh proc cpu sor CPU utilization for five seconds: 46%/0%; one minute: 11%; five minutes: 9% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 337 5365 127 42244 38.17% 3.11% 0.64% 1 Virtual Exec 75 129004 32458 3974 1.55% 1.69% 1.61% 0 RedEarth Tx Mana 39 3741 142 26345 0.62% 0.06% 0.01% 0 Per-minute Jobs 118 7011 12552 558 0.31% 0.10% 0.07% 0 hpm counter proc 169 13991 1706 8201 0.15% 0.16% 0.15% 0 HQM Stack Proces CPU: Troubleshooting Processes Switch# sh proc cpu history 70 * * 60 ** * * 50 ** * * 40 *** * * * 30 ***** ** * 20 **##*#* ** # 10 ################## 0....5....1....1....2....2....3....3....4....4....5....5.... 0 5 0 5 0 5 0 5 0 5 19
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public packets dropped before reaching the CPU Queue Switch# show platform port-asic stats drop Supervisor TxQueue Drop Statistics Queue 0: 0 ...... Queue 7: 10000 CPU: The 16 Different Queues  CPU buffer pools are named RxQ0 to RxQ15  Port ASIC can drop packets before reaching the CPU Q Check both locations (pools and asic queues) 0:rpc 1:stp 2:ipc 3:routing protocol 4:L2 protocol 5:remote console 6:sw forwarding 7:host 8:broadcast 9:cbt-to-spt 10:igmp snooping 11:icmp 12:logging 13:rpf-fail 14:dstats 15:cpu heartbeat 20
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch# debug platform cpu-queues software-fwd-q *Mar 1 10:37:33.205 AEDT: SW-FWD-Q:IP packet: Local Port Fwding L3If:Vlan1 L2If:GigabitEthernet2/0/2 DI:0x2F, LT:7, Vlan:1 SrcGPN:56, SrcGID:56, ACLLogIdx:0x0, MacDA:c471.fe1e.f0c0, MacSA: 0007.7d75.88c0 IP_SA:14.160.38.1 IP_DA:14.160.38.130 IP_Proto:1 IP Opts TPFFD:D8C00038_00010001_00A00076-0000002F_E2C50000_00000000 CPU: Software Forwarding Queue (Q6)  For Traffic that hardware cannot process  SW forwarding performance is much lower than HW To debug any CPU Q SMAC of the host sending the traffic Physical interface where the traffic is coming in 21
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: Best Practices Storm Control can help to protect CPU. Configuring Traffic Storm Control to avoid packets flood the LAN, creating excessive traffic and degrading network performance. (config-if)#storm-control broadcast level level[.level] (config-if)#storm-control action ? shutdown Shutdown this interface if a storm occurs trap Send SNMP trap if a storm occurs Example of Syslog message for high CPU *Mar 1 01:03:15.601: %SYS-1-CPURISINGTHRESHOLD: Threshold: Process CPU Utilisation (Total/Intr): 18%/0%, Top 3 processes(Pid/Util): 4/10%, 75/1%, 164/0% Configuring the CPU threshold can help you identifying when the CPU goes over certain limits Switch(config)# process cpu threshold type {total | process | interrupt} rising percentage interval seconds [falling fall-percentage interval seconds] 22
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public %PLATFORM_RPC-0-RESOURCE_CRASH: System is unable to alloc memory for RPC TS: Memory Utilisation  Potential issues • Is Free steady? • Is Free steadily decreasing? Switch# sh memory statistics Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 2641D6C 81519252 31192204 50327048 49241540 48621848 I/O 7400000 12574720 8532852 4041868 3821068 4039616 Memory available now The lowest free since boot up Largest block switch can allocate FIB−2−FIBDOWN : CEF has been disabled due to a low memory condition. %SYS−2−MALLOCFAIL: Memory allocation of 1028 bytes failed from 0x601617A4, pool Processor, alignment 0 −Process= "IP Input", ipl= 2, pid= 21 %FRNTEND_CTRLR-1-SUB_I2C_ERR: Sub 0 reported 36B5B98 I2C errors %FRNTEND_CTRLR-2-SUB_INACTIVE: The front end controller 0 is inactive 23
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public TS: Memory Utilisation  Run commands multiple times to create benchmark Switch# show processes memory sorted … PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 74539888 23738156 47199076 0 0 *Init* 0 0 3399716 17490880 1590292 10657136 553112 *Dead* 65 0 712620 27424 594488 0 0 Stack Mgr Notifi 324 0 19794764 19262624 539264 0 0 hulc running con 304 0 366680 344 370420 0 0 CEF: IPv4 proces 165 0 294516 2524 294516 0 0 HL2MCM 164 0 294460 2496 294460 0 0 HL2MCM 17 0 230568 0 240620 99792 0 EEM ED Syslog 11 0 228060 14940 226488 0 0 ARP Input … Is any process steadily increasing held memory? 24
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public TCAM Utilisation  TCAM space is limited  Problem when Used Masks/Values = MAX Change SDM Template/optimise ACLs, Routing entries. Security ACLs Permit/d eny Layer 3 routing Switch# show platform tcam utilization CAM Utilization for ASIC# 0 Max Used Masks/Values Masks/values Unicast mac addresses: 784/6272 14/40 IPv4 IGMP groups + multicast routes: 144/1152 7/27 IPv4 unicast directly-connected routes: 784/6272 14/40 IPv4 unicast indirectly-connected routes: 2048/2048 2047/2047 IPv4 policy based routing aces: 0/0 0/0 IPv4 qos aces: 768/768 260/260 IPv4 security aces: 1024/1024 723/723 25
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public TCAM Overload  An error message will get generated  Traffic forwarding will be done (partly) in Software  CPU utilisation will go up – packets punted to CPU for processing %ACLMGR-4-UNLOADING: Unloading ACL input label 1 VLAN interfaces 101 IPv4/Mac feature %ACLMGR-4-ACLTCAMFULL: ACL TCAM Full. Software Forwarding packets on Input label 1 on L3 L2 Switch# sh platform acl label 1 detail Unloaded due to lack of space: Switch# sh platform acl oacltcamfull Vlan oacl_tcam_full_bitmap notify_apps 101 0x 0 NOT-FULL Means ACL not fully programmed in TCAM 26
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public TCAM: Switch Database Manager (SDM)  SDM defines how TCAM resources are allocated  Changing SDM template requires reboot  All stack members must use same SDM template Switch# show sdm prefer default "desktop default" template: The selected template optimizes the resources in the switch to support this level of features for 8 routed interfaces and 1024 VLANs. number of unicast mac addresses: 6K number of IPv4 IGMP groups + multicast routes: 1K number of IPv4 unicast routes: 8K number of directly-connected IPv4 hosts: 6K number of indirect IPv4 routes: 2K number of IPv4 policy based routing aces: 0 number of IPv4/MAC qos aces: 0.5K number of IPv4/MAC security aces: 1K 27
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting CPU/Memory Troubleshooting Steps Commands Current CPU Utilisation show processes cpu sorted show processes cpu history Statistics for Packets Fwd to CPU show platform port-asic stats drop show controllers cpu-interface Details of packets received by CPU per ingress queue debug platform cpu-queues <queue> Memory Issues Show memory <> Show processes memory <> Command Summary 28
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting TCAM Troubleshooting Steps Commands Utilisation show platform tcam utilization Check HW resource show platform acl oacltcamfull show platform acl label <> detail SDM Template show sdm prefer Command Summary 29
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Stack# show switch H/W Current Switch# Role Mac Address Priority Version State ---------------------------------------------------------- *1 Master 0018.ba60.de00 15 1 Ready 2 Member 0018.ba60.ce00 14 1 Ready 3 Member 0016.9d0c.7500 1 2 Version Mismatch Troubleshooting Stacks  Conditions that can prevent a switch from joining a stack: • Incompatible IOS Versions between the stack members. • A defective stack cable • Not properly connected • Incomplete connection if only one stack cable is connected. • SDM Template mismatch  The following example shows a switch that can not join the stack: %STACKMGR−6−SWITCH_ADDED_VM 30
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Stacks Version Mismatch  Software Version Mismatch ‒IOS version should be either the same or compatible ‒show version will show IOS version of all switches in a stack.  Switches with different Major Version numbers are incompatible and cannot exist in the same switch stack. ‒Occurs on switch member addition, or RMA replacement IOS Versions should match Major versions must match 3750E# show version Switch Ports Model SW Version SW Image ------ ----- ----- ---------- ---------- * 1 30 WS-C3750E-24TD 12.2(50)SE C3750E-UNIVERSAL-M 3 30 WS-C3750E-24PD 12.2(50)SE C3750E-UNIVERSAL-M 3750E# show platform stack manager all … H/W Current Switch# Role Mac Address Priority Version State ---------------------------------------------------------- *1 Master 001b.545f.2800 12 1 Ready 3 Member 001d.46be.7500 8 1 Ready 31
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting: Stack Commands 3750# show switch detail Current Switch# Role Mac Address Priority State ------------------------------------------------------ 1 Slave 000c.30ae.4f00 9 Ready *2 Master 000d.bd5c.1680 15 Ready Stack Port Status Neighbors Switch# Port 1 Port 2 Port 1 Port 2 ------------------------------------------------------ 1 Ok Ok 2 2 2 Ok Ok 1 1 3750# show switch stack-ring activity Switch Frames sent to stack ring (approximate) ------------------------------------------------ 1 5781 2 4928 Total frames sent to stack ring : 10709 Note: these counts do not include frames sent to the ring by certain output features such as output SPAN and output ACLs. Commands to give stack details  Use the mode button on the switch to determine its stack switch number  LED on the port with the corresponding switch number will illuminate  For ex, if the switch is # 4 in the stack, port 4’s LED will light up 3750E# show switch stack-ring speed Stack Ring Speed : 32G Stack Ring Configuration: Full Stack Ring Protocol : StackWisePlus 32
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting: Stack Commands 3750# sh switch Switch/Stack Mac Address : c471.fe1e.f080 H/W Current Switch# Role Mac Address Priority Version State ---------------------------------------------------------- 1 Member c471.fe23.3780 1 1 Ready *2 Master c471.fe1e.f080 1 1 Ready 3 Member 0000.0000.0000 0 1 Provisioned 3750# show switch stack-ports summary Switch#/ Stack Neighbor Cable Link Link Sync # In Port# Port Length OK Active OK Changes Loopback Status To LinkOK -------- ------ -------- -------- ---- ------ ---- --------- -------- 1/1 OK 3 50 cm Yes Yes Yes 1 No 1/2 Down None 50 cm No No No 0 No 3/1 Down None 50 cm No No No 0 No 3/2 OK 1 50 cm Yes Yes Yes 1 No Details on the stack ports, members 1 and 2 active 33
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Stacking Troubleshooting Steps Commands Stack status show switch [detail] show platform stack manager show switch stack-ring <> show controllers utilization show switch stack-ports summary (New) Test Stack Ports switch <> stack port <> enable/disable From IOS 12.2(50) Command Summary 34
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public GOLD (Generic Online Diagnostics) (config)# [no] diagnostic monitor interval { switch <1-9> } test { test-id | test-id-range | all } hh:mm:ss { ms <0-999> } { days <0-20> } diagnostic start {switch <1:9>} test {test-num | test range | all | basic | non-disruptive } Switch(config)#[no] diagnostic schedule { switch <1-9> } test { test-id | test-id-range | all } daily {hh:mm} On-Demand Health-Monitoring Scheduled To run Non-disruptive tests in the background Serves as HA trigger All diagnostics tests can be run on demand, for troubleshooting purposes. It can also be used as a pre-deployment tool. All diagnostic tests can be Scheduled, for verification and troubleshooting purposes Runtime diagnostics Run During System Bootup, Makes sure faulty hardware is taken out of service (POST = Power On Self Test) Boot-Up diagnostics show diagnostic post 35
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public GOLD: Test Options OnDemand 3750E# show diagnostic content switch 1 Test Interval ID Test Name Attributes day hh:mm:ss.ms Threshold ==== ====================== ============ ========== ==== ======== 1) TestPortASICStackPortLoopback ---> B*N****I** 005 01:10:25.05 n/a 2) TestPortASICLoopback ----------------> B*D*X**IR* not configured n/a 3) TestPortASICCam -----------------------> B*D*X**IR* not configured n/a 4) TestPortASICRingLoopback ----------> B*D*X**IR* not configured n/a 5) TestMicRingLoopback ----------------> B*D*X**IR* not configured n/a 6) TestPortASICMem ----------------------> B*D*X**IR* not configured n/a 7) TestInlinePwrCtlr -----------------------> B*D*X**IR* not configured n/a What Tests Can I Run? 36
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public GOLD: CLI OnDemand diagnostic start {switch <1:9>} test {test-num | test range | all | basic | non-disruptive } 3700X-1#diagnostic start switch 1 test all Diagnostic[Switch 1]: Running test(s) 2-6 will cause the switch under test to reload after completion of the test list. Diagnostic[Switch 1]: Running test(s) 2-6 may disrupt normal system operation Do you want to continue? [no]: yes Disruptive Test: Users will be prompted if the test causes a lose of stack connectivity: Switch 3: Running test(s) 2 will cause the switch under test to reload after completion of the test list. Switch 3: Running test(s) 2 may disrupt normal system operation Do you want to continue? [no]: Disruptive Test: Users will be prompted if the test causes stack partitioning: Switch 6: Running test(s) 2 will cause the switch under test to reload after completion of the test list. Switch 6: Running test(s) 2 will partition stack Switch 6: Running test(s) 2 may disrupt normal system operation Do you want to continue? [no]: Note: Tests Run to Completion (No Stop Command) 37
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public GOLD: Results OnDemand 3750X# show diagnostic status shows what diagnostics are currently running 3750X# show diagnostic result switch 1 detail … Overall diagnostic result: PASS … 1) TestPortASICStackPortLoopback ---> . Error code ----------------------> 0 (DIAG_SUCCESS) … Last test execution time ----> Mar 01 1993 00:16:34 … Last test pass time ---------> Mar 01 1993 00:16:34 … 38
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public On-Board Failure Logging (OBFL) 3750E/3560E  Provides “flight recorder” capability  Enabled by default  Collects operational data about switch and FRUs like PS, RPS & SFPs  Stores the data as a circular buffer on flash (2M). Older data is compressed with less detail  Each switch records its own OBFL data  Information can be seen with show logging onboard 39
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Agenda  Architecture Overview  Areas of Troubleshooting Hardware components Forwarding 3750 3560 2960 40
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  The normal L2 CAM behaviour in a switch is as follows… Learning Switch CAM Table MAC A C Port 1 3 A B C 1 2 3 A>B A>B A>B Source Mac address of frame are used to learn mac-address location and build L2 table – Learning A on port 1 Destination Mac address of frame are used as lookup key in mac-address-table to know where to forward. A destination mac address miss will cause Unicast flooding B is unknown hence packet is flooded 41
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  The normal CAM behaviour in a switch is as follows… Learning … and Aging Switch CAM Table MAC A B C Port 1 2 3 A B C 1 2 3 B>A B>A Reply from B is coming back to A lead to learning B in Cam table A is already known hence sent only to destination port. Each entry is aged out when inactive for 5 min (default) 42
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Unicast Forwarding  Symptom: Host cannot reach server  Steps Layer 1 operational between host and switch? Switch receiving traffic on that interface? MAC address learned? MAC address of next hop correct? Spanning tree state forwarding? Check HW programming  Consider possibilities  Create and execute action plan 43
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  Step 1: Verify if the link is up  Step 2: Verify if the port is in the right vlan and is forwarding  Step 3: Check if the packets are being received/sent on the port Switch# show spanning-tree interface Gi1/0/3 Vlan Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- ----------------------- VLAN010 Desg FWD 19 128.2 P2p L2 Forwarding: Troubleshooting - 1 Switch# show interface Gi1/0/3 status Port Name Status Vlan Duplex Speed Type Gi1/0/3 connected 10 a-full a-100 10/100/ 1000BaseTX Switch# show interfaces gigabitEthernet 1/0/3 counters Port InOctets InUcastPkts InMcastPkts InBcastPkts Gi1/0/3 2108289 48 0 6813 Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts Gi1/0/3 36817803 48229 252940 72564 44
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  Step 4a: Verify if the Mac-address is correctly learned on the port  Step 4b: Verify if the destination Mac-address is learned on the switch on the expected port Switch# sh mac address-table interface gigabitEthernet 1/0/3 Mac Address Table ------------------------------------------- Vlan Mac Address Type Ports ---- ----------- -------- ----- 10 00b1.a3d3.4321 DYNAMIC Gi1/0/3 Total Mac Addresses for this criterion: 1 Layer 2 Forwarding: MAC Checking Switch# sh mac address-table dynamic address 00b1.a3d3.1234 Mac Address Table ------------------------------------------- Vlan Mac Address Type Ports ---- ----------- -------- ----- 10 00b1.a3d3.1234 DYNAMIC Gi1/0/4 Total Mac Addresses for this criterion: 1 45
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch#show spanning-tree vlan 10 VLAN0010 Spanning tree enabled protocol ieee Root ID Priority 32778 Address 0003.fd6b.0700 This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32778 (priority 32768 sys-id-ext 10) Address 0003.fd6b.0700 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Aging Time 300 sec Interface Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- ------------------------ Gi1/0/3 Desg FWD 4 128.3 P2p Gi1/0/4 Desg FWD 4 128.4 P2p Edge Layer 2 Forwarding: Spanning Tree  Step 5: Spanning tree state forwarding in software? Interfaces are Forwarding 46
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Layer 2 Forwarding: Advanced Techniques  Step 6: Use show platform forward to verify the hardware programming – find Egress Interface Destination Interface Incoming interface src mac dst mac Switch# show platform forward gigabitEthernet 1/0/3 00b1.a3d3.4321 00b1.a3d3.1234 Ingress: Global Port Number: 3, lpn: 1 ASIC Number: 6 Source Vlan Id: Real 10, Mapped 2. L2EncapType 0, L3EncapType 3 Hashes: L2Src 0x00 L2Dst 0x0B L3Src 0x00 L3Dst 0x0B Lookup Key-Used Index-Hit A-Data … ========================================== Egress: ASIC 6, switch 1 Source Vlan Id: Real 10, Mapped 2. L2EncapType 0, L3EncapType 3 portMap 0x4, non-SPAN portMap 0x4 Output Packets: ------------------------------------------ GigabitEthernet1/0/4 Packet 1 Lookup Key-Used Index-Hit A-Data OutptACL 30_00F00000_00001234-00_00000000_00004321 01FFC 01000000 Port Vlan SrcMac DstMac Cos Dscpv Gi1/0/4 0010 00b1.a3d3.4321 00b1.a3d3.1234 47
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Unicast Forwarding Troubleshooting Steps Commands Verify Layer 1 is operational between host and switch show interface <interface> status Verify switch receives traffic on the interface show interfaces <interface> counters show interfaces <interface> counters errors MAC Address changes show spanning-tree vlan <> detail 48
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Steps Commands Verify host MAC address is learned show mac address-table interface <interface> show mac address-table dynamic address <mac> Verify spanning tree state is forwarding show spanning-tree vlan <vlan> Verify MAC address of next hop is correct Local and remote switches: show mac address-table vlan <vlan> Verify other features are not preventing traffic flow Show port-security interface <interface> show ip access-lists interface <interface> Show hardware programming for MAC Address show platform forward <src interface> <src-mac> <dest-mac> Troubleshooting Unicast Forwarding 49
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Layer 3 IP Unicast Routing  Use the switch to debug end to end IP issues  Verify IP reachability from switch end host  Verify destination reachability from the switch  Verify hardware forwarding from source to destination (and back) 3750 370 Source IP: 100.1.1.2 Mac: 0018.ba88.1fc1 Gi1/0/1 Gi1/0/2 Destination IP: 172.16.100.100 VLAN:101 IP: 100.1.1.1 Mac: 000f.f7e8.e042 Vlan:100 IP: 10.1.1.1 Mac :000f.f7e8.e041 3750 3750 50
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public L3: Verify Source Reachability 3750# ping 100.1.1.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 100.1.1.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms 3750# ping 100.1.1.2 source lo0 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 100.1.1.2, timeout is 2 seconds: Packet sent with a source address of 99.1.1.1 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/maz = 1/4/9 ms 3750# sh ip arp vlan 101 Protocol Address Age (min) Hardware Addr Type Interface Internet 100.1.1.1 - 000f.f7e8.e042 ARPA Vlan101 Internet 100.1.1.2 23 0018.ba88.1fc1 ARPA Vlan101 3750# sh mac address-table address 0018.ba88.1fc1 Mac Address Table ------------------------------------------ Vlan Mac Address Type Ports ---- ----------- -------- ----- 101 0018.ba88.1fc1 DYNAMIC Gi1/0/2 Total Mac Addresses for this criterion: 1 Change source IP to loopback 51
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public L3: Verify Source Reachability  Verify packets from the source are getting to the CPU Packet arriving on CPU queue 7 (host) 3750#show platform for Gi1/0/2 0018.ba88.1fc1 000f.f7e8.e042 ip 100.1.1.2 100.1.1.1 icmp 0 0 Ingress: Global Port Number: 1, lpn: 3 Asic Number: 1 Source Vlan Id: Real 101, Mapped 9. L2EncapType 0, L3EncapType 0 ... Station Descriptor: 00B00000, DestIndex: 00B0, RewriteIndex: 0000 ========================================== <output removed> Output Packets: ========================================== Egress: Asic 0, switch 2 CPU queues: 7 14. 52
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public L3: Verify Destination Reachability Switch# sh ip route 172.16.100.100 Routing entry for 172.16.100.0/24 Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 1 Last update from 10.1.1.2 on Vlan100, 00:08:54 ago Routing Descriptor Blocks: * 10.1.1.2, from 100.1.1.2, 00:08:54 ago, via Vlan100 Route metric is 20, traffic share count is 1 Switch # sh ip arp 10.1.1.2 Protocol Address Age (min) Hardware Addr Type Interface Internet 10.1.1.2 9 0018.ba88.1fc1 ARPA Vlan100 Switch# ping 172.16.100.100 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.16.100.100, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms Switch# ping 172.16.100.100 source vlan 101 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.16.100.100, timeout is 2 seconds: Packet sent with a source address of 192.168.100.1 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms 53
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public L3: Verify Hardware Forwarding  Show platform forward to verify HW programming 3750# show plat for Gi1/0/2 0018.ba88.1fc1 000f.f7e8.e042 ip 100.1.1.2 172.16.100.100 icmp 0 0 Ingress:Global Port Number: 2, lpn: 2 ASIC Number: 1 Source Vlan Id: Real 101, Mapped 10. L2EncapType 0, L3EncapType 0 <snip> <snip> Output Packets: GigabitEthernet1/0/1 Packet 1 Lookup Key-Used Index-Hit A-Data OutptACL 50_AC106464_64010102-00_01000000_00000100 01FFE 03000000 Port Vlan SrcMac DstMac Cos Dscpv Gi1/0/1 0100 000f.f7e8.e041 0018.ba88.1fc1 54
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting L3 Troubleshooting Steps Commands Verify source reachability ping show ip arp vlan sh mac address-table address Verify destination reachability show ip route show ip arp ping Verify HW programming show platform forward <ingress intf> <srcmac> <dstmac> ip <srcip> <dstip> Command Summary 55
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 3750# show ip igmp snooping mrouter vlan 100 Vlan ports 100 Gi1/0/1(dynamic) # debug ip igmp snooping router *Mar 1 03:33:44.075: IGMPSN: router: Received non igmp pak on Vlan 100, port Gi1/0/1 *Mar 1 03:33:44.075: IGMPSN: router: PIMV2 Hello packet received in 100 *Mar 1 03:33:44.075: IGMPSN: router: Is a router port on Vlan 100, port Gi1/0/1 *Mar 1 03:33:44.075: IGMPSN: router: Learning port: Gi1/0/1 as rport on Vlan 100 # debug platform cpu-queue igmp-snooping *Mar 1 03:39:09.469: Pak recvd on IGMP-SNOOP-Q: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0x12FC, LT:7, Vlan:100 SrcGPN:24, SrcGID:24, ACLLogIdx:0x0, MacDA:0100.5e00.0005, MacSA: 0011.21e6.5a40 IP_SA:10.160.16.1 IP_DA:224.0.0.5 IP_Proto:89 TPFFD:E841C018_00640064_00A0005E-000012FC_43330000_00000000 IGMP Multicast Router Port  Gets learned dynamically by listening either to PIM/DVMRP or to CGMP packets  Mrouter port should be learned dynamically View pkts to CPU 56
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public IGMP Client Join IGMP Joins received are sent to the CPU to be processed Switch# debug ip igmp snooping group *Mar 6 04:19:39.175: IGMPSN: Received IGMPv2 Report for group 239.100.100.100 received on Vlan 101, port Gi1/0/2 *Mar 6 04:19:39.175: IGMPSN: router: Is not a router port on Vlan 101, port Gi1/0/2 *Mar 6 04:19:39.175: IGMPSN: group: Skip client info adding - ip 10.101.1.100, port_id Gi1/0/2, on vlan 101 *Mar 6 04:19:39.175: IGMPSN: MCAST IP address 239.100.100.100, MAC address 0100.5e64.6464 *Mar 6 04:19:39.175: IGMPSN: Can not Locate gce 0100.5e64.6464, on Vlan 101 *Mar 6 04:19:39.175: IGMPSN: MCAST IP address 239.100.100.100, MAC address 0100.5e64.6464 *Mar 6 04:19:39.175: IGMPSN: Can not Locate gce 0100.5e64.6464, on Vlan 101 *Mar 6 04:19:39.175: IGMPSN: mgt: created gce 0100.5e64.6464, on Vlan 101 *Mar 6 04:19:39.175: l2mcm_group_create: creating a group 239.100.100.100 on vlan 101, dummy NO *Mar 6 04:19:39.175: l2mcm_group_create: timer stop: vlan 101, group 239.100.100.100 *Mar 6 04:19:39.175: IGMPSN: mgt: created group 239.100.100.100, on Vlan 101 *Mar 6 04:19:39.175: IGMPSN: mgt: Vlan 101 gce 0100.5e64.6464 add port Gi1/0/2 *Mar 6 04:19:39.175: L2MM: setting Gi1/0/2 in gce->mbr_blist 57
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public IP Multicast Routing  Verify PIM is working fine (not covered in this session)  Verify client is correctly joined via IGMP  Verify the switch is routing the flow correctly Switch# show ip mroute 239.100.100.100 10.99.1.100 IP Multicast Routing Table <output removed> (10.99.1.100, 239.100.100.100), 11:32:59/00:02:56, flags: JT Incoming interface: Vlan100, RPF nbr 10.100.1.1 Outgoing interface list: Vlan101, Forward/Sparse-Dense, 11:32:59/00:02:22 Switch# show ip igmp snooping groups vlan 101 239.100.100.100 Vlan Group Type Version Port List ----------------------------------------------------------------------- 101 239.100.100.100 igmp v2 Gi1/0/2 58
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public IP Multicast Routing  show forward can be used to verify if the ASICs are setup correctly to route the multicast flow Switch# show platform forward Gig 1/0/1 vlan 100 18.ba88.1fc2 0100.5e64.6464 ip 10.99.1.100 239.100.100.100 udp 0 0 Ingress: Global Port Number: 1, lpn: 3 ASIC Number: 1 Source Vlan Id: Real 100, Mapped 9. L2EncapType 0, L3EncapType 0 <output removed> Output Packets: GigabitEthernet1/0/2 Packet 1 Lookup Key-Used Index-Hit A-Data OutptACL 50_EF646464_0A630164-00_41000000_0000A87E 01FFE 03000000 Port Vlan SrcMac DstMac Cos Dscpv Gi1/0/2 0101 000f.f7e8.e042 0100.5e64.6464 59
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Multicast Troubleshooting Steps Commands IGMP sh ip igmp snooping mrouter vlan debug ip igmp snooping mrouter debug platform cpu-queue igmp- snooping debug ip igmp snooping group L3 Multicast sh ip mroute sh ip igmp snooping groups vlan show platform forward Command Summary 60
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Cisco Catalyst 3750 QoS Overview Classification • Inspect incoming packets • Based on ACLs or configuration, determine classification label Policing • Ensure conformance to a specified rate • On an aggregate or individual flow basis • Up to 256 policers per Port ASIC • Support for rate and burst Marking • Act on policer decision • Reclass or drop out-of-profile Egress Queue/ Schedule Congestion Control • Four SRR queues/port shared or shaped servicing • One queue is configurable for strict priority servicing • WTD for congestion control (three thresholds per queue) • Egress queue shaping • Egress port rate limiting Ingress Queue/ Schedule Congestion Control • Two queues/port ASIC shared servicing • One queue is configurable for strict priority servicing • WTD for congestion control (three thresholds per queue) • SRR is performed Ingress Egress Policer Policer Marker Policer Policer Marker Marker Marker SRR SRR Classify Traffic Stack Ring Egress Queues Ingress Queues 61
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public QoS Troubleshooting - Ingress  10,000 packets were received, DSCP value 34  1,467 packets were in profile  8,533 were dropped due to exceeding the policer 10000 IP packets with DSCP 34 access dot1q 3750 Gig 1/0/2 Gig 1/0/1 Switch# show mls qos interface gigabit 1/0/2 statistics GigabitEthernet1/0/2 (All statistics are in packets) dscp: incoming ------------------------------- 0 - 4 : 0 0 0 0 0 30 - 34 : 0 0 0 0 10000 ... Policer: Inprofile: 1467 OutofProfile: 8533 62
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch#sh mls qos interface gigabitEthernet 1/0/1 statistics GigabitEthernet1/0/1 (All statistics are in packets) <output removed> dscp: outgoing ------------------------------- <output removed> 25 - 29 : 0 0 0 0 0 30 - 34 : 0 0 0 0 1467 <output removed> QoS Troubleshooting - Egress  1467 packets were in profile and made it to the egress port  DSCP is 34 10000 IP packets with DSCP 34 access dot1q 3750 Gig 1/0/2 Gig 1/0/1 63
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch#sh mls qos interface gigabitEthernet 1/0/1 statistics GigabitEthernet1/0/1 (All statistics are in packets) <output removed> 0 – 4 : 1467 0 0 0 0 30 - 34 : 0 0 0 0 0 QoS Troubleshooting – Egress (2)  1467 packets were in profile and made it to the egress port but with DSCP 0 instead of 34.  Possible reasons: Attached service policy does not mark or trust dscp value Traffic is being routed via the CPU 10000 IP packets with DSCP 34 access dot1q 3750 Gig 1/0/2 Gig 1/0/1 64
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public QoS Troubleshooting - Egress Q Maps  10000 packets are received and will egress on Q4, threshold 1 10000 IP packets with DSCP 34 100Mb/s 10Mb/s Gig 1/0/2 Gig 1/0/1 3750 Switch# show mls qos maps dscp-output-q Dscp-outputq-threshold map: d1 :d2 0 1 2 3 4 5 6 7 8 9 ------------------------------------------------------------ 0 : 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 1 : 02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01 2 : 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 3 : 03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 4 : 01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01 5 : 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 6 : 04-01 04-01 04-01 04-01 Switch# show mls qos interface gi 1/0/2 statistics GigabitEthernet1/0/2 (All statistics are in packets) dscp: incoming ------------------------------- 0 - 4 : 0 0 0 0 0 30 - 34 : 0 0 0 0 10000 65
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 10000 IP packets with DSCP 34 100Mb/s 10Mb/s Gig 1/0/2 Gig 1/0/1 QoS Troubleshooting – Egress Queue Ts  1080 packets will egress on Q4, threshold 1  Remaining pkts dropped 3750 Switch# show mls qos interface Gig 1/0/1 statistics dscp: outgoing ------------------------------- 0 - 4 : 0 0 0 0 0 30 - 34 : 0 0 0 0 1080 ... output queues enqueued: queue: threshold1 threshold2 threshold3 ----------------------------------------- queue 0: 2 0 0 queue 1: 0 6 4560 queue 2: 0 0 0 queue 3: 1080 0 0 output queues dropped: queue: threshold1 threshold2 threshold3 ----------------------------------------- queue 0: 0 0 0 queue 1: 0 0 0 queue 2: 0 0 0 queue 3: 8920 0 0 66
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public QoS Troubleshooting - Port-ASIC  10000 packets were received, 8920 were dropped on egress 10000 IP packets with DSCP 34 100Mb/s 10Mb/s Viewing Egress Congestion (another way) with port-asic command Gig 1/0/2 Gig 1/0/1 3750 Switch# show platform port-asic stats drop gigabitEthernet 1/0/1 Interface Gi1/0/1 TxQueue Drop Statistics Queue 0 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 … Queue 3 Weight 0 Frames 8920 Weight 1 Frames 0 Weight 2 Frames 0 67
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch# show mls qos int gi1/0/1 buffers GigabitEthernet1/0/1 The port is mapped to qset : 1 The allocations between the queues are : 25 25 25 25 Switch# show mls qos queue-set Queueset: 1 Queue : 1 2 3 4 ---------------------------------------------- buffers : 25 25 25 25 threshold1: 200 200 100 100 threshold2: 200 200 100 100 QoS Troubleshooting - Buffer Tuning  Queue-sets define the buffer allocation  Default values can be modified  2 Queue-sets are available  Reserved - how many buffers will be reserved for this port ‒Default Queue-set values listed below Identifies Queue-set assigned to interface Dropped on this Queue and Threshold Tuning Buffers and Thresholds to fix congestion 68
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public QoS Troubleshooting - Buffer Tuning (2) 400 IP packets with DSCP 34 100Mb/s 10Mb/s Packet drops with current Queue-set configuration No additional Packet drops after Queue-set change Threshold increased to 300 3750 Gig 1/0/2 Gig 1/0/1 Switch# show platform port-asic stats drop gigabitEthernet 1/0/1 Interface Gi1/0/1 TxQueue Drop Statistics Queue 3 Weight 0 Frames 8920 Switch(config)# mls qos queue-set output 1 threshold 4 300 300 50 400 Switch# show mls qos queue-set Queueset: 1 Queue : 1 2 3 4 ---------------------------------------------- buffers : 25 25 25 25 threshold1: 100 100 100 300 threshold2: 100 100 100 300 reserved : 50 50 50 50 maximum : 400 400 400 400 Switch# show platform port-asic stats drop gigabitEthernet 1/0/1 Interface Gi1/0/1 TxQueue Drop Statistics Queue 3 Weight 0 Frames 8920 69
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting QoS Issues Troubleshooting Steps Commands Check for Errors Ingress and Egress ports show mls qos interface <> stats Check Queue mapping show mls qos maps dscp-output-q Check Egress Queue details show platform port-asic stats drop <> Check and tune buffers show mls qos queue-set mls qos queue-set output <> threshold Command Summary 70
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Making Life Easier…  Review open caveats sections in release notes  Search Bug Toolkit for known issues  Reference Output Interpreter to decode command output  Reference the Error and System Messages for recovery procedures 71
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Cat 2K-3K Troubleshooting Summary  Know your network, have baselines: CPU, Memory, TCAM Is the value normal?  Check the logs Error messages? Warnings?  Follow the packet Use the architecture to understand the flow Narrow down possibilities  Check overall health Is the HW OK? Are the members of the stack ready? ‘show post’ – to view results of last self check on bootup 72
Q & A
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Complete Your Online Session Evaluation Give us your feedback and receive a Cisco Live 2013 Polo Shirt! Complete your Overall Event Survey and 5 Session Evaluations.  Directly from your mobile device on the Cisco Live Mobile App  By visiting the Cisco Live Mobile Site www.ciscoliveaustralia.com/mobile  Visit any Cisco Live Internet Station located throughout the venue Polo Shirts can be collected in the World of Solutions on Friday 8 March 12:00pm-2:00pm Don’t forget to activate your Cisco Live 365 account for access to all session material, 74 communities, and on-demand and live activities throughout the year. Log into your Cisco Live portal and click the "Enter Cisco Live 365" button. www.ciscoliveaustralia.com/portal/login.ww
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 75
Appendix 76
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Local Switching and Stacking • With StackWise, locally destined packets must traverse the entire stack ring. Relevant to 3750v2 StackWise StackWise Plus Local Switching With StackWise Plus, whether in a homogeneous or mixed-hardware stack, locally destined packets on an “E” or “X” series switch are never put on the stack ring. 77
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU CPU CPU TCAMs TCAMs TCAMs MAC Address Management Distributed MAC learning  Stack members learn MAC addresses and updates TCAM entries  System synchronises MAC address tables across the stack  How it is distributed: ‒ A switch learns an address and sends a message to other switches in the stack ‒ Learning an address that was previously learned on a different port (either same or different switch) is considered as move MAC B MAC A 78
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Tips…  Enable NTP to troubleshoot across switches ntp server <ip>  When debugging send the output to the buffer, not to the console and include date and time in messages configure terminal ‒ no logging console ‒ logging buffered 128000 ‒ service timestamps log datetime localtime msec show-timezone ‒ service timestamps debug datetime localtime msec show-timezone 79
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Tips…  Include comments on the console as reminders ‒ C3750#!!! Comments here  Execute ‘show’ command from ‘config term’ mode ‒ C3750(config)# do show running int Gi1/0/1  Session to another switch member ‒ C3750#session <member #> or C3750#remote command <1-9|all> “IOS command” 80
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU Utilisation Troubleshooting  High CPU Utilisation is problematic when: ‒Delays in forwarding of network traffic ‒Catalyst switch unable to respond to network problems in timely fashion ‒Switch management can become blocked, as CPU does not respond  Baseline CPU Utilisation varies by Model ‒Catalyst 2960, 3560, 3560G: ~6% (non-stacked models) ‒Catalyst 3750, 3750G: ~7% (stacked) ‒Catalyst 3750E: ~9% (stacked) ‒Catalyst 3750X: ~22% (stacked) ‒Catalyst 2960S: ~20% (stacked or non-stacked) ‒Feature set (LAN BASE, IP BASE, or IP SERVICES) will impact CPU util as well 81
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: Troubleshooting Processes  Process Utilisation  Processes taking up resources  For ex “show tech” causes the virtual exec process to use some CPU resources  Capturing process utilisation at the “right” moment is key for identifying the cause  Traffic Forwarding  High component of utilisation due to interrupts  Data traffic not forwarded by ASIC  Excessive Control Plane / Management traffic:  DoS attacks (TTL=1)  SVI ping test  Requires inspecting CPU queues and ASIC 82 High CPU Utilisation is Due to:
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: Layer 2 Control Protocol Queues  STP has its own queue – Queue 1  Layer 2 protocols queue for the rest – Queue 4 ‒ CDP , PAgP, LACP, DTP, LLDP, UDLD  Drops on these queues 1 or 4 can cause instability on the network Switch# show controllers cpu-interface cpu-queue-frames retrieved dropped invalid hol-block stray ----------------- ---------- ---------- ---------- ---------- ----- rpc 132917740 0 0 0 0 stp 31879262 0 23288714 0 0 ipc 10746915 0 0 0 0 routing protocol 267 0 0 0 0 L2 protocol 424610 0 0 0 0 remote console 1121711 0 105531 0 0 sw forwarding 0 0 0 0 0 host 345 0 0 0 0 broadcast 13931 0 55724 0 0 cbt-to-spt 0 0 0 0 0 igmp snooping 0 0 0 0 0 icmp 0 0 0 0 0 logging 0 0 0 0 0 rpf-fail 0 0 0 0 0 dstats 132935598 0 0 0 0 cpu heartbeat 82903147 0 0 0 0 83
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: Routing Protocol Queue (Q3)  Receives all traffic for routing protocols, like BGP, OSPF, EIGRP, HSRP, etc.  Debug traffic received by CPU. ‒ In case below “routing-protocol-q” is shown ‒ Packet ingress intf, Dest MAC, SrcMAC, Dest IP, Src IP are shown Switch# debug platform cpu-queues routing-protocol-q Switch# debug standby HSRP debugging is on *Mar 6 00:47:39.260: RT-Q:Queued: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0x12FC, LT:7, Vlan:100 SrcGPN:1, SrcGID:1, ACLLogIdx:0x0, MacDA:0100.5e00.0002, MacSA: 0018.ba88.1fc1 IP_SA:10.1.1.2 IP_DA:224.0.0.2 IP_Proto:17 *Mar 6 00:47:39.260: HSRP: Vl100 Grp 0 Hello in 10.1.1.2 Standby pri 100 vIP 10.1.1.55 84
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch# debug platform cpu-queues host-q *Mar 6 00:01:46.648: Host-Q:Queued L3If: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0xB0, LT:7, Vlan:100 SrcGPN:489, SrcGID:488, ACLLogIdx:0x0, MacDA:000f.f7e8.e041, MacSA: 0018.ba88.1fc1 IP_SA:10.1.1.2 IP_DA:10.1.1.1 IP_Proto:1 CPU: Host Queue (Q7)  Used for all unicast traffic sent to the switch. ‒ TACACS, SSH, telnet, ping, SNMP 85
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: Host Queue (Q7) – Drops  Show buffer shows current buffer usage (RxQ7)  When free buffers reaches below watermark(32), throttling might occur, resulting in packet drops  slow responsiveness to network management Misses equals drops 86 Switch# debug platform cpu-queues host-q *Mar 6 00:01:46.648: Host-Q:Queued L3If: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0xB0, LT:7, Vlan:100 SrcGPN:489, SrcGID:488, ACLLogIdx:0x0, MacDA:000f.f7e8.e041, MacSA: 0018.ba88.1fc1 IP_SA:10.1.1.2 IP_DA:10.1.1.1 IP_Proto:1 TPFFD:DC0001E9_00000064_00B00076-000000B0_A68A0000_00000000 Switch# show buffer | begin RxQ7 RxQ7 buffers, 2040 bytes (total 192, permanent 192): 64 in free list (0 min, 192 max allowed) 294 hits, 0 misses
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: ICMP Queue (Q11)  Receives all traffic for which an ICMP message needs to be generated (excluding PING)  Receives a copy of the traffic for which an ICMP packet needs to be generated. Hardware forwarding of the packet still occurs Switch# debug ip icmp Switch# debug platform cpu-queues icmp-q *Mar 9 21:34:30.695: ICMP-Q:Queued to Process, use GW:10.1.1.3: Remote Port Blocked L3If:Vlan100 L2If:GigabitEthernet4/0/1 DI:0xB4, LT:7, Vlan:100 SrcGPN:163, SrcGID:163, ACLLogIdx:0x0, MacDA:0018.ba88.1fc1, MacSA: 000f.f7e8.e041 IP_SA:10.1.1.1 IP_DA:77.1.1.1 IP_Proto:1 *Mar 9 21:34:30.695: ICMP: redirect sent to 10.1.1.1 for dest 77.1.1.1, use gw 10.1.1.3 87
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: ICMP Queue (Q11)  Symptoms: ‒ relatively high CPU (due to throttling mechanism it won’t reach 99%) of which majority is due to interrupts ‒ Low process utilisation ‒ ICMP Queue heavily utilised 88 ICMP Unreachables Example Switch# show processes cpu sorted CPU utilization for five seconds: 53%/47%; one minute: 31%; five minutes: 18% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 149 397089 3879429 102 0.63% 0.34% 0.45% 0 Spanning Tree 112 325474 117735 2764 0.31% 0.15% 0.09% 0 HRPC qos request Switch# clear controllers cpu Switch# show controllers cpu-interface | include icmp icmp 133148 0 0 0 0
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: ICMP Queue (Q11)  Causes: ‒ High amount of traffic is dropped because of a “deny” statement in an ACL ‒ CPU is interrupted to send ICMP unreachable packets back to the source  Solution ‒ Disable ICMP unreachables on the ingress interface 89 ICMP Unreachables Example Switch(config)# interface GigabitEthernet1/0/2 Switch(config-if)# no ip unreachable Switch(config-if)# end
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch# clear controllers cpu Switch# show controllers cpu-interface | include sw forwarding sw forwarding 71558 0 0 0 0 Switch# debug platform cpu-queues software-fwd-q SW-FWD-Q:Consumed by SW-Bridging: Remote Port Blocked L3If:Vlan101 L2If:GigabitEthernet1/0/2 DI:0x2FD, LT:7, Vlan:101 SrcGPN:2, SrcGID:2, ACLLogIdx:0x0, MacDA:000f.f7e8.e042, MacSA: 0000.00bb.87df IP_SA:10.101.1.100 IP_DA:10.99.1.100 IP_Proto:255 CPU: Software Forwarding Queue (Q6)  For Traffic that hardware cannot process  SW forwarding performance is much lower than HW  Requires ASIC forwarding troubleshooting 90 Causing High CPU Utilisation
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  CPU utilisation sustained below 50% will not cause problems  3750-X and 2960-S will have a higher CPU utilisation ‒ It is normal around 30% and 20% respectively  Example of Syslog message for high CPU ‒002182: *Jul 20 04:23:36: %SYS-1- CPURISINGTHRESHOLD: Threshold: Process CPU Utilization(Total/Intr): 9%/0%, Top 3 processes(Pid/Util): 214/3%, 153/0%, 159/0%  Sorting the output is better than filtering the output with “exclude 0.00%” because that will exclude processes that you want to see. ‒Switch# show process cpu sorted CPU Utilisation: Summary 92
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public TS: I/O Memory Buffers  I/O memory for incoming CPU bound packets  Used by Routers for control and data packets ‒ On only control packets  Shows CPU bound packets ‒ Not HW switched packets Switch# show buffers Buffer elements: 1679 in free list (500 max allowed) 27109526 hits, 0 misses, 1641 created Public buffer pools: Small buffers, 104 bytes (total 50, permanent 50, peak 181 @ 3w5d): 49 in free list (20 min, 150 max allowed) 129877853 hits, 141 misses, 390 trims, 390 created 0 failures (0 no memory) Middle buffers, 600 bytes (total 25, permanent 25, peak 94 @ 7w0d): 25 in free list (10 min, 150 max allowed) 616791 hits, 54 misses, 162 trims, 162 created 0 failures (0 no memory) . . . 93
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch# show interfaces GigabitEthernet 1/0/1 counters errors Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards Gi1/0/1 0 0 0 0 0 0 Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants Gi1/0/1 0 0 0 0 0 0 0 Switch# show interfaces counters errors Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards Gi1/0/1 0 0 0 0 0 0 Gi1/0/2 0 0 0 0 0 0 <snip> Gi2/0/12 0 0 0 0 0 0 Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants Gi1/0/1 0 0 0 0 0 0 0 Gi1/0/2 0 0 0 0 0 0 0 Link Issues: What Kind of Errors? See slides in the appendix for Error Explanation 94
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: MACsec Encryption (1/3)  MACsec encryption is performed in the PHY at line rate  More efficient than L3 or application encryption  Overhead of 32 bytes per packet might cause drops in the PHY  Recommendation: rate-limit egress interface at 85% to benefit from ASIC buffering and QoS 95 Symptom: Traffic drops Switch(config)# interface gigabitethernet2/0/1 Switch(config-if)# srr-queue bandwidth limit 85
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Checking Physical Cabling 96 Gi4/1/1 Gi1/1/1 dBm -2.0 -4.2 Fibre Attenuation 1-3 dBm / km Connectors & Patchcord Loss ~ 1 dBm Wrong fibre type! MM instead of SM Loss Budget Analysis (Northbound) Insert a 5 dB attenuator to solve the problem Dirty Connector
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Checking Physical Cabling  Use the DOM feature on the port to determine possible optical transmission issues: dirty connectors, wrong attenuation, transceiver issues 97 Gi4/1/1 Gi1/1/1 Switch# show interface GigabitEthernet4/1/1 transceiver details Optical High Alarm High Warn Low Warn Low Alarm Transmit Power Threshold Threshold Threshold Threshold Port (dBm) (dBm) (dBm) (dBm) (dBm) ------- ----------------- ---------- --------- --------- --------- Gi4/1/1 -2.7 8.1 5.0 0.0 -4.0 Switch# show interface GigabitEthernet1/1/1 transceiver ITU Channel not available (Wavelength not available), Transceiver is internally calibrated. ++ : high alarm, + : high warning, - : low warning, -- : low alarm. NA or N/A: not applicable, Tx: transmit, Rx: receive. mA: milliamperes, dBm: decibels (milliwatts). Optical Optical Temperature Voltage Tx Power Rx Power Port (Celsius) (Volts) (dBm) (dBm) --------- ----------- ------- -------- -------- Gi1/1/1 38.6 3.29 -2.0 -11.3
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Checking Physical Cabling 98 Gi4/1/1 Gi1/1/1 dBm -2.7 -11.3 Loss Budget Analysis (Southbound) Connector not firmly plugged in
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 2960-S FlexStack Ease of Use 3750-X StackWise Plus Ease of Use and High Availability Device Limit 4 units 9 units Stack Bandwidth 20G 64G Architecture HW Drop Table Ring (Destination stripping) Dynamic Ring Load Balancing No Yes Stack Convergence 1-2 seconds Few milliseconds Stack QoS Applied hop by hop Applied on ingress Management Single IP address, SNMP, SYSLOG Single IP address, SNMP, SYSLOG Configuration Single config and CLI, auto image and config update Single config and CLI, auto image and config update Show and Debug Commands Unified Unified Single Forwarding and Control Plane Synchronise ARP, MAC Address, IGMP, VLAN tables Synchronise ARP, MAC Address, IGMP, VLAN, Routing tables Cross-Stack Features Yes Yes Single Bridge-ID Yes Yes Preprovison members Yes Yes Redundancy Stack master 1:N redundancy Stack master 1:N redundancy Easy member replacement Yes Yes FlexStack Vs. StackWise Plus 99
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Appendix A Error Counters Definition  FCS-Err is the number of valid size frames with FCS (Frame Check Sequence) errors but no framing errors: this is typically a physical issue (cabling, bad port, NIC card,…) but can also indicate a duplex mismatch  Align-Err is the number of frames with alignment errors (frames that do not end with an even number of octets and have a bad CRC) received on the port; these usually indicate a physical problem (cabling, bad port, NIC card,…) but can also indicate a duplex mismatch; when the cable is first connected to the port, some of these errors may occur; also, if there is a hub connected to the port then collisions between other devices on the hub may cause these errors  Late-Coll (Late Collisions) is the number of times that a collision is detected on a particular port late in the transmission process; for a 10mbit/s port this is later than 512 bit-times into the transmission of a packet; five hundred and twelve bit-times corresponds to 51.2 microseconds on a 10 Mbit/s system; this error can indicate a duplex mismatch among other things; for the duplex mismatch scenario the late collision would be seen on the half duplex side; as the half duplex side is transmitting, the full duplex side does not wait its turn and transmits simultaneously causing a late collision; late collisions can also indicate an Ethernet cable/segment that is too long; collisions should not be seen on ports configured as full duplex  Single-Coll (Single Collision) is the number of times one collision occurred before the port transmitted a frame to the media successfully; collisions are normal for port configured as half duplex but should not be seen on full duplex ports; if collisions are increasing dramatically this points to a highly utilised link or possibly a duplex mismatch with the attached device  Multi-Coll (Multiple Collision) is the number of times multiple collisions occurred before the port transmitted a frame to the media successfully; collisions are normal for port configured as half duplex but should not be seen on full duplex ports; if collisions are increasing dramatically this points to a highly utilised link or possibly a duplex mismatch with the attached device 100
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Appendix A Error Counters Definition (2)  Excess-Coll (Excessive Collisions) is a count of frames for which transmission on a particular port fails due to excessive collisions; an excessive collision happens when a packet has a collision 16 times in a row; the packet is then dropped; excessive collisions is typically an indication that the load on the segment needs to be split across multiple segments but can also point to a duplex mismatch with the attached device; collisions should not be seen on ports configured as full duplex  Carri-Sen (Carrier Sense) occurs every time an Ethernet controller wants to send data on a half duplex connection; the controller senses the wire and check if it is not busy before transmitting; this is normal on an half-duplex Ethernet segment  Undersize are frames received that are smaller than the minimum IEEE 802.3 frame size of 64bytes long (excluding framing bits, but including FCS octets) that were otherwise well formed; check the device sending out these frames  Runts are frames received that are smaller than the minimum IEEE 802.3 frame size (64 bytes for Ethernet) and with a bad CRC; this can be caused by duplex mismatch and physical problems like a bad cable, port, or NIC card on the attached device  Giants exceed the maximum IEEE 802.3 frame size (1518 bytes for non-jumbo Ethernet); try to find the offending device and remove it from the network  http://www.cisco.com/warp/public/473/164.html#show_interface 101
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Port Access-List Switch# sh run | inc access-list access-list 123 permit ip host 10.100.1.2 any Switch# sh run int gi 1/0/2 Building configuration... Current configuration : 134 bytes ! interface GigabitEthernet1/0/2 switchport access vlan 101 ip access-group 123 in mls qos trust dscp spanning-tree portfast end 102
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Port Access-List Details Switch# sh platform acl interface gigabitEthernet 1/0/2 portlabels detail Port based ACL: (asic 1) ---------------------------- Input Label: 4 Op Select Index: 255 Interface(s): Gi1/0/2 Access Group: 123, 3 VMRs Mask: 00000000 FFFFFFFF 00000000 00000000 00000000 Value: 00000000 0A640102 00000000 00000000 00000000 Result: 0x09 --- Permit IP Source address Mask: 00000000 00000000 00000000 00000000 00000000 Value: 00000000 00000000 00000000 00000000 00000000 Result: 0x00 --- Deny Mask & Value all 0 = any any Mask: 00000000 00000000 00000000 00000000 00000000 Value: 00000000 00000000 00000000 00000000 00000000 Result: 0x09 IP Source Guard: 0 VMRs LPIP: 0 VMRs MAC Access Group: (none), 0 VMRs access-list 123 permit ip host 10.100.1.2 any 103
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Router Access-List Configuration : ! interface Vlan101 ip address 10.101.1.1 255.255.255.0 ip access-group 123 in ! access-list 123 permit ip host 10.100.1.2 any Switch# sh platform acl interface vlan 101 Input Label: 1 Output Label: 0 (default) Input IPv6 Label: 1 Output IPv6 Label: 0 (default) 104
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Router Access-List Details Switch# sh platform acl label 1 detail IPv4/MAC ACL label ------------------ Input Op Select Index 255: Output Op Select Index 255: Input Features: Interfaces or VLANs: Vl101 Vlan Map: (none) Access Group: 123, 5 VMRs. Mask: 00000000 FFFFFFFF 00000000 00000000 00000000 Value: 00000000 0A640102 00000000 00000000 00000000 Result: 0x09 Mask: 00000000 00000000 05000000 00000000 00000000 <output removed> 10.100.1.2 105
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Vlan Access-List vlan access-map FilterMap 10 action drop match ip address 123 ! vlan filter FilterMap vlan-list 101 access-list 123 permit ip host 10.100.1.2 any Switch# sh vlan filter VLAN Map FilterMap is filtering VLANs: 101 Switch# show platform acl vlan 101 Input Label: 1 Output Label: 1 Input IPv6 Label: 1 Output IPv6 Label: 1 106
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 107 Supported ACL TCAM Entry Types Commonly Known As… Configured On… Controls… Direction MAC ACL MACL L2 port Non-IP packets In-bound Port ACL PACL L2 Port IP packets Non-IP packets using MACL In-bound Router ACL RACL L3 port L3 EtherChannel port Switched Virtual Interface (SVI) Routed IP traffic In-bound Out-bound VLAN ACL VACL VLAN IP packets routed into or out of a VLAN IP packets bridged within a VLAN Directionless
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Cisco Catalyst 2960-S QoS Model Classification • Inspect incoming packets • Based on ACLs or configuration, determine classification label Policing • Ensure conformance to a specified rate • On an aggregate or individual flow basis • Up to 256 policers per switch • Support for rate and burst Marking • Act on policer decision • Reclass or drop out-of-profile Egress Queue/ Schedule Congestion Control • Four SRR queues/port shared or shaped servicing • One queue is configurable for strict priority servicing • WTD for congestion control (three thresholds per queue) • Egress queue shaping • Egress port rate limiting Ingress Egress Policer Policer Marker Policer Policer Marker Marker Marker SRR Classify Traffic Egress Queues Traffic 108
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Catalyst 2/3000 QoS Cheat sheet  Aggregate Policer – Marking in policy-map ‒Check Configuration ‒Sh mls qos int gig x/y statistics ‒!!! NOT SUPPORTED : sh policy-map interface  Queueing and scheduling : ‒show platform port-asic stats drop gig x/y ‒show platform port-asic stats enqueue gig x/y  General QoS command : Show running-config Show mls qos Show platform tcam utilization 109
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public IGMP Snooping on Catalyst 3750  IGMP snooping entry are not created per mac-address, but they are created for IP multicast group 3750#sh ip igmp snooping groups Vlan Group Type Version Port List ------------------------------------------------------------- 1 239.1.2.3 user Po1 3750#sh mac address-table address 0100.5e01.0203 Mac Address Table ------------------------------------------- Vlan Mac Address Type Ports ---- ----------- -------- -----  A packet with destination mac 0100.5E01.0203 won’t be constrained per IGMP snooping if its ip is not 239.1.2.3! An entry is present with IP address in IGMP Snooping table There is nothing in mac-address-table for the group 110
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public References  Troubleshooting Catalyst 3750: http://www.cisco.com/en/US/products/hw/switches/ps5023/prod_troubleshooting_guides_list.html  Online Resources on http://www.cisco.com: ‒Troubleshooting High CPU Utilisation ‒Troubleshooting Power over Ethernet (PoE) ‒Troubleshooting Switch Stacks ‒Cisco Catalyst 3750 QoS Configuration Examples (Doc 91862) ‒Auto Negotiation issues: (Document 17053) 111
© 2013 Cisco and/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 112

More Related Content

PDF
Troubleshooting the Cisco Catalyst 9000 Series Switches - BRKTRS-3090.pdf
byTestTest449467
 
PDF
Cisco Catalyst Access Series Switching – Architecture and Troubleshooting
byFernando Andrade
 
PPT
05 module managing your network enviornment
byAsif
 
PDF
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches (20...
byaaajjj4
 
PDF
BRKRST-3066 - Troubleshooting Nexus 7000 (2013 Melbourne) - 2 Hours.pdf
byaaajjj4
 
PPT
05- Module Managing your network Enviornment.ppt
byMuhammadHaris117196
 
PDF
BRKACI-2102_Tshoot.pdf
byCcieOfPeople
 
PDF
CISCO 200 125-ccna-v3
byLeo Juntilla
 
Troubleshooting the Cisco Catalyst 9000 Series Switches - BRKTRS-3090.pdf
byTestTest449467
 
Cisco Catalyst Access Series Switching – Architecture and Troubleshooting
byFernando Andrade
 
05 module managing your network enviornment
byAsif
 
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches (20...
byaaajjj4
 
BRKRST-3066 - Troubleshooting Nexus 7000 (2013 Melbourne) - 2 Hours.pdf
byaaajjj4
 
05- Module Managing your network Enviornment.ppt
byMuhammadHaris117196
 
BRKACI-2102_Tshoot.pdf
byCcieOfPeople
 
CISCO 200 125-ccna-v3
byLeo Juntilla
 

Similar to BRKRST-3068 Troubleshooting Catalyst 2K and 3K.pdf

PDF
200 125-ccna-v3
byRuben Gagita
 
PDF
200-125-ccna-v3
byIbby Nuj
 
PDF
CCNA 2016 SYLLABUS
byNafis Khan Chowdhury
 
PPTX
Technical Overview of Cisco Catalyst 9200 Series Switches
byRobb Boyd
 
PPT
Cisco data center support
byKrunal Shah
 
DOC
Simple network troubleshooting
byMohsen B
 
PPT
Introduction to routers
bySantosh Kulkarni
 
PDF
Interconnecting_Cisco_Networking_Devices.pdf
byDaginni78
 
PDF
Troubleshooting BGP.pdf.................
byAntonioIsipJr1
 
PPT
CCNA Icnd110 s02l05
bycomputerlenguyen
 
PDF
Samplab19
byNoman Ashraf (CCNA, MCTS)
 
DOC
Output drops due to qo s on cisco 2960 3560 3750 switches
bycandy tang
 
PPTX
Chapter: Basic Router Configuration Cisco
bysamiulsuman
 
PPT
CCNA Icnd110 s02l08
bycomputerlenguyen
 
PPTX
PLNOG 13: P. Kupisiewicz, O. Pelerin: Make IOS-XE Troubleshooting Easy – Pack...
byPROIDEA
 
PDF
Cisco: Care and Feeding of Smart Licensing
bydaxtindavon
 
PDF
CCIE_RS_Quick_Review_Kit
byChris S Chen
 
PDF
CCIE
byLahcene Berkani
 
PDF
Cisco -Ccie rs quick_review_kit
byStoyan Stoyanov
 
PPT
managing your network environment
byscooby_doo
 
200 125-ccna-v3
byRuben Gagita
 
200-125-ccna-v3
byIbby Nuj
 
CCNA 2016 SYLLABUS
byNafis Khan Chowdhury
 
Technical Overview of Cisco Catalyst 9200 Series Switches
byRobb Boyd
 
Cisco data center support
byKrunal Shah
 
Simple network troubleshooting
byMohsen B
 
Introduction to routers
bySantosh Kulkarni
 
Interconnecting_Cisco_Networking_Devices.pdf
byDaginni78
 
Troubleshooting BGP.pdf.................
byAntonioIsipJr1
 
CCNA Icnd110 s02l05
bycomputerlenguyen
 
Samplab19
byNoman Ashraf (CCNA, MCTS)
 
Output drops due to qo s on cisco 2960 3560 3750 switches
bycandy tang
 
Chapter: Basic Router Configuration Cisco
bysamiulsuman
 
CCNA Icnd110 s02l08
bycomputerlenguyen
 
PLNOG 13: P. Kupisiewicz, O. Pelerin: Make IOS-XE Troubleshooting Easy – Pack...
byPROIDEA
 
Cisco: Care and Feeding of Smart Licensing
bydaxtindavon
 
CCIE_RS_Quick_Review_Kit
byChris S Chen
 
CCIE
byLahcene Berkani
 
Cisco -Ccie rs quick_review_kit
byStoyan Stoyanov
 
managing your network environment
byscooby_doo
 

More from ssusercbaa33

PPTX
ReaR 를 이용하여 Linux OS Backup 설정방법.pptx
byssusercbaa33
 
PDF
MariaDB Xpand 고객사례 안내.pdf
byssusercbaa33
 
PDF
Cisco Cat.4500Series License Installation Guide.pdf
byssusercbaa33
 
PDF
BeeGFS Training.pdf
byssusercbaa33
 
PDF
MariaDB 기본 소개 20201216.pdf
byssusercbaa33
 
PDF
All_Things_ASA_v3_1_20140609.pdf
byssusercbaa33
 
PDF
Sourcefire_Technical_Bootcamp_20140609.pdf
byssusercbaa33
 
PDF
ipv6_study.pdf
byssusercbaa33
 
PDF
Cisco VxLAN, LTRDCT-1223 Implementing VXLAN in a Data Center.pdf
byssusercbaa33
 
PDF
Cisco IOS XRv Router.pdf
byssusercbaa33
 
PDF
Cisco Wireless LAN Controller Command Reference, Release 7.4.pdf
byssusercbaa33
 
ReaR 를 이용하여 Linux OS Backup 설정방법.pptx
byssusercbaa33
 
MariaDB Xpand 고객사례 안내.pdf
byssusercbaa33
 
Cisco Cat.4500Series License Installation Guide.pdf
byssusercbaa33
 
BeeGFS Training.pdf
byssusercbaa33
 
MariaDB 기본 소개 20201216.pdf
byssusercbaa33
 
All_Things_ASA_v3_1_20140609.pdf
byssusercbaa33
 
Sourcefire_Technical_Bootcamp_20140609.pdf
byssusercbaa33
 
ipv6_study.pdf
byssusercbaa33
 
Cisco VxLAN, LTRDCT-1223 Implementing VXLAN in a Data Center.pdf
byssusercbaa33
 
Cisco IOS XRv Router.pdf
byssusercbaa33
 
Cisco Wireless LAN Controller Command Reference, Release 7.4.pdf
byssusercbaa33
 

Recently uploaded

PDF
Reality Drift: Cultural and Academic Footprints Across Media, Classrooms, and...
byDr. Samuel Wellington
 
PDF
The 13 satanic codes for Self Power and Sovereignty
byhikariaoki1
 
PDF
Dark Humanism 200 Principles for living a life in shadow
byhikariaoki1
 
PPTX
Generating-Patterns: Arithmetic Sequences, Arithmetic Mean,
byChanelleObate
 
PDF
Unlocking-the-Future-AI-Software-Development-Services.pdf
byPrologicTechnologies
 
PDF
Communicable Disease Screening Form from ministry of health pdf
byDr.Anjalatchi Muthukumaran
 
PPTX
The First Internet Message was sent on 29 Oct 1969. However . . .
byBob Mayer
 
PPTX
UNIT –I community health CHN P.B.Sc ppt.pptx
byDr.Anjalatchi Muthukumaran
 
PPTX
一比一原版(StuttgartH毕业证书)斯图加特工程应用技术大学毕业证办理成绩单学历认证
by办硕士文凭昆士兰科技大学学历认证【Q微号:1954 292 140】QUT毕业证成绩单
 
PPTX
Protection from Cyber attacks and Security Firewall
byAjayTiwari684365
 
PPTX
Presentation contoso chart 1 2 3 4 5 6 7 8
byJohannReyes3
 
PPTX
TREC_2024_Lateral_Reading_Purpose_and_Need.pptx
byGichxIocHiIvdug
 
PPTX
最新版英国伯恩茅斯艺术大学毕业证（AUB毕业证书）定制成绩单在读证明学历认证
by本科文凭旧金山艺术大学学历认证【Q微号:1954 292 140】AAU毕业证成绩单
 
PDF
kjhhhhhhhhhhhhhhhhhhhhhhhhhhhhhkjhkkkkkkkkkkkkkkkkjh
bybishnoijordan29
 
PDF
Nome completo de Laurence Gregory Watkins
bydiariodocentrodomundo
 
PDF
Buying, Aged Twitter Accounts_ The Secret to Instant Growth.pdf
byBEST IT SMM
 
PPTX
Viva_Digitally_Online_Meeting_Presentation.pptx
byHubraSEO
 
PDF
WHMCS Croster 2.1.2 Nulled Pre Activated
byCroster 2.1.2 Nulled
 
PDF
BrightonSEO Oct 2025 - How Migrations Have Changed - Nick Samuel
byNick Samuel
 
Reality Drift: Cultural and Academic Footprints Across Media, Classrooms, and...
byDr. Samuel Wellington
 
The 13 satanic codes for Self Power and Sovereignty
byhikariaoki1
 
Dark Humanism 200 Principles for living a life in shadow
byhikariaoki1
 
Generating-Patterns: Arithmetic Sequences, Arithmetic Mean,
byChanelleObate
 
Unlocking-the-Future-AI-Software-Development-Services.pdf
byPrologicTechnologies
 
Communicable Disease Screening Form from ministry of health pdf
byDr.Anjalatchi Muthukumaran
 
The First Internet Message was sent on 29 Oct 1969. However . . .
byBob Mayer
 
UNIT –I community health CHN P.B.Sc ppt.pptx
byDr.Anjalatchi Muthukumaran
 
一比一原版(StuttgartH毕业证书)斯图加特工程应用技术大学毕业证办理成绩单学历认证
by办硕士文凭昆士兰科技大学学历认证【Q微号:1954 292 140】QUT毕业证成绩单
 
Protection from Cyber attacks and Security Firewall
byAjayTiwari684365
 
Presentation contoso chart 1 2 3 4 5 6 7 8
byJohannReyes3
 
TREC_2024_Lateral_Reading_Purpose_and_Need.pptx
byGichxIocHiIvdug
 
最新版英国伯恩茅斯艺术大学毕业证（AUB毕业证书）定制成绩单在读证明学历认证
by本科文凭旧金山艺术大学学历认证【Q微号:1954 292 140】AAU毕业证成绩单
 
kjhhhhhhhhhhhhhhhhhhhhhhhhhhhhhkjhkkkkkkkkkkkkkkkkjh
bybishnoijordan29
 
Nome completo de Laurence Gregory Watkins
bydiariodocentrodomundo
 
Buying, Aged Twitter Accounts_ The Secret to Instant Growth.pdf
byBEST IT SMM
 
Viva_Digitally_Online_Meeting_Presentation.pptx
byHubraSEO
 
WHMCS Croster 2.1.2 Nulled Pre Activated
byCroster 2.1.2 Nulled
 
BrightonSEO Oct 2025 - How Migrations Have Changed - Nick Samuel
byNick Samuel
 

BRKRST-3068 Troubleshooting Catalyst 2K and 3K.pdf

  • 2.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Catalyst 2K and 3K BRKRST-3068
  • 3.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Session Goals  Identify various system resources and monitor their usage.  Select the right steps to solve common access layer incidents.  Diagnose a former black-box with confidence. 3
  • 4.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Agenda  Architecture Overview  Areas of Troubleshooting Hardware components Forwarding 3750 3560 2960 4
  • 5.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Catalyst 2K/3K Architecture Overview:  Port ASICs, CPU, Memory, Stack Phy, TCAM, Switch Fabric  The number of interfaces per Port ASIC varies by platform. Memory CPU Stack PHY Port ASIC Port ASIC Port ASIC Switch Fabric Stack Phy Flash Serial Modular PHY 10/100 10G or 1G 12 Port PHY 12 Port PHY 12 Port PHY 12X1G 12X1G 24X1G POE 12 Port PHY 12X1G 12X1G 24X1G POE Two Stack Cables TCAM SRAM TCAM SRAM TCAM SRAM Stack errors High? Buffers? QoS TCAM usage Running out? Interface Flaps? 5
  • 6.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Agenda  Architecture Overview  Areas of Troubleshooting Hardware components Forwarding 3750 3560 2960 6
  • 7.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Link Issues  Is the link coming up as expected?  Are packets being sent and received on the port?  Are there errors on the port?  Link Issue: Failure for the physical interface to remain operational  Can be caused by a Layer 1 or Layer 2 problem  Layer 1: Interface PHY does media conversion (10/100/1000Mbps, 10G)  Layer 2: Interface PHY ensures 802.3 compliance 7
  • 8.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Port Status and Counters 3700X-1#sh int gig 1/0/1 GigabitEthernet1/0/1 is down, line protocol is down (notconnect) Hardware is Gigabit Ethernet, address is 70ca.9b2d.9f81 (bia 70ca.9b2d.9f81) MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX input flow-control is off, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output never, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec Traditional interface level statistics command 8
  • 9.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Link Not Coming Up  Verify the configured duplex and speed on both switch and attached host; fixing speed and duplex should be done on both sides  Upgrade the NIC drivers on the host to the latest version available from the vendor  Try a different cable/NIC and switchport to exclude faulty hardware Switch#show interfaces status | inc connected Gi1/0/1 connected trunk a-full 10 10/100/1000BaseTX Gi1/0/2 connected 101 a-full a-100 10/100/1000BaseTX Gi1/0/24 connected 1 a-full a-1000 10/100/1000BaseTX 9
  • 10.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public SFP Link Issue Prevention EEM Tcl Policy Use Case  1000 Base-T (copper) and 100 Base-FX SFPs have embedded PHYs, allowing speed and duplex to be configured on their respective interface  However, these settings are cleared as soon as the SFP is unplugged  If the same SFP type is re-inserted, its configuration is not recovered LinkUpApplyConfig.tcl can be downloaded at the following hyperlink: https://supportforums.cisco.com/docs/DOC-23267  LinkUpApplyConfig is a Tcl policy that monitors an SFP link-up event  Speed and duplex settings (in startup-config) are automatically re- applied to the SFP interface 1 2 1 2 EEM Tcl Policy 10
  • 11.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Checking Physical Cabling  Use the TDR feature on the port to determine possible cabling issues: miswiring or cable breaks  Interfaces will be brought down and up when run on active ports Switch# test cable-diagnostics tdr interface GigabitEthernet4/0/1 TDR test started on interface Gi4/0/1 A TDR test can take a few seconds to run on an interface Use 'show cable-diagnostics tdr' to read the TDR results. Switch# show cable-diagnostics tdr interface GigabitEthernet4/0/1 TDR test last run on: March 01 03:11:11 Interface Speed Local pair Pair length Remote pair Pair status --------- ----- ---------- ------------------ ----------- -------------------- Gi4/0/1 1000M Pair A 3 +/- 1 meters Pair A Normal Pair B 2 +/- 1 meters Pair B Normal Pair C 3 +/- 1 meters Pair C Normal Pair D 3 +/- 1 meters Pair D Normal 11 Open?
  • 12.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public *Mar 1 05:19:41.222: %SFF8472-5-THRESHOLD_VIOLATION: Gi4/1/1: Rx power high warning; Operating value: -4.2 dBm, Threshold value: -7.0 dBm. Link Issues: Checking Physical Cabling  Use the DOM feature on the link ports to determine possible optical transmission issues: dirty connectors, wrong attenuation, transceiver issues 12 Gi4/1/1 Gi1/1/1 Switch# show interface GigabitEthernet4/1/1 transceiver details Optical High Alarm High Warn Low Warn Low Alarm Receive Power Threshold Threshold Threshold Threshold Port (dBm) (dBm) (dBm) (dBm) (dBm) ------- ----------------- ---------- --------- --------- --------- Gi4/1/1 -4.2 + -3.0 -7.0 -27.9 -32.2 Switch# show interface GigabitEthernet1/1/1 transceiver ITU Channel not available (Wavelength not available), Transceiver is internally calibrated. ++ : high alarm, + : high warning, - : low warning, -- : low alarm. NA or N/A: not applicable, Tx: transmit, Rx: receive. mA: milliamperes, dBm: decibels (milliwatts). Optical Optical Temperature Voltage Tx Power Rx Power Port (Celsius) (Volts) (dBm) (dBm) --------- ----------- ------- -------- -------- Gi1/1/1 38.6 3.29 -2.0 -11.3
  • 13.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: MACsec Encryption Switch2# show macsec GigabitEthernet 2/1/4 MACsec is enabled Replay protect : enabled Replay window : 0 Include SCI : yes Cipher : GCM-AES-128 Confidentiality Offset : 0 Capabilities Max. Rx SA : 16 Max. Tx SA : 16 Validate Frames : strict PN threshold notification support : Yes Ciphers supported : GCM-AES-128 Transmit Secure Channels SCI : D0D0FDA045830000 Elapsed time : 00:43:26 SC Statistics Auth-only (0 / 0) Encrypt (157426 / 0) Receive Secure Channels SCI : EC4476E280950000 Elapsed time : 00:43:26 Current AN: 1 Previous AN: 0 Check Details on Encryption Tunnel negotiation on both link ends Switch1# show macsec GigabitEthernet 1/0/1 MACsec is enabled Replay protect : enabled Replay window : 0 Include SCI : yes Cipher : GCM-AES-128 Confidentiality Offset : 0 Capabilities Max. Rx SA : 16 Max. Tx SA : 16 Validate Frames : strict PN threshold notification support : Yes Ciphers supported : GCM-AES-128 Transmit Secure Channels SCI : EC4476E280950000 Elapsed time : 00:42:36 SC Statistics Auth-only (0 / 0) Encrypt (10207 / 0) Receive Secure Channels SCI : D0D0FDA045830000 Elapsed time : 00:42:36 Current AN: 1 Previous AN: 0 13 Symptom: Link not coming up
  • 14.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: MACsec Encryption IEEE 802.1ae Statistics computed by the MACsec PHY Switch# show macsec GigabitEthernet 1/0/1 ... SC Statistics Notvalid pkts 0 Invalid pkts 0 Valid pkts 193333 Late pkts 0 Uncheck pkts 0 Delay pkts 0 Port Statistics Ingress untag pkts 0 Ingress notag pkts 343 Ingress badtag pkts 0 Ingress unknownSCI pkts 0 Ingress noSCI pkts 0 Unused pkts 0 Notusing pkts 0 Decrypt bytes 20486316 Ingress miss pkts 79 14
  • 15.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Link Issues Troubleshooting Steps Commands Cabling issues test cable-diagnostics tdr interface show interface transceiver Interface not coming up show interface status show interface .. counters errors show macsec interface Command Summary 15
  • 16.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch Hardware Components: CPU Functions  Runs the IOS  Processes Control Plane traffic (LACP / PAgP / VTP / STP / CDP)  Processes packets that are not switched in Hardware Packets with IP options, Packets with expired TTL, ARP, Snooping, Software ACLs and SNMP Memory CPU Stack PHY Port ASIC Switch Fabric 10G or 1G TCAM High? 18
  • 17.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  CPU Utilisation can become high due to 2 reasons: • Processes taking up resources • Forwarded Network Traffic  Using CPU cycles is not a problem  6-8% is minimum - depending upon IOS Feature set  Normal or Expected CPU Utilisation 10-12% 3700X-1#sh proc cpu sor CPU utilization for five seconds: 46%/0%; one minute: 11%; five minutes: 9% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 337 5365 127 42244 38.17% 3.11% 0.64% 1 Virtual Exec 75 129004 32458 3974 1.55% 1.69% 1.61% 0 RedEarth Tx Mana 39 3741 142 26345 0.62% 0.06% 0.01% 0 Per-minute Jobs 118 7011 12552 558 0.31% 0.10% 0.07% 0 hpm counter proc 169 13991 1706 8201 0.15% 0.16% 0.15% 0 HQM Stack Proces CPU: Troubleshooting Processes Switch# sh proc cpu history 70 * * 60 ** * * 50 ** * * 40 *** * * * 30 ***** ** * 20 **##*#* ** # 10 ################## 0....5....1....1....2....2....3....3....4....4....5....5.... 0 5 0 5 0 5 0 5 0 5 19
  • 18.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public packets dropped before reaching the CPU Queue Switch# show platform port-asic stats drop Supervisor TxQueue Drop Statistics Queue 0: 0 ...... Queue 7: 10000 CPU: The 16 Different Queues  CPU buffer pools are named RxQ0 to RxQ15  Port ASIC can drop packets before reaching the CPU Q Check both locations (pools and asic queues) 0:rpc 1:stp 2:ipc 3:routing protocol 4:L2 protocol 5:remote console 6:sw forwarding 7:host 8:broadcast 9:cbt-to-spt 10:igmp snooping 11:icmp 12:logging 13:rpf-fail 14:dstats 15:cpu heartbeat 20
  • 19.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch# debug platform cpu-queues software-fwd-q *Mar 1 10:37:33.205 AEDT: SW-FWD-Q:IP packet: Local Port Fwding L3If:Vlan1 L2If:GigabitEthernet2/0/2 DI:0x2F, LT:7, Vlan:1 SrcGPN:56, SrcGID:56, ACLLogIdx:0x0, MacDA:c471.fe1e.f0c0, MacSA: 0007.7d75.88c0 IP_SA:14.160.38.1 IP_DA:14.160.38.130 IP_Proto:1 IP Opts TPFFD:D8C00038_00010001_00A00076-0000002F_E2C50000_00000000 CPU: Software Forwarding Queue (Q6)  For Traffic that hardware cannot process  SW forwarding performance is much lower than HW To debug any CPU Q SMAC of the host sending the traffic Physical interface where the traffic is coming in 21
  • 20.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: Best Practices Storm Control can help to protect CPU. Configuring Traffic Storm Control to avoid packets flood the LAN, creating excessive traffic and degrading network performance. (config-if)#storm-control broadcast level level[.level] (config-if)#storm-control action ? shutdown Shutdown this interface if a storm occurs trap Send SNMP trap if a storm occurs Example of Syslog message for high CPU *Mar 1 01:03:15.601: %SYS-1-CPURISINGTHRESHOLD: Threshold: Process CPU Utilisation (Total/Intr): 18%/0%, Top 3 processes(Pid/Util): 4/10%, 75/1%, 164/0% Configuring the CPU threshold can help you identifying when the CPU goes over certain limits Switch(config)# process cpu threshold type {total | process | interrupt} rising percentage interval seconds [falling fall-percentage interval seconds] 22
  • 21.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public %PLATFORM_RPC-0-RESOURCE_CRASH: System is unable to alloc memory for RPC TS: Memory Utilisation  Potential issues • Is Free steady? • Is Free steadily decreasing? Switch# sh memory statistics Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 2641D6C 81519252 31192204 50327048 49241540 48621848 I/O 7400000 12574720 8532852 4041868 3821068 4039616 Memory available now The lowest free since boot up Largest block switch can allocate FIB−2−FIBDOWN : CEF has been disabled due to a low memory condition. %SYS−2−MALLOCFAIL: Memory allocation of 1028 bytes failed from 0x601617A4, pool Processor, alignment 0 −Process= "IP Input", ipl= 2, pid= 21 %FRNTEND_CTRLR-1-SUB_I2C_ERR: Sub 0 reported 36B5B98 I2C errors %FRNTEND_CTRLR-2-SUB_INACTIVE: The front end controller 0 is inactive 23
  • 22.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public TS: Memory Utilisation  Run commands multiple times to create benchmark Switch# show processes memory sorted … PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 74539888 23738156 47199076 0 0 *Init* 0 0 3399716 17490880 1590292 10657136 553112 *Dead* 65 0 712620 27424 594488 0 0 Stack Mgr Notifi 324 0 19794764 19262624 539264 0 0 hulc running con 304 0 366680 344 370420 0 0 CEF: IPv4 proces 165 0 294516 2524 294516 0 0 HL2MCM 164 0 294460 2496 294460 0 0 HL2MCM 17 0 230568 0 240620 99792 0 EEM ED Syslog 11 0 228060 14940 226488 0 0 ARP Input … Is any process steadily increasing held memory? 24
  • 23.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public TCAM Utilisation  TCAM space is limited  Problem when Used Masks/Values = MAX Change SDM Template/optimise ACLs, Routing entries. Security ACLs Permit/d eny Layer 3 routing Switch# show platform tcam utilization CAM Utilization for ASIC# 0 Max Used Masks/Values Masks/values Unicast mac addresses: 784/6272 14/40 IPv4 IGMP groups + multicast routes: 144/1152 7/27 IPv4 unicast directly-connected routes: 784/6272 14/40 IPv4 unicast indirectly-connected routes: 2048/2048 2047/2047 IPv4 policy based routing aces: 0/0 0/0 IPv4 qos aces: 768/768 260/260 IPv4 security aces: 1024/1024 723/723 25
  • 24.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public TCAM Overload  An error message will get generated  Traffic forwarding will be done (partly) in Software  CPU utilisation will go up – packets punted to CPU for processing %ACLMGR-4-UNLOADING: Unloading ACL input label 1 VLAN interfaces 101 IPv4/Mac feature %ACLMGR-4-ACLTCAMFULL: ACL TCAM Full. Software Forwarding packets on Input label 1 on L3 L2 Switch# sh platform acl label 1 detail Unloaded due to lack of space: Switch# sh platform acl oacltcamfull Vlan oacl_tcam_full_bitmap notify_apps 101 0x 0 NOT-FULL Means ACL not fully programmed in TCAM 26
  • 25.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public TCAM: Switch Database Manager (SDM)  SDM defines how TCAM resources are allocated  Changing SDM template requires reboot  All stack members must use same SDM template Switch# show sdm prefer default "desktop default" template: The selected template optimizes the resources in the switch to support this level of features for 8 routed interfaces and 1024 VLANs. number of unicast mac addresses: 6K number of IPv4 IGMP groups + multicast routes: 1K number of IPv4 unicast routes: 8K number of directly-connected IPv4 hosts: 6K number of indirect IPv4 routes: 2K number of IPv4 policy based routing aces: 0 number of IPv4/MAC qos aces: 0.5K number of IPv4/MAC security aces: 1K 27
  • 26.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting CPU/Memory Troubleshooting Steps Commands Current CPU Utilisation show processes cpu sorted show processes cpu history Statistics for Packets Fwd to CPU show platform port-asic stats drop show controllers cpu-interface Details of packets received by CPU per ingress queue debug platform cpu-queues <queue> Memory Issues Show memory <> Show processes memory <> Command Summary 28
  • 27.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting TCAM Troubleshooting Steps Commands Utilisation show platform tcam utilization Check HW resource show platform acl oacltcamfull show platform acl label <> detail SDM Template show sdm prefer Command Summary 29
  • 28.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Stack# show switch H/W Current Switch# Role Mac Address Priority Version State ---------------------------------------------------------- *1 Master 0018.ba60.de00 15 1 Ready 2 Member 0018.ba60.ce00 14 1 Ready 3 Member 0016.9d0c.7500 1 2 Version Mismatch Troubleshooting Stacks  Conditions that can prevent a switch from joining a stack: • Incompatible IOS Versions between the stack members. • A defective stack cable • Not properly connected • Incomplete connection if only one stack cable is connected. • SDM Template mismatch  The following example shows a switch that can not join the stack: %STACKMGR−6−SWITCH_ADDED_VM 30
  • 29.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Stacks Version Mismatch  Software Version Mismatch ‒IOS version should be either the same or compatible ‒show version will show IOS version of all switches in a stack.  Switches with different Major Version numbers are incompatible and cannot exist in the same switch stack. ‒Occurs on switch member addition, or RMA replacement IOS Versions should match Major versions must match 3750E# show version Switch Ports Model SW Version SW Image ------ ----- ----- ---------- ---------- * 1 30 WS-C3750E-24TD 12.2(50)SE C3750E-UNIVERSAL-M 3 30 WS-C3750E-24PD 12.2(50)SE C3750E-UNIVERSAL-M 3750E# show platform stack manager all … H/W Current Switch# Role Mac Address Priority Version State ---------------------------------------------------------- *1 Master 001b.545f.2800 12 1 Ready 3 Member 001d.46be.7500 8 1 Ready 31
  • 30.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting: Stack Commands 3750# show switch detail Current Switch# Role Mac Address Priority State ------------------------------------------------------ 1 Slave 000c.30ae.4f00 9 Ready *2 Master 000d.bd5c.1680 15 Ready Stack Port Status Neighbors Switch# Port 1 Port 2 Port 1 Port 2 ------------------------------------------------------ 1 Ok Ok 2 2 2 Ok Ok 1 1 3750# show switch stack-ring activity Switch Frames sent to stack ring (approximate) ------------------------------------------------ 1 5781 2 4928 Total frames sent to stack ring : 10709 Note: these counts do not include frames sent to the ring by certain output features such as output SPAN and output ACLs. Commands to give stack details  Use the mode button on the switch to determine its stack switch number  LED on the port with the corresponding switch number will illuminate  For ex, if the switch is # 4 in the stack, port 4’s LED will light up 3750E# show switch stack-ring speed Stack Ring Speed : 32G Stack Ring Configuration: Full Stack Ring Protocol : StackWisePlus 32
  • 31.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting: Stack Commands 3750# sh switch Switch/Stack Mac Address : c471.fe1e.f080 H/W Current Switch# Role Mac Address Priority Version State ---------------------------------------------------------- 1 Member c471.fe23.3780 1 1 Ready *2 Master c471.fe1e.f080 1 1 Ready 3 Member 0000.0000.0000 0 1 Provisioned 3750# show switch stack-ports summary Switch#/ Stack Neighbor Cable Link Link Sync # In Port# Port Length OK Active OK Changes Loopback Status To LinkOK -------- ------ -------- -------- ---- ------ ---- --------- -------- 1/1 OK 3 50 cm Yes Yes Yes 1 No 1/2 Down None 50 cm No No No 0 No 3/1 Down None 50 cm No No No 0 No 3/2 OK 1 50 cm Yes Yes Yes 1 No Details on the stack ports, members 1 and 2 active 33
  • 32.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Stacking Troubleshooting Steps Commands Stack status show switch [detail] show platform stack manager show switch stack-ring <> show controllers utilization show switch stack-ports summary (New) Test Stack Ports switch <> stack port <> enable/disable From IOS 12.2(50) Command Summary 34
  • 33.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public GOLD (Generic Online Diagnostics) (config)# [no] diagnostic monitor interval { switch <1-9> } test { test-id | test-id-range | all } hh:mm:ss { ms <0-999> } { days <0-20> } diagnostic start {switch <1:9>} test {test-num | test range | all | basic | non-disruptive } Switch(config)#[no] diagnostic schedule { switch <1-9> } test { test-id | test-id-range | all } daily {hh:mm} On-Demand Health-Monitoring Scheduled To run Non-disruptive tests in the background Serves as HA trigger All diagnostics tests can be run on demand, for troubleshooting purposes. It can also be used as a pre-deployment tool. All diagnostic tests can be Scheduled, for verification and troubleshooting purposes Runtime diagnostics Run During System Bootup, Makes sure faulty hardware is taken out of service (POST = Power On Self Test) Boot-Up diagnostics show diagnostic post 35
  • 34.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public GOLD: Test Options OnDemand 3750E# show diagnostic content switch 1 Test Interval ID Test Name Attributes day hh:mm:ss.ms Threshold ==== ====================== ============ ========== ==== ======== 1) TestPortASICStackPortLoopback ---> B*N****I** 005 01:10:25.05 n/a 2) TestPortASICLoopback ----------------> B*D*X**IR* not configured n/a 3) TestPortASICCam -----------------------> B*D*X**IR* not configured n/a 4) TestPortASICRingLoopback ----------> B*D*X**IR* not configured n/a 5) TestMicRingLoopback ----------------> B*D*X**IR* not configured n/a 6) TestPortASICMem ----------------------> B*D*X**IR* not configured n/a 7) TestInlinePwrCtlr -----------------------> B*D*X**IR* not configured n/a What Tests Can I Run? 36
  • 35.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public GOLD: CLI OnDemand diagnostic start {switch <1:9>} test {test-num | test range | all | basic | non-disruptive } 3700X-1#diagnostic start switch 1 test all Diagnostic[Switch 1]: Running test(s) 2-6 will cause the switch under test to reload after completion of the test list. Diagnostic[Switch 1]: Running test(s) 2-6 may disrupt normal system operation Do you want to continue? [no]: yes Disruptive Test: Users will be prompted if the test causes a lose of stack connectivity: Switch 3: Running test(s) 2 will cause the switch under test to reload after completion of the test list. Switch 3: Running test(s) 2 may disrupt normal system operation Do you want to continue? [no]: Disruptive Test: Users will be prompted if the test causes stack partitioning: Switch 6: Running test(s) 2 will cause the switch under test to reload after completion of the test list. Switch 6: Running test(s) 2 will partition stack Switch 6: Running test(s) 2 may disrupt normal system operation Do you want to continue? [no]: Note: Tests Run to Completion (No Stop Command) 37
  • 36.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public GOLD: Results OnDemand 3750X# show diagnostic status shows what diagnostics are currently running 3750X# show diagnostic result switch 1 detail … Overall diagnostic result: PASS … 1) TestPortASICStackPortLoopback ---> . Error code ----------------------> 0 (DIAG_SUCCESS) … Last test execution time ----> Mar 01 1993 00:16:34 … Last test pass time ---------> Mar 01 1993 00:16:34 … 38
  • 37.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public On-Board Failure Logging (OBFL) 3750E/3560E  Provides “flight recorder” capability  Enabled by default  Collects operational data about switch and FRUs like PS, RPS & SFPs  Stores the data as a circular buffer on flash (2M). Older data is compressed with less detail  Each switch records its own OBFL data  Information can be seen with show logging onboard 39
  • 38.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Agenda  Architecture Overview  Areas of Troubleshooting Hardware components Forwarding 3750 3560 2960 40
  • 39.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  The normal L2 CAM behaviour in a switch is as follows… Learning Switch CAM Table MAC A C Port 1 3 A B C 1 2 3 A>B A>B A>B Source Mac address of frame are used to learn mac-address location and build L2 table – Learning A on port 1 Destination Mac address of frame are used as lookup key in mac-address-table to know where to forward. A destination mac address miss will cause Unicast flooding B is unknown hence packet is flooded 41
  • 40.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  The normal CAM behaviour in a switch is as follows… Learning … and Aging Switch CAM Table MAC A B C Port 1 2 3 A B C 1 2 3 B>A B>A Reply from B is coming back to A lead to learning B in Cam table A is already known hence sent only to destination port. Each entry is aged out when inactive for 5 min (default) 42
  • 41.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Unicast Forwarding  Symptom: Host cannot reach server  Steps Layer 1 operational between host and switch? Switch receiving traffic on that interface? MAC address learned? MAC address of next hop correct? Spanning tree state forwarding? Check HW programming  Consider possibilities  Create and execute action plan 43
  • 42.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  Step 1: Verify if the link is up  Step 2: Verify if the port is in the right vlan and is forwarding  Step 3: Check if the packets are being received/sent on the port Switch# show spanning-tree interface Gi1/0/3 Vlan Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- ----------------------- VLAN010 Desg FWD 19 128.2 P2p L2 Forwarding: Troubleshooting - 1 Switch# show interface Gi1/0/3 status Port Name Status Vlan Duplex Speed Type Gi1/0/3 connected 10 a-full a-100 10/100/ 1000BaseTX Switch# show interfaces gigabitEthernet 1/0/3 counters Port InOctets InUcastPkts InMcastPkts InBcastPkts Gi1/0/3 2108289 48 0 6813 Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts Gi1/0/3 36817803 48229 252940 72564 44
  • 43.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  Step 4a: Verify if the Mac-address is correctly learned on the port  Step 4b: Verify if the destination Mac-address is learned on the switch on the expected port Switch# sh mac address-table interface gigabitEthernet 1/0/3 Mac Address Table ------------------------------------------- Vlan Mac Address Type Ports ---- ----------- -------- ----- 10 00b1.a3d3.4321 DYNAMIC Gi1/0/3 Total Mac Addresses for this criterion: 1 Layer 2 Forwarding: MAC Checking Switch# sh mac address-table dynamic address 00b1.a3d3.1234 Mac Address Table ------------------------------------------- Vlan Mac Address Type Ports ---- ----------- -------- ----- 10 00b1.a3d3.1234 DYNAMIC Gi1/0/4 Total Mac Addresses for this criterion: 1 45
  • 44.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch#show spanning-tree vlan 10 VLAN0010 Spanning tree enabled protocol ieee Root ID Priority 32778 Address 0003.fd6b.0700 This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32778 (priority 32768 sys-id-ext 10) Address 0003.fd6b.0700 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Aging Time 300 sec Interface Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- ------------------------ Gi1/0/3 Desg FWD 4 128.3 P2p Gi1/0/4 Desg FWD 4 128.4 P2p Edge Layer 2 Forwarding: Spanning Tree  Step 5: Spanning tree state forwarding in software? Interfaces are Forwarding 46
  • 45.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Layer 2 Forwarding: Advanced Techniques  Step 6: Use show platform forward to verify the hardware programming – find Egress Interface Destination Interface Incoming interface src mac dst mac Switch# show platform forward gigabitEthernet 1/0/3 00b1.a3d3.4321 00b1.a3d3.1234 Ingress: Global Port Number: 3, lpn: 1 ASIC Number: 6 Source Vlan Id: Real 10, Mapped 2. L2EncapType 0, L3EncapType 3 Hashes: L2Src 0x00 L2Dst 0x0B L3Src 0x00 L3Dst 0x0B Lookup Key-Used Index-Hit A-Data … ========================================== Egress: ASIC 6, switch 1 Source Vlan Id: Real 10, Mapped 2. L2EncapType 0, L3EncapType 3 portMap 0x4, non-SPAN portMap 0x4 Output Packets: ------------------------------------------ GigabitEthernet1/0/4 Packet 1 Lookup Key-Used Index-Hit A-Data OutptACL 30_00F00000_00001234-00_00000000_00004321 01FFC 01000000 Port Vlan SrcMac DstMac Cos Dscpv Gi1/0/4 0010 00b1.a3d3.4321 00b1.a3d3.1234 47
  • 46.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Unicast Forwarding Troubleshooting Steps Commands Verify Layer 1 is operational between host and switch show interface <interface> status Verify switch receives traffic on the interface show interfaces <interface> counters show interfaces <interface> counters errors MAC Address changes show spanning-tree vlan <> detail 48
  • 47.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Steps Commands Verify host MAC address is learned show mac address-table interface <interface> show mac address-table dynamic address <mac> Verify spanning tree state is forwarding show spanning-tree vlan <vlan> Verify MAC address of next hop is correct Local and remote switches: show mac address-table vlan <vlan> Verify other features are not preventing traffic flow Show port-security interface <interface> show ip access-lists interface <interface> Show hardware programming for MAC Address show platform forward <src interface> <src-mac> <dest-mac> Troubleshooting Unicast Forwarding 49
  • 48.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Layer 3 IP Unicast Routing  Use the switch to debug end to end IP issues  Verify IP reachability from switch end host  Verify destination reachability from the switch  Verify hardware forwarding from source to destination (and back) 3750 370 Source IP: 100.1.1.2 Mac: 0018.ba88.1fc1 Gi1/0/1 Gi1/0/2 Destination IP: 172.16.100.100 VLAN:101 IP: 100.1.1.1 Mac: 000f.f7e8.e042 Vlan:100 IP: 10.1.1.1 Mac :000f.f7e8.e041 3750 3750 50
  • 49.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public L3: Verify Source Reachability 3750# ping 100.1.1.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 100.1.1.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms 3750# ping 100.1.1.2 source lo0 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 100.1.1.2, timeout is 2 seconds: Packet sent with a source address of 99.1.1.1 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/maz = 1/4/9 ms 3750# sh ip arp vlan 101 Protocol Address Age (min) Hardware Addr Type Interface Internet 100.1.1.1 - 000f.f7e8.e042 ARPA Vlan101 Internet 100.1.1.2 23 0018.ba88.1fc1 ARPA Vlan101 3750# sh mac address-table address 0018.ba88.1fc1 Mac Address Table ------------------------------------------ Vlan Mac Address Type Ports ---- ----------- -------- ----- 101 0018.ba88.1fc1 DYNAMIC Gi1/0/2 Total Mac Addresses for this criterion: 1 Change source IP to loopback 51
  • 50.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public L3: Verify Source Reachability  Verify packets from the source are getting to the CPU Packet arriving on CPU queue 7 (host) 3750#show platform for Gi1/0/2 0018.ba88.1fc1 000f.f7e8.e042 ip 100.1.1.2 100.1.1.1 icmp 0 0 Ingress: Global Port Number: 1, lpn: 3 Asic Number: 1 Source Vlan Id: Real 101, Mapped 9. L2EncapType 0, L3EncapType 0 ... Station Descriptor: 00B00000, DestIndex: 00B0, RewriteIndex: 0000 ========================================== <output removed> Output Packets: ========================================== Egress: Asic 0, switch 2 CPU queues: 7 14. 52
  • 51.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public L3: Verify Destination Reachability Switch# sh ip route 172.16.100.100 Routing entry for 172.16.100.0/24 Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 1 Last update from 10.1.1.2 on Vlan100, 00:08:54 ago Routing Descriptor Blocks: * 10.1.1.2, from 100.1.1.2, 00:08:54 ago, via Vlan100 Route metric is 20, traffic share count is 1 Switch # sh ip arp 10.1.1.2 Protocol Address Age (min) Hardware Addr Type Interface Internet 10.1.1.2 9 0018.ba88.1fc1 ARPA Vlan100 Switch# ping 172.16.100.100 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.16.100.100, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms Switch# ping 172.16.100.100 source vlan 101 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.16.100.100, timeout is 2 seconds: Packet sent with a source address of 192.168.100.1 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms 53
  • 52.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public L3: Verify Hardware Forwarding  Show platform forward to verify HW programming 3750# show plat for Gi1/0/2 0018.ba88.1fc1 000f.f7e8.e042 ip 100.1.1.2 172.16.100.100 icmp 0 0 Ingress:Global Port Number: 2, lpn: 2 ASIC Number: 1 Source Vlan Id: Real 101, Mapped 10. L2EncapType 0, L3EncapType 0 <snip> <snip> Output Packets: GigabitEthernet1/0/1 Packet 1 Lookup Key-Used Index-Hit A-Data OutptACL 50_AC106464_64010102-00_01000000_00000100 01FFE 03000000 Port Vlan SrcMac DstMac Cos Dscpv Gi1/0/1 0100 000f.f7e8.e041 0018.ba88.1fc1 54
  • 53.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting L3 Troubleshooting Steps Commands Verify source reachability ping show ip arp vlan sh mac address-table address Verify destination reachability show ip route show ip arp ping Verify HW programming show platform forward <ingress intf> <srcmac> <dstmac> ip <srcip> <dstip> Command Summary 55
  • 54.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 3750# show ip igmp snooping mrouter vlan 100 Vlan ports 100 Gi1/0/1(dynamic) # debug ip igmp snooping router *Mar 1 03:33:44.075: IGMPSN: router: Received non igmp pak on Vlan 100, port Gi1/0/1 *Mar 1 03:33:44.075: IGMPSN: router: PIMV2 Hello packet received in 100 *Mar 1 03:33:44.075: IGMPSN: router: Is a router port on Vlan 100, port Gi1/0/1 *Mar 1 03:33:44.075: IGMPSN: router: Learning port: Gi1/0/1 as rport on Vlan 100 # debug platform cpu-queue igmp-snooping *Mar 1 03:39:09.469: Pak recvd on IGMP-SNOOP-Q: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0x12FC, LT:7, Vlan:100 SrcGPN:24, SrcGID:24, ACLLogIdx:0x0, MacDA:0100.5e00.0005, MacSA: 0011.21e6.5a40 IP_SA:10.160.16.1 IP_DA:224.0.0.5 IP_Proto:89 TPFFD:E841C018_00640064_00A0005E-000012FC_43330000_00000000 IGMP Multicast Router Port  Gets learned dynamically by listening either to PIM/DVMRP or to CGMP packets  Mrouter port should be learned dynamically View pkts to CPU 56
  • 55.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public IGMP Client Join IGMP Joins received are sent to the CPU to be processed Switch# debug ip igmp snooping group *Mar 6 04:19:39.175: IGMPSN: Received IGMPv2 Report for group 239.100.100.100 received on Vlan 101, port Gi1/0/2 *Mar 6 04:19:39.175: IGMPSN: router: Is not a router port on Vlan 101, port Gi1/0/2 *Mar 6 04:19:39.175: IGMPSN: group: Skip client info adding - ip 10.101.1.100, port_id Gi1/0/2, on vlan 101 *Mar 6 04:19:39.175: IGMPSN: MCAST IP address 239.100.100.100, MAC address 0100.5e64.6464 *Mar 6 04:19:39.175: IGMPSN: Can not Locate gce 0100.5e64.6464, on Vlan 101 *Mar 6 04:19:39.175: IGMPSN: MCAST IP address 239.100.100.100, MAC address 0100.5e64.6464 *Mar 6 04:19:39.175: IGMPSN: Can not Locate gce 0100.5e64.6464, on Vlan 101 *Mar 6 04:19:39.175: IGMPSN: mgt: created gce 0100.5e64.6464, on Vlan 101 *Mar 6 04:19:39.175: l2mcm_group_create: creating a group 239.100.100.100 on vlan 101, dummy NO *Mar 6 04:19:39.175: l2mcm_group_create: timer stop: vlan 101, group 239.100.100.100 *Mar 6 04:19:39.175: IGMPSN: mgt: created group 239.100.100.100, on Vlan 101 *Mar 6 04:19:39.175: IGMPSN: mgt: Vlan 101 gce 0100.5e64.6464 add port Gi1/0/2 *Mar 6 04:19:39.175: L2MM: setting Gi1/0/2 in gce->mbr_blist 57
  • 56.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public IP Multicast Routing  Verify PIM is working fine (not covered in this session)  Verify client is correctly joined via IGMP  Verify the switch is routing the flow correctly Switch# show ip mroute 239.100.100.100 10.99.1.100 IP Multicast Routing Table <output removed> (10.99.1.100, 239.100.100.100), 11:32:59/00:02:56, flags: JT Incoming interface: Vlan100, RPF nbr 10.100.1.1 Outgoing interface list: Vlan101, Forward/Sparse-Dense, 11:32:59/00:02:22 Switch# show ip igmp snooping groups vlan 101 239.100.100.100 Vlan Group Type Version Port List ----------------------------------------------------------------------- 101 239.100.100.100 igmp v2 Gi1/0/2 58
  • 57.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public IP Multicast Routing  show forward can be used to verify if the ASICs are setup correctly to route the multicast flow Switch# show platform forward Gig 1/0/1 vlan 100 18.ba88.1fc2 0100.5e64.6464 ip 10.99.1.100 239.100.100.100 udp 0 0 Ingress: Global Port Number: 1, lpn: 3 ASIC Number: 1 Source Vlan Id: Real 100, Mapped 9. L2EncapType 0, L3EncapType 0 <output removed> Output Packets: GigabitEthernet1/0/2 Packet 1 Lookup Key-Used Index-Hit A-Data OutptACL 50_EF646464_0A630164-00_41000000_0000A87E 01FFE 03000000 Port Vlan SrcMac DstMac Cos Dscpv Gi1/0/2 0101 000f.f7e8.e042 0100.5e64.6464 59
  • 58.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Multicast Troubleshooting Steps Commands IGMP sh ip igmp snooping mrouter vlan debug ip igmp snooping mrouter debug platform cpu-queue igmp- snooping debug ip igmp snooping group L3 Multicast sh ip mroute sh ip igmp snooping groups vlan show platform forward Command Summary 60
  • 59.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Cisco Catalyst 3750 QoS Overview Classification • Inspect incoming packets • Based on ACLs or configuration, determine classification label Policing • Ensure conformance to a specified rate • On an aggregate or individual flow basis • Up to 256 policers per Port ASIC • Support for rate and burst Marking • Act on policer decision • Reclass or drop out-of-profile Egress Queue/ Schedule Congestion Control • Four SRR queues/port shared or shaped servicing • One queue is configurable for strict priority servicing • WTD for congestion control (three thresholds per queue) • Egress queue shaping • Egress port rate limiting Ingress Queue/ Schedule Congestion Control • Two queues/port ASIC shared servicing • One queue is configurable for strict priority servicing • WTD for congestion control (three thresholds per queue) • SRR is performed Ingress Egress Policer Policer Marker Policer Policer Marker Marker Marker SRR SRR Classify Traffic Stack Ring Egress Queues Ingress Queues 61
  • 60.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public QoS Troubleshooting - Ingress  10,000 packets were received, DSCP value 34  1,467 packets were in profile  8,533 were dropped due to exceeding the policer 10000 IP packets with DSCP 34 access dot1q 3750 Gig 1/0/2 Gig 1/0/1 Switch# show mls qos interface gigabit 1/0/2 statistics GigabitEthernet1/0/2 (All statistics are in packets) dscp: incoming ------------------------------- 0 - 4 : 0 0 0 0 0 30 - 34 : 0 0 0 0 10000 ... Policer: Inprofile: 1467 OutofProfile: 8533 62
  • 61.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch#sh mls qos interface gigabitEthernet 1/0/1 statistics GigabitEthernet1/0/1 (All statistics are in packets) <output removed> dscp: outgoing ------------------------------- <output removed> 25 - 29 : 0 0 0 0 0 30 - 34 : 0 0 0 0 1467 <output removed> QoS Troubleshooting - Egress  1467 packets were in profile and made it to the egress port  DSCP is 34 10000 IP packets with DSCP 34 access dot1q 3750 Gig 1/0/2 Gig 1/0/1 63
  • 62.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch#sh mls qos interface gigabitEthernet 1/0/1 statistics GigabitEthernet1/0/1 (All statistics are in packets) <output removed> 0 – 4 : 1467 0 0 0 0 30 - 34 : 0 0 0 0 0 QoS Troubleshooting – Egress (2)  1467 packets were in profile and made it to the egress port but with DSCP 0 instead of 34.  Possible reasons: Attached service policy does not mark or trust dscp value Traffic is being routed via the CPU 10000 IP packets with DSCP 34 access dot1q 3750 Gig 1/0/2 Gig 1/0/1 64
  • 63.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public QoS Troubleshooting - Egress Q Maps  10000 packets are received and will egress on Q4, threshold 1 10000 IP packets with DSCP 34 100Mb/s 10Mb/s Gig 1/0/2 Gig 1/0/1 3750 Switch# show mls qos maps dscp-output-q Dscp-outputq-threshold map: d1 :d2 0 1 2 3 4 5 6 7 8 9 ------------------------------------------------------------ 0 : 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 1 : 02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01 2 : 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 3 : 03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 4 : 01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01 5 : 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 6 : 04-01 04-01 04-01 04-01 Switch# show mls qos interface gi 1/0/2 statistics GigabitEthernet1/0/2 (All statistics are in packets) dscp: incoming ------------------------------- 0 - 4 : 0 0 0 0 0 30 - 34 : 0 0 0 0 10000 65
  • 64.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 10000 IP packets with DSCP 34 100Mb/s 10Mb/s Gig 1/0/2 Gig 1/0/1 QoS Troubleshooting – Egress Queue Ts  1080 packets will egress on Q4, threshold 1  Remaining pkts dropped 3750 Switch# show mls qos interface Gig 1/0/1 statistics dscp: outgoing ------------------------------- 0 - 4 : 0 0 0 0 0 30 - 34 : 0 0 0 0 1080 ... output queues enqueued: queue: threshold1 threshold2 threshold3 ----------------------------------------- queue 0: 2 0 0 queue 1: 0 6 4560 queue 2: 0 0 0 queue 3: 1080 0 0 output queues dropped: queue: threshold1 threshold2 threshold3 ----------------------------------------- queue 0: 0 0 0 queue 1: 0 0 0 queue 2: 0 0 0 queue 3: 8920 0 0 66
  • 65.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public QoS Troubleshooting - Port-ASIC  10000 packets were received, 8920 were dropped on egress 10000 IP packets with DSCP 34 100Mb/s 10Mb/s Viewing Egress Congestion (another way) with port-asic command Gig 1/0/2 Gig 1/0/1 3750 Switch# show platform port-asic stats drop gigabitEthernet 1/0/1 Interface Gi1/0/1 TxQueue Drop Statistics Queue 0 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 … Queue 3 Weight 0 Frames 8920 Weight 1 Frames 0 Weight 2 Frames 0 67
  • 66.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch# show mls qos int gi1/0/1 buffers GigabitEthernet1/0/1 The port is mapped to qset : 1 The allocations between the queues are : 25 25 25 25 Switch# show mls qos queue-set Queueset: 1 Queue : 1 2 3 4 ---------------------------------------------- buffers : 25 25 25 25 threshold1: 200 200 100 100 threshold2: 200 200 100 100 QoS Troubleshooting - Buffer Tuning  Queue-sets define the buffer allocation  Default values can be modified  2 Queue-sets are available  Reserved - how many buffers will be reserved for this port ‒Default Queue-set values listed below Identifies Queue-set assigned to interface Dropped on this Queue and Threshold Tuning Buffers and Thresholds to fix congestion 68
  • 67.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public QoS Troubleshooting - Buffer Tuning (2) 400 IP packets with DSCP 34 100Mb/s 10Mb/s Packet drops with current Queue-set configuration No additional Packet drops after Queue-set change Threshold increased to 300 3750 Gig 1/0/2 Gig 1/0/1 Switch# show platform port-asic stats drop gigabitEthernet 1/0/1 Interface Gi1/0/1 TxQueue Drop Statistics Queue 3 Weight 0 Frames 8920 Switch(config)# mls qos queue-set output 1 threshold 4 300 300 50 400 Switch# show mls qos queue-set Queueset: 1 Queue : 1 2 3 4 ---------------------------------------------- buffers : 25 25 25 25 threshold1: 100 100 100 300 threshold2: 100 100 100 300 reserved : 50 50 50 50 maximum : 400 400 400 400 Switch# show platform port-asic stats drop gigabitEthernet 1/0/1 Interface Gi1/0/1 TxQueue Drop Statistics Queue 3 Weight 0 Frames 8920 69
  • 68.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting QoS Issues Troubleshooting Steps Commands Check for Errors Ingress and Egress ports show mls qos interface <> stats Check Queue mapping show mls qos maps dscp-output-q Check Egress Queue details show platform port-asic stats drop <> Check and tune buffers show mls qos queue-set mls qos queue-set output <> threshold Command Summary 70
  • 69.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Making Life Easier…  Review open caveats sections in release notes  Search Bug Toolkit for known issues  Reference Output Interpreter to decode command output  Reference the Error and System Messages for recovery procedures 71
  • 70.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Cat 2K-3K Troubleshooting Summary  Know your network, have baselines: CPU, Memory, TCAM Is the value normal?  Check the logs Error messages? Warnings?  Follow the packet Use the architecture to understand the flow Narrow down possibilities  Check overall health Is the HW OK? Are the members of the stack ready? ‘show post’ – to view results of last self check on bootup 72
  • 71.
  • 72.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Complete Your Online Session Evaluation Give us your feedback and receive a Cisco Live 2013 Polo Shirt! Complete your Overall Event Survey and 5 Session Evaluations.  Directly from your mobile device on the Cisco Live Mobile App  By visiting the Cisco Live Mobile Site www.ciscoliveaustralia.com/mobile  Visit any Cisco Live Internet Station located throughout the venue Polo Shirts can be collected in the World of Solutions on Friday 8 March 12:00pm-2:00pm Don’t forget to activate your Cisco Live 365 account for access to all session material, 74 communities, and on-demand and live activities throughout the year. Log into your Cisco Live portal and click the "Enter Cisco Live 365" button. www.ciscoliveaustralia.com/portal/login.ww
  • 73.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 75
  • 74.
  • 75.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Local Switching and Stacking • With StackWise, locally destined packets must traverse the entire stack ring. Relevant to 3750v2 StackWise StackWise Plus Local Switching With StackWise Plus, whether in a homogeneous or mixed-hardware stack, locally destined packets on an “E” or “X” series switch are never put on the stack ring. 77
  • 76.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU CPU CPU TCAMs TCAMs TCAMs MAC Address Management Distributed MAC learning  Stack members learn MAC addresses and updates TCAM entries  System synchronises MAC address tables across the stack  How it is distributed: ‒ A switch learns an address and sends a message to other switches in the stack ‒ Learning an address that was previously learned on a different port (either same or different switch) is considered as move MAC B MAC A 78
  • 77.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Tips…  Enable NTP to troubleshoot across switches ntp server <ip>  When debugging send the output to the buffer, not to the console and include date and time in messages configure terminal ‒ no logging console ‒ logging buffered 128000 ‒ service timestamps log datetime localtime msec show-timezone ‒ service timestamps debug datetime localtime msec show-timezone 79
  • 78.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Tips…  Include comments on the console as reminders ‒ C3750#!!! Comments here  Execute ‘show’ command from ‘config term’ mode ‒ C3750(config)# do show running int Gi1/0/1  Session to another switch member ‒ C3750#session <member #> or C3750#remote command <1-9|all> “IOS command” 80
  • 79.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU Utilisation Troubleshooting  High CPU Utilisation is problematic when: ‒Delays in forwarding of network traffic ‒Catalyst switch unable to respond to network problems in timely fashion ‒Switch management can become blocked, as CPU does not respond  Baseline CPU Utilisation varies by Model ‒Catalyst 2960, 3560, 3560G: ~6% (non-stacked models) ‒Catalyst 3750, 3750G: ~7% (stacked) ‒Catalyst 3750E: ~9% (stacked) ‒Catalyst 3750X: ~22% (stacked) ‒Catalyst 2960S: ~20% (stacked or non-stacked) ‒Feature set (LAN BASE, IP BASE, or IP SERVICES) will impact CPU util as well 81
  • 80.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: Troubleshooting Processes  Process Utilisation  Processes taking up resources  For ex “show tech” causes the virtual exec process to use some CPU resources  Capturing process utilisation at the “right” moment is key for identifying the cause  Traffic Forwarding  High component of utilisation due to interrupts  Data traffic not forwarded by ASIC  Excessive Control Plane / Management traffic:  DoS attacks (TTL=1)  SVI ping test  Requires inspecting CPU queues and ASIC 82 High CPU Utilisation is Due to:
  • 81.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: Layer 2 Control Protocol Queues  STP has its own queue – Queue 1  Layer 2 protocols queue for the rest – Queue 4 ‒ CDP , PAgP, LACP, DTP, LLDP, UDLD  Drops on these queues 1 or 4 can cause instability on the network Switch# show controllers cpu-interface cpu-queue-frames retrieved dropped invalid hol-block stray ----------------- ---------- ---------- ---------- ---------- ----- rpc 132917740 0 0 0 0 stp 31879262 0 23288714 0 0 ipc 10746915 0 0 0 0 routing protocol 267 0 0 0 0 L2 protocol 424610 0 0 0 0 remote console 1121711 0 105531 0 0 sw forwarding 0 0 0 0 0 host 345 0 0 0 0 broadcast 13931 0 55724 0 0 cbt-to-spt 0 0 0 0 0 igmp snooping 0 0 0 0 0 icmp 0 0 0 0 0 logging 0 0 0 0 0 rpf-fail 0 0 0 0 0 dstats 132935598 0 0 0 0 cpu heartbeat 82903147 0 0 0 0 83
  • 82.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: Routing Protocol Queue (Q3)  Receives all traffic for routing protocols, like BGP, OSPF, EIGRP, HSRP, etc.  Debug traffic received by CPU. ‒ In case below “routing-protocol-q” is shown ‒ Packet ingress intf, Dest MAC, SrcMAC, Dest IP, Src IP are shown Switch# debug platform cpu-queues routing-protocol-q Switch# debug standby HSRP debugging is on *Mar 6 00:47:39.260: RT-Q:Queued: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0x12FC, LT:7, Vlan:100 SrcGPN:1, SrcGID:1, ACLLogIdx:0x0, MacDA:0100.5e00.0002, MacSA: 0018.ba88.1fc1 IP_SA:10.1.1.2 IP_DA:224.0.0.2 IP_Proto:17 *Mar 6 00:47:39.260: HSRP: Vl100 Grp 0 Hello in 10.1.1.2 Standby pri 100 vIP 10.1.1.55 84
  • 83.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch# debug platform cpu-queues host-q *Mar 6 00:01:46.648: Host-Q:Queued L3If: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0xB0, LT:7, Vlan:100 SrcGPN:489, SrcGID:488, ACLLogIdx:0x0, MacDA:000f.f7e8.e041, MacSA: 0018.ba88.1fc1 IP_SA:10.1.1.2 IP_DA:10.1.1.1 IP_Proto:1 CPU: Host Queue (Q7)  Used for all unicast traffic sent to the switch. ‒ TACACS, SSH, telnet, ping, SNMP 85
  • 84.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: Host Queue (Q7) – Drops  Show buffer shows current buffer usage (RxQ7)  When free buffers reaches below watermark(32), throttling might occur, resulting in packet drops  slow responsiveness to network management Misses equals drops 86 Switch# debug platform cpu-queues host-q *Mar 6 00:01:46.648: Host-Q:Queued L3If: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0xB0, LT:7, Vlan:100 SrcGPN:489, SrcGID:488, ACLLogIdx:0x0, MacDA:000f.f7e8.e041, MacSA: 0018.ba88.1fc1 IP_SA:10.1.1.2 IP_DA:10.1.1.1 IP_Proto:1 TPFFD:DC0001E9_00000064_00B00076-000000B0_A68A0000_00000000 Switch# show buffer | begin RxQ7 RxQ7 buffers, 2040 bytes (total 192, permanent 192): 64 in free list (0 min, 192 max allowed) 294 hits, 0 misses
  • 85.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: ICMP Queue (Q11)  Receives all traffic for which an ICMP message needs to be generated (excluding PING)  Receives a copy of the traffic for which an ICMP packet needs to be generated. Hardware forwarding of the packet still occurs Switch# debug ip icmp Switch# debug platform cpu-queues icmp-q *Mar 9 21:34:30.695: ICMP-Q:Queued to Process, use GW:10.1.1.3: Remote Port Blocked L3If:Vlan100 L2If:GigabitEthernet4/0/1 DI:0xB4, LT:7, Vlan:100 SrcGPN:163, SrcGID:163, ACLLogIdx:0x0, MacDA:0018.ba88.1fc1, MacSA: 000f.f7e8.e041 IP_SA:10.1.1.1 IP_DA:77.1.1.1 IP_Proto:1 *Mar 9 21:34:30.695: ICMP: redirect sent to 10.1.1.1 for dest 77.1.1.1, use gw 10.1.1.3 87
  • 86.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: ICMP Queue (Q11)  Symptoms: ‒ relatively high CPU (due to throttling mechanism it won’t reach 99%) of which majority is due to interrupts ‒ Low process utilisation ‒ ICMP Queue heavily utilised 88 ICMP Unreachables Example Switch# show processes cpu sorted CPU utilization for five seconds: 53%/47%; one minute: 31%; five minutes: 18% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 149 397089 3879429 102 0.63% 0.34% 0.45% 0 Spanning Tree 112 325474 117735 2764 0.31% 0.15% 0.09% 0 HRPC qos request Switch# clear controllers cpu Switch# show controllers cpu-interface | include icmp icmp 133148 0 0 0 0
  • 87.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public CPU: ICMP Queue (Q11)  Causes: ‒ High amount of traffic is dropped because of a “deny” statement in an ACL ‒ CPU is interrupted to send ICMP unreachable packets back to the source  Solution ‒ Disable ICMP unreachables on the ingress interface 89 ICMP Unreachables Example Switch(config)# interface GigabitEthernet1/0/2 Switch(config-if)# no ip unreachable Switch(config-if)# end
  • 88.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch# clear controllers cpu Switch# show controllers cpu-interface | include sw forwarding sw forwarding 71558 0 0 0 0 Switch# debug platform cpu-queues software-fwd-q SW-FWD-Q:Consumed by SW-Bridging: Remote Port Blocked L3If:Vlan101 L2If:GigabitEthernet1/0/2 DI:0x2FD, LT:7, Vlan:101 SrcGPN:2, SrcGID:2, ACLLogIdx:0x0, MacDA:000f.f7e8.e042, MacSA: 0000.00bb.87df IP_SA:10.101.1.100 IP_DA:10.99.1.100 IP_Proto:255 CPU: Software Forwarding Queue (Q6)  For Traffic that hardware cannot process  SW forwarding performance is much lower than HW  Requires ASIC forwarding troubleshooting 90 Causing High CPU Utilisation
  • 89.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public  CPU utilisation sustained below 50% will not cause problems  3750-X and 2960-S will have a higher CPU utilisation ‒ It is normal around 30% and 20% respectively  Example of Syslog message for high CPU ‒002182: *Jul 20 04:23:36: %SYS-1- CPURISINGTHRESHOLD: Threshold: Process CPU Utilization(Total/Intr): 9%/0%, Top 3 processes(Pid/Util): 214/3%, 153/0%, 159/0%  Sorting the output is better than filtering the output with “exclude 0.00%” because that will exclude processes that you want to see. ‒Switch# show process cpu sorted CPU Utilisation: Summary 92
  • 90.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public TS: I/O Memory Buffers  I/O memory for incoming CPU bound packets  Used by Routers for control and data packets ‒ On only control packets  Shows CPU bound packets ‒ Not HW switched packets Switch# show buffers Buffer elements: 1679 in free list (500 max allowed) 27109526 hits, 0 misses, 1641 created Public buffer pools: Small buffers, 104 bytes (total 50, permanent 50, peak 181 @ 3w5d): 49 in free list (20 min, 150 max allowed) 129877853 hits, 141 misses, 390 trims, 390 created 0 failures (0 no memory) Middle buffers, 600 bytes (total 25, permanent 25, peak 94 @ 7w0d): 25 in free list (10 min, 150 max allowed) 616791 hits, 54 misses, 162 trims, 162 created 0 failures (0 no memory) . . . 93
  • 91.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Switch# show interfaces GigabitEthernet 1/0/1 counters errors Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards Gi1/0/1 0 0 0 0 0 0 Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants Gi1/0/1 0 0 0 0 0 0 0 Switch# show interfaces counters errors Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards Gi1/0/1 0 0 0 0 0 0 Gi1/0/2 0 0 0 0 0 0 <snip> Gi2/0/12 0 0 0 0 0 0 Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants Gi1/0/1 0 0 0 0 0 0 0 Gi1/0/2 0 0 0 0 0 0 0 Link Issues: What Kind of Errors? See slides in the appendix for Error Explanation 94
  • 92.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: MACsec Encryption (1/3)  MACsec encryption is performed in the PHY at line rate  More efficient than L3 or application encryption  Overhead of 32 bytes per packet might cause drops in the PHY  Recommendation: rate-limit egress interface at 85% to benefit from ASIC buffering and QoS 95 Symptom: Traffic drops Switch(config)# interface gigabitethernet2/0/1 Switch(config-if)# srr-queue bandwidth limit 85
  • 93.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Checking Physical Cabling 96 Gi4/1/1 Gi1/1/1 dBm -2.0 -4.2 Fibre Attenuation 1-3 dBm / km Connectors & Patchcord Loss ~ 1 dBm Wrong fibre type! MM instead of SM Loss Budget Analysis (Northbound) Insert a 5 dB attenuator to solve the problem Dirty Connector
  • 94.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Checking Physical Cabling  Use the DOM feature on the port to determine possible optical transmission issues: dirty connectors, wrong attenuation, transceiver issues 97 Gi4/1/1 Gi1/1/1 Switch# show interface GigabitEthernet4/1/1 transceiver details Optical High Alarm High Warn Low Warn Low Alarm Transmit Power Threshold Threshold Threshold Threshold Port (dBm) (dBm) (dBm) (dBm) (dBm) ------- ----------------- ---------- --------- --------- --------- Gi4/1/1 -2.7 8.1 5.0 0.0 -4.0 Switch# show interface GigabitEthernet1/1/1 transceiver ITU Channel not available (Wavelength not available), Transceiver is internally calibrated. ++ : high alarm, + : high warning, - : low warning, -- : low alarm. NA or N/A: not applicable, Tx: transmit, Rx: receive. mA: milliamperes, dBm: decibels (milliwatts). Optical Optical Temperature Voltage Tx Power Rx Power Port (Celsius) (Volts) (dBm) (dBm) --------- ----------- ------- -------- -------- Gi1/1/1 38.6 3.29 -2.0 -11.3
  • 95.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Link Issues: Checking Physical Cabling 98 Gi4/1/1 Gi1/1/1 dBm -2.7 -11.3 Loss Budget Analysis (Southbound) Connector not firmly plugged in
  • 96.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 2960-S FlexStack Ease of Use 3750-X StackWise Plus Ease of Use and High Availability Device Limit 4 units 9 units Stack Bandwidth 20G 64G Architecture HW Drop Table Ring (Destination stripping) Dynamic Ring Load Balancing No Yes Stack Convergence 1-2 seconds Few milliseconds Stack QoS Applied hop by hop Applied on ingress Management Single IP address, SNMP, SYSLOG Single IP address, SNMP, SYSLOG Configuration Single config and CLI, auto image and config update Single config and CLI, auto image and config update Show and Debug Commands Unified Unified Single Forwarding and Control Plane Synchronise ARP, MAC Address, IGMP, VLAN tables Synchronise ARP, MAC Address, IGMP, VLAN, Routing tables Cross-Stack Features Yes Yes Single Bridge-ID Yes Yes Preprovison members Yes Yes Redundancy Stack master 1:N redundancy Stack master 1:N redundancy Easy member replacement Yes Yes FlexStack Vs. StackWise Plus 99
  • 97.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Appendix A Error Counters Definition  FCS-Err is the number of valid size frames with FCS (Frame Check Sequence) errors but no framing errors: this is typically a physical issue (cabling, bad port, NIC card,…) but can also indicate a duplex mismatch  Align-Err is the number of frames with alignment errors (frames that do not end with an even number of octets and have a bad CRC) received on the port; these usually indicate a physical problem (cabling, bad port, NIC card,…) but can also indicate a duplex mismatch; when the cable is first connected to the port, some of these errors may occur; also, if there is a hub connected to the port then collisions between other devices on the hub may cause these errors  Late-Coll (Late Collisions) is the number of times that a collision is detected on a particular port late in the transmission process; for a 10mbit/s port this is later than 512 bit-times into the transmission of a packet; five hundred and twelve bit-times corresponds to 51.2 microseconds on a 10 Mbit/s system; this error can indicate a duplex mismatch among other things; for the duplex mismatch scenario the late collision would be seen on the half duplex side; as the half duplex side is transmitting, the full duplex side does not wait its turn and transmits simultaneously causing a late collision; late collisions can also indicate an Ethernet cable/segment that is too long; collisions should not be seen on ports configured as full duplex  Single-Coll (Single Collision) is the number of times one collision occurred before the port transmitted a frame to the media successfully; collisions are normal for port configured as half duplex but should not be seen on full duplex ports; if collisions are increasing dramatically this points to a highly utilised link or possibly a duplex mismatch with the attached device  Multi-Coll (Multiple Collision) is the number of times multiple collisions occurred before the port transmitted a frame to the media successfully; collisions are normal for port configured as half duplex but should not be seen on full duplex ports; if collisions are increasing dramatically this points to a highly utilised link or possibly a duplex mismatch with the attached device 100
  • 98.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Appendix A Error Counters Definition (2)  Excess-Coll (Excessive Collisions) is a count of frames for which transmission on a particular port fails due to excessive collisions; an excessive collision happens when a packet has a collision 16 times in a row; the packet is then dropped; excessive collisions is typically an indication that the load on the segment needs to be split across multiple segments but can also point to a duplex mismatch with the attached device; collisions should not be seen on ports configured as full duplex  Carri-Sen (Carrier Sense) occurs every time an Ethernet controller wants to send data on a half duplex connection; the controller senses the wire and check if it is not busy before transmitting; this is normal on an half-duplex Ethernet segment  Undersize are frames received that are smaller than the minimum IEEE 802.3 frame size of 64bytes long (excluding framing bits, but including FCS octets) that were otherwise well formed; check the device sending out these frames  Runts are frames received that are smaller than the minimum IEEE 802.3 frame size (64 bytes for Ethernet) and with a bad CRC; this can be caused by duplex mismatch and physical problems like a bad cable, port, or NIC card on the attached device  Giants exceed the maximum IEEE 802.3 frame size (1518 bytes for non-jumbo Ethernet); try to find the offending device and remove it from the network  http://www.cisco.com/warp/public/473/164.html#show_interface 101
  • 99.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Port Access-List Switch# sh run | inc access-list access-list 123 permit ip host 10.100.1.2 any Switch# sh run int gi 1/0/2 Building configuration... Current configuration : 134 bytes ! interface GigabitEthernet1/0/2 switchport access vlan 101 ip access-group 123 in mls qos trust dscp spanning-tree portfast end 102
  • 100.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Port Access-List Details Switch# sh platform acl interface gigabitEthernet 1/0/2 portlabels detail Port based ACL: (asic 1) ---------------------------- Input Label: 4 Op Select Index: 255 Interface(s): Gi1/0/2 Access Group: 123, 3 VMRs Mask: 00000000 FFFFFFFF 00000000 00000000 00000000 Value: 00000000 0A640102 00000000 00000000 00000000 Result: 0x09 --- Permit IP Source address Mask: 00000000 00000000 00000000 00000000 00000000 Value: 00000000 00000000 00000000 00000000 00000000 Result: 0x00 --- Deny Mask & Value all 0 = any any Mask: 00000000 00000000 00000000 00000000 00000000 Value: 00000000 00000000 00000000 00000000 00000000 Result: 0x09 IP Source Guard: 0 VMRs LPIP: 0 VMRs MAC Access Group: (none), 0 VMRs access-list 123 permit ip host 10.100.1.2 any 103
  • 101.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Router Access-List Configuration : ! interface Vlan101 ip address 10.101.1.1 255.255.255.0 ip access-group 123 in ! access-list 123 permit ip host 10.100.1.2 any Switch# sh platform acl interface vlan 101 Input Label: 1 Output Label: 0 (default) Input IPv6 Label: 1 Output IPv6 Label: 0 (default) 104
  • 102.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Router Access-List Details Switch# sh platform acl label 1 detail IPv4/MAC ACL label ------------------ Input Op Select Index 255: Output Op Select Index 255: Input Features: Interfaces or VLANs: Vl101 Vlan Map: (none) Access Group: 123, 5 VMRs. Mask: 00000000 FFFFFFFF 00000000 00000000 00000000 Value: 00000000 0A640102 00000000 00000000 00000000 Result: 0x09 Mask: 00000000 00000000 05000000 00000000 00000000 <output removed> 10.100.1.2 105
  • 103.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Vlan Access-List vlan access-map FilterMap 10 action drop match ip address 123 ! vlan filter FilterMap vlan-list 101 access-list 123 permit ip host 10.100.1.2 any Switch# sh vlan filter VLAN Map FilterMap is filtering VLANs: 101 Switch# show platform acl vlan 101 Input Label: 1 Output Label: 1 Input IPv6 Label: 1 Output IPv6 Label: 1 106
  • 104.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 107 Supported ACL TCAM Entry Types Commonly Known As… Configured On… Controls… Direction MAC ACL MACL L2 port Non-IP packets In-bound Port ACL PACL L2 Port IP packets Non-IP packets using MACL In-bound Router ACL RACL L3 port L3 EtherChannel port Switched Virtual Interface (SVI) Routed IP traffic In-bound Out-bound VLAN ACL VACL VLAN IP packets routed into or out of a VLAN IP packets bridged within a VLAN Directionless
  • 105.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Cisco Catalyst 2960-S QoS Model Classification • Inspect incoming packets • Based on ACLs or configuration, determine classification label Policing • Ensure conformance to a specified rate • On an aggregate or individual flow basis • Up to 256 policers per switch • Support for rate and burst Marking • Act on policer decision • Reclass or drop out-of-profile Egress Queue/ Schedule Congestion Control • Four SRR queues/port shared or shaped servicing • One queue is configurable for strict priority servicing • WTD for congestion control (three thresholds per queue) • Egress queue shaping • Egress port rate limiting Ingress Egress Policer Policer Marker Policer Policer Marker Marker Marker SRR Classify Traffic Egress Queues Traffic 108
  • 106.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public Troubleshooting Catalyst 2/3000 QoS Cheat sheet  Aggregate Policer – Marking in policy-map ‒Check Configuration ‒Sh mls qos int gig x/y statistics ‒!!! NOT SUPPORTED : sh policy-map interface  Queueing and scheduling : ‒show platform port-asic stats drop gig x/y ‒show platform port-asic stats enqueue gig x/y  General QoS command : Show running-config Show mls qos Show platform tcam utilization 109
  • 107.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public IGMP Snooping on Catalyst 3750  IGMP snooping entry are not created per mac-address, but they are created for IP multicast group 3750#sh ip igmp snooping groups Vlan Group Type Version Port List ------------------------------------------------------------- 1 239.1.2.3 user Po1 3750#sh mac address-table address 0100.5e01.0203 Mac Address Table ------------------------------------------- Vlan Mac Address Type Ports ---- ----------- -------- -----  A packet with destination mac 0100.5E01.0203 won’t be constrained per IGMP snooping if its ip is not 239.1.2.3! An entry is present with IP address in IGMP Snooping table There is nothing in mac-address-table for the group 110
  • 108.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public References  Troubleshooting Catalyst 3750: http://www.cisco.com/en/US/products/hw/switches/ps5023/prod_troubleshooting_guides_list.html  Online Resources on http://www.cisco.com: ‒Troubleshooting High CPU Utilisation ‒Troubleshooting Power over Ethernet (PoE) ‒Troubleshooting Switch Stacks ‒Cisco Catalyst 3750 QoS Configuration Examples (Doc 91862) ‒Auto Negotiation issues: (Document 17053) 111
  • 109.
    © 2013 Ciscoand/or its affiliates. All rights reserved. BRKRST-3068 Cisco Public 112