NXOS Troubleshooting

Expand Your Toolset with Feature Rich Nexus 9000

Vinay Sawant, Technical Escalation Engineer

Cristian Soto, Technical Consulting Engineer

@iamvinayvsawant
@iamcrissoto2024

TACDCN-2010

#CiscoLive
 • Introduction
• Unveiling the tools
Agenda • Real-world Applications and
Success Stories
• Summary

TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Tools Reviewed Today
Tool #1: Ethanalyzer

Tool #2: SPAN-to-CPU

Tool #3: ELAM

Tool #4: Consistency Checker

Tool #5: Show troubleshoot

Tool #6: iCAM

#CiscoLive
Network Topology
Spine-1

Eth1/47 Eth1/3 Lo1

192.168.100.2
Eth1/47 Eth1/3

Leaf-1 Leaf-2

VLAN 20
VLAN 10 Eth1/1 Eth1/2 10.20.20.1
10.10.10.1
VLAN 10 VLAN 20

Host 1
10.10.10.2 Host 2
MAC 10.20.20.20
10B3.D6A4.BA17

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Tool #1: Ethanalyzer
Ethanalyzer
Packet Capture Tool

• Captures traffic going to and from CPU of Nexus switch.

Syntax

Leaf-1#Ethanalyzer local interface [Interface-type] [display-filer|capture-filter] limit-captured-frame [size] write

bootflash:Filename.pcap detail

Interface-types : inband, mgmt, front-panel, port-channel

Detail : Gives detailed packet view
Decode-internal : Let you see which interface the packet received or sent
Capture-ring-buffer : Let you save the capture in multiple files
Limit-capture-frame : Let you limits capture size. 0 means unlimited packet capture size.

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Ethanalyzer
Packet Capture Tool

Capture-filter : Captures on the packet we asked to capture

Display-filter : Captures all packets but display only we ask to display
Interface-types : inband, mgmt, front-panel, port-channel
Inband Interface – Front panel ports
Mgmt Interface – Mgmt0 port
Front-panel Interface- Only captures control-plane traffic coming on front panel ports.
Limit-capture-frame : Let you define capture frame size. 0 means unlimited packet capture size.

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 Ethanalyzer Leaf-1
Real Word Example [platform N9K-C93180YC-FX / 10.2(6) Reply VLAN 10
10.10.10.1

Eth1/1
Example use case 1:Capture ICMP

Host 1
10.10.10.2
MAC
Request 10B3.D6A4.BA17

Leaf-1# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0

Capturing on 'ps-inb'
4 2024-04-18 15:33:48.226147776 10.10.10.2 → 10.10.10.1 ICMP 102 Echo (ping) request id=0x0a59, seq=0/0, ttl=255
5 2024-04-18 15:33:48.226507747 10.10.10.1 → 10.10.10.2 ICMP 102 Echo (ping) reply id=0x0a59, seq=0/0, ttl=255 (request in 4)
2

Leaf-1#

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Ethanalyzer
Real Word Example [platform N9K-C93180YC-FX / 10.2(6)
Decode Internal

Example use case 2: Capture ICMP with Decode-internal

Leaf-1# ethanalyzer local interface inband decode-internal display-filter "icmp" limit-captured-frames 0

------Output omitted for brevity-----
Dest Port: 17 (0x11)
------Output omitted for brevity----- Packet Leaving
This port
Nexus9K-1# show interface hardware-mappings
-------------------------------------------------------------------------------------------------------------------------------------
Name Ifindex Smod Unit HPort FPort NPort VPort Slice SPort SrcId MacId MacSP VIF Block BlkSrcID
-------------------------------------------------------------------------------------------------------------------------------------
Eth1/1 1a000200 1 0 17 255 4 -1 0 17 34 4 2 5 0 34

Maps to E1/1

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 Ethanalyzer Leaf-1
Real Word Example [platform N9K-C93180YC-FX / 10.2(6) Reply VLAN 10
10.10.10.1

Example use case 3: Capture ARP Eth1/1

Host 1
VLAN 10
Request 10.10.10.2

ARP Request
Leaf-1# ethanalyzer local interface inband display-filter "arp" limit-captured-frames 0 Received
Capturing on 'ps-inb'
23 2024-04-18 16:08:07.492460719 10:B3:D6:A4:BA:17→ ff:ff:ff:ff:ff:ff ARP 64 Who has 10.10.10.1? Tell 10.10.10.2
24 2024-04-18 16:08:07.492902971 e4:1f:7b:2f:a5:c7 → 10:B3:D6:A4:BA:17 ARP 64 10.10.10.1 is at e4:1f:7b:2f:a5:c7
2

Leaf-1# ARP Reply

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Ethanalyzer
Real Word Example [platform N9K-C93180YC-FX / 10.2(6)
ARP filter based
on sender IP
Example use case 4: Capture ARP with more filters

Leaf-1# ethanalyzer local interface inband display-filter "arp.src.proto_ipv4==10.10.10.2" limit-captured-frames 0

Capturing on 'ps-inb'
10 2024-04-18 16:16:36.974236159 10:B3:D6:A4:BA:17→ ff:ff:ff:ff:ff:ff ARP 64 Who has 10.10.10.1? Tell 10.10.10.2
1
Leaf-1 #
ARP filter based
on Target IP

Leaf-1#ethanalyzer local interface inband display-filter "arp.dst.proto_ipv4==10.10.10.1" limit-captured-frames 0

Capturing on 'ps-inb'
7 2024-04-18 16:17:50.166112161 10:B3:D6:A4:BA:17→ ff:ff:ff:ff:ff:ff ARP 64 Who has 10.10.10.1? Tell 10.10.10.2
1
Leaf-1#

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 Spine-1

Ethanalyzer
Real Word Example [platform N9K-C93180YC-FX / 10.2(6)
Eth1/47 Eth1/3 Lo1
Example use case 5: Check if packets are getting 192.168.100.2
Eth1/47 Eth1/3
software Switched
Leaf-1 Leaf-2

VLAN 20
VLAN 10 Eth1/1 Eth1/2 10.20.20.1
10.10.10.1
VLAN 10 VLAN 20

Host 1
10.10.10.2 Host 2
MAC 10.20.20.20
10B3.D6A4.BA17

Leaf-1# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0

Capturing on 'ps-inb'
4 2024-04-18 15:33:48.226147776 10.10.10.2 → 10.20.20.20 ICMP 102 Echo (ping) request id=0x0a59, seq=0/0, ttl=255

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Ethanalyzer
Real Word Example [platform N9K-C93180YC-FX / 10.2(6)

Example use case 6: Troubleshooting BGP neighbor flap using ethanalyzer

Leaf-1# ethanalyzer local interface inband display-filter "ip.addr==10.10.10.1 && ip.addr==10.10.10.2" limit-
captured-frames 0
Capturing on 'ps-inb’ Syn Syn Ack

34 2024-04-18 17:22:46 10.10.10.2 → 10.10.10.1 TCP 78 51278 → 179 [SYN] Seq=0 Win=29200 Len=0 MSS=1460
35 2024-04-18 17:22:46 10.10.10.1 → 10.10.10.2 TCP 78 179 → 51278 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0
MSS=1460 36 2024-04-18 17:22:46 10.10.10.2 → 10.10.10.1 TCP 70 51278 → 179 [ACK] Seq=1 Ack=1 Win=29200
Len=0 TSval=22641527
37 2024-04-18 17:22:46 10.10.10.1 → 10.10.10.2 BGP 146 OPEN Message Ack
44 2024-04-18 17:22:47 10.10.10.1 → 10.10.10.2 BGP 118 UPDATE Message, KEEPALIVE Message
45 2024-04-18 17:22:47 10.10.10.2 → 10.10.10.1 BGP 166 UPDATE Message, KEEPALIVE Message, UPDATE Message
46 2024-04-18 17:22:472 10.10.10.1 → 10.10.10.2 TCP 70 179 → 51278 [ACK] Seq=144 Ack=192 Win=65536 Len=0
Update

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Ethanalyzer
Real Word Example [platform N9K-C93180YC-FX / 10.2(6)

Example use case 6: Troubleshooting BGP neighbor flap using ethanalyzer

Leaf-1# ethanalyzer local interface inbound-hi display-filter "bgp && ip.addr==10.20.20.1" limit-captured-frames 0
Update packet
Capturing on inband dropped
2024-05-04 21:51:28.977444 10.10.10.1 -> 10.20.20.1 BGP OPEN Message
2024-05-04 21:51:29.979955 10.10.10.1 -> 10.20.20.1 BGP KEEPALIVE Message
2024-05-04 21:51:30.996699 10.10.10.1 -> 10.20.20.1 BGP [TCP Retransmission] KEEPALIVE Message
2024-05-04 21:51:31.106224 10.10.10.1 -> 10.20.20.1 BGP UPDATE Message, UPDATE Message, UPDATE Message,
UPDATE Message, UPDATE Message

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Ethanalyzer
Real Word Example [platform N9K-C93180YC-FX / 10.2(6) Example use case 7: Capture COPP dropped traffic

Leaf-1# show policy-map interface control-plane class ciscolive-copp-class-l3uc-data

Control Plane

Service-policy input: ciscolive-copp-policy-strict Class name

class-map ciscolive-copp-class-l3uc-data (match-any)

match exception glean
set cos 1
Glean
police cir 10 bps , bc 10 bytes
module 1 :
transmitted 246600 bytes;
5-minute offered rate 664 bytes/sec
conformed 2400 peak-rate bytes/sec
at Fri Apr 19 15:30:48 2024 Drops

dropped 135580910 bytes;

5-min violate rate 364001 byte/sec
violated 1386246 peak-rate byte/sec at Fri Apr 19 15:30:01 2024
Nexus9K-1#

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Ethanalyzer
Real Word Example [platform N9K-C93180YC-FX / 10.2(6)

Example use case 7: Capture COPP dropped traffic

Leaf-1# show system internal access-list copp stats stage1 | in ciscolive-copp-class-l3uc-data

=======================================================================
Queue Name Transmitted(bytes) Dropped(bytes)
------------------------------------------------------------------------------------------------------------------

36 ciscolive-copp-class-l3uc-data 8808070 293202890

Leaf-1# Queue
Number

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Ethanalyzer
Real Word Example [platform N9K-C93180YC-FX / 10.2(6) Filter Base on
Queue Number
Example use case 7: Capture COPP dropped traffic

Leaf-1# ethanalyzer local interface inband decode-internal display-filter "cisco.blob.sup_qnum==36" limit-

captured-frames 0
------Output omitted for brevity-----
Internet Protocol Version 4, Src: 10.10.10.2, Dst: 10.20.10.3
0100 .... = Version: 4
Time to live: 255
Protocol: ICMP (1) Detailed about
Header checksum: 0x0503 [validation disabled] the packet
[Header checksum status: Unverified]
Source: 10.10.10.2
Destination: 10.20.10.3
------Output omitted for brevity-----

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 Ethanalyzer
Real Word Example [platform N9K-C93180YC-FX / 10.2(6)
Save to
Saving Capture and Ring Buffer bootflash

Leaf-1# ethanalyzer local interface inband limit-captured-frames 100 write bootflash:cpucap.pcap

Capturing on inband
100
Leaf-1# ethanalyzer local interface inband limit-captured-frames 0 capture-ring-buffer filesize 100 write bootflash:ringcap.pcap
Capturing on inband
5616
Leaf-1# dir | in pcap
11710 Apr 19 14:26:08 2024 cpucap.pcap
102452 Apr 19 14:26:34 2024 ringcap_00001_20240419142633.pcap Capture ring buffer
102480 Apr 19 14:26:36 2024 ringcap_00002_20240419142634.pcap
102448 Apr 19 14:26:36 2024 ringcap_00003_20240419142636.pcap
102452 Apr 19 14:26:39 2024 ringcap_00004_20240419142636.pcap
102452 Apr 19 14:26:41 2024 ringcap_00005_20240419142639.pcap
102448 Apr 19 14:26:41 2024 ringcap_00006_20240419142641.pcap
47700 Apr 19 14:26:42 2024 ringcap_00007_20240419142641.pcap
Leaf-1#

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Ethanalyzer
Real Word Example [platform N9K-C93180YC-FX / 10.2(6)
Local Read
Reading the capture locally

Leaf-1# ethanalyzer local read cpucap.pcap limit-captured-frames 0

2024-04-19 14:26:08.419367 10.10.10.2 -> 10.10.10.1 ICMP Echo (ping) request
2024-04-19 14:26:08.419749 10.10.10.1 -> 10.10.10.2 ICMP Echo (ping) reply
2024-04-19 14:26:08.420323 10.10.10.2 -> 10.10.10.1 ICMP Echo (ping) request
2024-04-19 14:26:08.420473 10.10.10.1 -> 10.10.10.2 ICMP Echo (ping) reply
2024-04-19 14:26:08.420945 10.10.10.2 -> 10.10.10.1 ICMP Echo (ping) request
2024-04-19 14:26:08.421073 10.10.10.1 -> 10.10.10.2 ICMP Echo (ping) reply
2024-04-19 14:26:08.421520 10.10.10.2 -> 10.10.10.1 ICMP Echo (ping) request

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Tool #2: SPAN-to-CPU
SPAN-to-CPU
Real Word Example [platform N9K-C93180YC-FX / 10.2(6)
How does it work? Spine-1

• Capture passthrough traffic

• SPAN Replication is done in hardware with no
Eth1/47 Eth1/3 Lo1
192.168.100.2
impact to CPU
Eth1/47 Eth1/3
• SPAN-to-CPU packets are rate-limited. Default
Leaf-1 Leaf-2
rate-list is 50 pps. “show hardware rate-limiter
span”
VLAN 20
10.20.20.1
• You can change this setting using command
VLAN 10 Eth1/1 Eth1/2
10.10.10.1
VLAN 20
“hardware rate-limiter span”
VLAN 10

• Filter using access-list, VLAN

Host 1
10.10.10.2 Host 2
MAC 10.20.20.20
10B3.D6A4.BA17

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
SPAN-to-CPU Spine-1 Capture
Real Word Example [platform N9K-C93180YC-FX / 10.2(6) Destination
Nexus CPU
Eth1/47
Configuration
Eth1/47

Leaf-1

Leaf-1# show run monitor VLAN 10

10.10.10.1
monitor session 1
source interface Ethernet1/1 both Eth1/1
destination interface sup-eth0
filter vlan 10 (We can either filter on VLAN or ACL) Capture
filter access-group test Source
no shut
10.10.10.2
Leaf-1# Leaf-1
MAC 10B3.D6A4.BA17

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
 Span to CPU
Real Word Example [platform N9K-C93180YC-FX / 10.2(6)

Example

Mirror

Leaf-1# ethanalyzer local interface inband mirror display-filter "icmp" limit-captured-frames 0

Capturing on 'ps-inb'
10 2024-04-19 19:48:32.118031065 10.10.10.2 → 10.20.20.20 ICMP 102 Echo (ping) request id=0x544c, seq=0/0
2024-04-19 19:48:32.118438013 10.20.20.20 → 10.10.10.2 ICMP 102 Echo (ping) reply id=0x544c, seq=0/0

Leaf-1#

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
SPAN-to-CPU
Changing default Span Rate
Default
50 pps
Leaf-1#show hardware rate-limiter span
Module: 1
R-L Class Config Allowed Dropped Total
+----------------+----------+--------------------+--------------------+--------------------+
span 50 0 0 0

Leaf-1#(config)# hardware rate-limiter span 100 Changing rate

Leaf-1# show hardware rate-limiter span

Module: 1
R-L Class Config Allowed Dropped Total
+----------------+----------+--------------------+--------------------+--------------------+
span 100 0 0 0
New Value 100
pps

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Tool #3: ELAM
ELAM
Packet Capture and Forwarding Verification Tool

• “Embedded Logic Analyzer Module”

• Captures a single packet on ingress going to switch’s control plane or through
the switch in the data plane
• Capture occurs just above the ASIC level of the switch (very close to the wire)
• Supported on all Cloud scale Nexus 9k switches
• Not supported on first generation or –R Series Nexus 9ks

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
ELAM
Nexus 9k Generations/Model Types

• First Generation: 9300-TX/PX/PQ, 9400-TX/PX/PQ, 9500-TX/PX/PQ

• Cloud scale: 9200, 9300-EX/FX/FXP/FX2/FX3/GX/GX2/H, 9400, 9700-
EX/FX/GX
• R Series (Fretta): 9600-R/RX/R2

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
ELAM
Syntax, Top of Rack

switch# debug platform internal tah elam

switch(TAH-elam)# trig init [slice <A>] [in-select <B>] [use-src-id <C>]
switch(TAH-elam-inselB)# set [inner|outer] [packet type] [options fields]
switch(TAH-elam-inselB)# start
switch(TAH-elam-inselB)# report

• A: Slice # of the ASIC

• B: #’s that indicate potential OSI layers and headers/encapsulation
• C: Source interface ID of the packet

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
ELAM
In-Select Options and Trigger options

switch(TAH-elam)# trig init in-select ?

• 10 {outer l4, inner l4, ieth} Less Common, generally used for checking L4 info
• 19 {udf_vec}
• 6 {outer l2, outer l3, outer l4} Most Common, used for normal IPv4/IPv6, ARP and more
• 7 {inner l2, inner l3, inner l4} Less Common, can be used for L2 flows or VXLAN
• 8 {outer l2, inner l2, ieth}
• 9 {outer l3, inner l3} More Commonly used with encapsulated traffic (VXLAN, MPLS)

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
ELAM
In-Select Options and Trigger options

switch(TAH-elam-insel6)# set [inner|outer] [packet type] [options fields]

• Inner/Outer: Inner or Outer frame/packet header

• Packet Type: L2, ARP, IPv4, IPv6, L4, MPLS, FCoE
• Options Fields: SRC/DST MACs/IPs/Ports, Vlans, DSCP, Protocol Number,
etc.

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
 ELAM
Scenario #1a: ICMP Failure – Does it reach me?
Spine-1
Leaf-1# debug
Leaf-1# debug platform
platform internal
internal tah
tah elam
elam

Lo1 Lo1
Leaf-1(TAH-elam)# trig
Leaf-1(TAH-elam)# trig init
init
Eth1/47 Eth1/3
192.168.100.1 192.168.100.2 Slot 1:
Slot 1: param
param values:
values: start
start asic
asic 0,
0, start
start slice
slice 0,
0, lu-a2d
lu-a2d 1,
1, in-
in-
Eth1/47 Eth1/3
select 6,
select 6, out-select
out-select 00
Leaf-1 Leaf-2

Leaf-1(TAH-elam-insel6)# set
Leaf-1(TAH-elam-insel6)# set outer
outer ipv4
ipv4 src_ip
src_ip 10.10.10.2
10.10.10.2
VLAN 10 VLAN 20 dst_ip 10.10.10.1
dst_ip 10.10.10.1
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1

VLAN 20
VLAN 10
Leaf-1(TAH-elam-insel6)# start
Leaf-1(TAH-elam-insel6)# start

Leaf-1(TAH-elam-insel6)# report
Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
ELAM
Asic Trigger Explanations and vPC Notes
Leaf-1(TAH-elam)# trig init
Slot 1: param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 0

• The asic value is only necessary for modular (EoR) Nexus 9ks due to the different linecards which may use
different ASIC values per port group
• The slice number corresponds to the partition of the ASIC to which the interfaces are associated
• In modern ELAM (post 7.0(3)I5(2) ), defining the lu-a2d value and out-select is not necessary, but it is
associated with the different aspects of the forwarding table
• When using vPC, it is important to set up the ELAM capture on both sides, since traffic can technically land
on either peer

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 ELAM
Scenario #1a: ICMP Failure – Does it reach me?
Spine-1 <prev>
Leaf-1(TAH-elam-insel6)# report
Dst MAC address:
SUGARBOWL ELAM REPORT 10:B3:D6:A4:75:A7
SUMMARY Src MAC address:
Lo1 Eth1/47 Eth1/3 Lo1
192.168.100.1 192.168.100.2
slot - 1, asic - 0, slice - 0 10:B3:D6:A4:BA:17
Eth1/47 Eth1/3 ============================ .1q Tag0 VLAN: 10, cos = 0x0
Leaf-1 Leaf-2 Incoming Interface: Eth1/1 Sup hit: 1, Sup Idx: 2788
Src Idx : 0x1, Src BD : 10
Outgoing Interface Info: dmod 0, Dst IPv4 address: 10.10.10.1
VLAN 10 VLAN 20 dpid 0
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1 Src IPv4 address: 10.10.10.2
Dst Idx : 0x5bf, Dst BD : 10 Ver = 4, DSCP = 0, Don't
VLAN 10 VLAN 20
Packet Type: IPv4 Fragment = 0
<cont>

Leaf-1# show system internal access-list sup-redirect-stats | i 2788

Host 1 Host 2 2788 copp-system-p-acl-icmp 4
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
ELAM
ELAM Report Component Notes

Leaf-1(TAH-elam-insel6)# report
ELAM not triggered yet on slot - 1, asic - 0, slice – 0
ELAM hit flop error on slot - 1, asic - 0, slice - 1. Try elam again.

• If you expect to receive the traffic on a specific interface or general and see “ELAM not triggered” there
are one of three possible scenarios:
• You may need to run the ELAM again (it is possible that you started the capture too late)
• You are not receiving the traffic on the interface that you think you are (particularly relevant if you do
not know on what interface you should receive the traffic)
• The Nexus 9k is not receiving the traffic
- If you see the error “ELAM hit flop error”, this is not a cause for concern. This simply means that you should
enter “reset” and then set the trigger with the “set” command again

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
ELAM
ELAM Report Component Notes
HOMEWOOD ELAM REPORT SUMMARY
slot - 1, asic - 3, slice - 0
============================

Incoming Interface: Eth1/36

Src Idx : 0x8d, Src BD : 4395
Outgoing Interface Info: dmod 2, dpid 48
Dst Idx : 0x79, Dst BD : 4390
<snip>
.1q Tag0 VLAN: 10, cos = 0x0
<snip>
Sup hit: 1, Sup Idx: 2788

• If your ingress interface or egress interface is an L3 port or loopback, the SRC/DST BD field for that interface will show as
a number outside of the normal range of vlans allowed by the Nexus, which is normally in the 4096+ range
• When using dot1q tunnels, a transit switch may show a different SRC/DST BD tag than the .1q Tag field ; this is expected
behavior
• If this packet is not destined for the Nexus 9k and you see a supervisor hit, there are one of two possible scenarios:
• You have an SVI on the switch active
• Your switch is incorrectly punting the packet into the control plane, which can cause latency and/or drops

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
ELAM
ELAM Report Component Notes
Drop Info:
----------

LUA:
LUB:
LUC:
LUD:
Final Drops:

vntag:
vntag_valid : 0
vntag_vir : 0
vntag_svif : 0

• Just because you see something under the “Drop Info” in the second half of the output does not mean
that it will drop. If the ELAM correctly registers the packet drop, you should see a reason under BOTH
LUA/B/C/D AND the Final Drops sections
- Although ELAM is normally very reliable, sometimes ELAM may show that it is forwarding even though it is
not. To verify if this is happening, do the following:
- Confirm, if possible, that the next hop receives the expected packet
- Confirm, through consistency checker and troubleshooting commands, that we have correctly
programmed routes/L2 adjacencies/protocols
#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 ELAM
Scenario #1b: ICMP Failure – Do I forward it correctly?
Spine-1

• Determine your SRC Interface and its slice number and source
Lo1 Eth1/47 Eth1/3 Lo1
192.168.100.1 192.168.100.2
ID:
Eth1/47 Eth1/3
Leaf-1# show system internal ethpm info interface eth1/1 | i dpid
Leaf-1 Leaf-2
IF_STATIC_INFO:
port_name=Ethernet1/1,if_index:0x1a000000,ltl=6144,slot=0,
VLAN 10
SRC IF VLAN 20 nxos_port=0,dmod=1,dpid=16,unit=0,queue=65535,xbar_unitbmp=
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1 0x0,ns_pid=255,slice_num=0,port_on_slice=16,src_id=32
VLAN 10 VLAN 20
• Confirm that the packet is non-encapsulated

Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
 ELAM
Scenario #1b: ICMP Failure – Do I forward it correctly?
Spine-1
Leaf-1# debug platform internal tah elam

Leaf-1(TAH-elam)# trig init slice 0 in-select 6 use-src-id 32

Lo1 Eth1/47 Eth1/3 Lo1
192.168.100.1 192.168.100.2 Slot 1: param values: start asic 0, slice 0, lu-a2d 1, in-
Eth1/47 Eth1/3
select 6, out-select 0, src_id 32
Leaf-1 Leaf-2

?
Leaf-1(TAH-elam-insel6)# set outer ipv4 src_ip 10.10.10.2
VLAN 10
SRC IF VLAN 20 dst_ip 10.20.20.20
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1

VLAN 10 VLAN 20 Leaf-1(TAH-elam-insel6)# start

Leaf-1(TAH-elam-insel6)# report
Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
 ELAM
Scenario #1b: ICMP Failure – Do I forward it correctly?
Spine-1
Leaf-1# report
SUGARBOWL ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 0
Lo1
============================
Eth1/47 Eth1/3 Lo1
192.168.100.1 192.168.100.2
Eth1/47 Eth1/3 Incoming Interface: Eth1/1
Leaf-1 Leaf-2 Src Idx : 0x1, Src BD : 10
Outgoing Interface Info: dmod 1, dpid 38
Dst Idx : 0xb9, Dst BD : 200
VLAN 10
SRC IF VLAN 20
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1
<snip>
VLAN 10 VLAN 20

Dst IPv4 address: 10.20.20.20

Src IPv4 address: 10.10.10.2

Host 1 Host 2 <snip>

10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 ELAM
Scenario #1b: ICMP Failure – Do I forward it correctly?
Spine-1

Leaf-1# show system internal ethpm info global | i dpid=38

Lo1 Eth1/47 Eth1/3 Lo1

IF_STATIC_INFO:
192.168.100.1 192.168.100.2 port_name=Ethernet1/47,if_index:0x1a005c00,ltl=5960,slot=0,
Eth1/47 Eth1/3 nxos_port=184,dmod=1,dpid=38,unit=0,queue=65535,xbar_unitbmp=
0x0,ns_pid=255,sli
Leaf-1 Leaf-2
ce_num=0,port_on_slice=38,src_id=68

SRC IF Leaf-1# show cdp neighbor interface e1/47

VLAN 10 VLAN 20
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1
Device-ID Local Intrfce Hldtme Capability Platform Port ID
VLAN 10 VLAN 20
Spine-1(FDO233201F9)
Eth1/47 124 R S HDMI N9K-C93180YC-EX Eth1/47

Total entries displayed: 1

Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
ELAM
ELAM Report: Optional way to run it

Leaf-1(TAH-elam-insel6)# show elam report

Dumping report for module 1 inst 0 slice 0 a_to_d 1 insel 6 outsel 0
Elam Report captured on Interface : Eth1/1
Outgoing Interface : Eth1/47
Final Drop reason: NA
Printing output related to l2_header
<snip>
Printing output related to l3_header

• Another way to run it which prevents you from having to decode the outgoing dmod/dpid values

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
 ELAM
Scenario #1c: ICMP Failure – Do I forward it correctly (End of Rack/9500s)
Spine-1
Spine-1(config)# debug platform internal tah elam
SRC IF Spine-1(TAH-elam)# trig init
Slot 1: param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 0
Slot 22: param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 0
Lo1 Eth1/31 Eth1/36 Lo1
192.168.100.1
Slot 23: param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 0
192.168.100.2
Eth1/47 Eth1/48 Slot 24: param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 0
Slot 26: param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 0
Leaf-1 Leaf-2 switch(TAH-elam-insel6)#

VLAN 10 VLAN 20 • If you have a Nexus 9500, you likely have multiple linecards and
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1 fabric modules
VLAN 10 VLAN 20 • If you try to run an ELAM wide open on a Nexus 9500, it will
attempt to trigger on all of the modules, and only on one slice of a
specific ASIC of module 1
• To trigger for a packet correctly on a Nexus 9500, you must
Host 1 Host 2 capture on the correct linecard/module, the right ASIC and the
10.10.10.2 10.20.20.20 right slice where the traffic may appear

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 ELAM
Scenario #1c: ICMP Failure – Do I forward it correctly (End of Rack/9500s)
Spine-1
Spine-1# show system internal ethpm info int e1/36 | i dpid
IF_STATIC_INFO: port_name=Ethernet1/36,if_index:0x1a004600,ltl=6004,slot=0,
SRC IF nxos_port=140,dmod=4,dpid=44,unit=3,queue=65535,xbar_unitbmp=0x0,ns_pid=255,
slice_num=0,port_on_slice=44,src_id=88
Lo1 Eth1/31 Eth1/36 Lo1
192.168.100.1 192.168.100.2
Eth1/47 Eth1/48
• Keep in mind that if you have multiple interfaces (like ECMP, port-
Leaf-1 Leaf-2 channels with multiple links, etc.), you will need the highlighted
info for all of the interfaces
• If the interfaces are on different modules, you will need to attach
VLAN 10 VLAN 20
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1 to each module and check each one separately
VLAN 10 VLAN 20
Spine-1# attach mod 1
Attaching to module 1 ...
To exit type 'exit', to abort type '$.'
Last login: Sat May 11 16:44:59 UTC 2024 from sup27 on pts/1

Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
 ELAM
Scenario #1c: ICMP Failure – Do I forward it correctly (End of Rack/9500s)
Spine-1

module-1# debug platform internal tah elam asic 3

SRC IF module-1(TAH-elam)# trig init slice 0 in-select 6 use-src-id 88
Slot 1: param values: start asic 0, slice 0, lu-a2d 1, in-select 6, out-select
Lo1 Eth1/31 Eth1/36 Lo1 0, src_id 88
192.168.100.1 192.168.100.2 module-1(TAH-elam-insel6)# set outer ipv4 src 172.16.2.2 dst 9.9.9.9
Eth1/47 Eth1/48 module-1(TAH-elam-insel6)# start
module-1(TAH-elam-insel6)# report
Leaf-1 Leaf-2 ELAM not triggered yet on slot - 1, asic - 0, slice - 0
ELAM not triggered yet on slot - 1, asic - 1, slice - 0
ELAM not triggered yet on slot - 1, asic - 2, slice - 0
VLAN 10 VLAN 20 HOMEWOOD ELAM REPORT SUMMARY
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1 slot - 1, asic - 3, slice - 0
============================
VLAN 10 VLAN 20
Incoming Interface: Eth1/36
Src Idx : 0x8d, Src BD : 4395
Outgoing Interface Info: dmod 2, dpid 48
Dst Idx : 0x79, Dst BD : 4390
Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 ELAM
Scenario #1c: ICMP Failure – Do I forward it correctly (End of Rack/9500s)
Spine-1

SRC IF module-1(TAH-elam-insel6)# end

module-1# exit
Lo1 Eth1/31 Eth1/36 Lo1 Spine-1# show system internal ethpm info glob | i dpid=48
192.168.100.1 192.168.100.2 IF_STATIC_INFO: port_name=Ethernet1/29,if_index:0x1a003800,ltl=6032,slot=0,
Eth1/47 Eth1/48 nxos_port=112,dmod=1,dpid=48,unit=0,queue=65535,xbar_unitbmp=0x0,ns_pid=
255,slice_num=0,port_on_slice=48,src_id=96
Leaf-1 Leaf-2 IF_STATIC_INFO: port_name=Ethernet1/31,if_index:0x1a003c00,ltl=6024,slot=0,
nxos_port=120,dmod=2,dpid=48,unit=1,queue=65535,xbar_unitbmp=0x0,ns_pid=
255,slice_num=0,port_on_slice=48,src_id=96
VLAN 10 VLAN 20 IF_STATIC_INFO: port_name=Ethernet1/33,if_index:0x1a004000,ltl=6016,slot=0,
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1 nxos_port=128,dmod=3,dpid=48,unit=2,queue=65535,xbar_unitbmp=0x0,ns_pid=
255,slice_num=0,port_on_slice=48,src_id=96
VLAN 10 VLAN 20 IF_STATIC_INFO: port_name=Ethernet1/35,if_index:0x1a004400,ltl=6008,slot=0,
nxos_port=136,dmod=4,dpid=48,unit=3,queue=65535,xbar_unitbmp=0x0,ns_pid=
255,slice_num=0,port_on_slice=48,src_id=96
switch#

Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 ELAM
Scenario #2: Am I receiving the ARP Request and Flooding it?
Spine-1
Leaf-1# debug platform internal tah elam

Lo1 Eth1/47 Eth1/3 Lo1

Leaf-1(TAH-elam)# trig init slice 0 in-select 6 use-src-id 32
192.168.100.1 192.168.100.2 Slot 1: param values: start asic 0, slice 0, lu-a2d 1, in-
Eth1/47 Eth1/3
select 6, out-select 0, src_id 32
Leaf-1 Leaf-2

Leaf-1(TAH-elam-insel6)# set outer arp source-ip-addr

VLAN 10
SRC IF VLAN 20 10.10.10.2 target-ip-addr 10.10.10.3
10.10.10.1 Eth1/1 Eth1/3 10.20.20.1

VLAN 10 VLAN 10
Leaf-1(TAH-elam-insel6)# start

Leaf-1(TAH-elam-insel6)# report
Host 1 Host 3
10.10.10.2 10.10.10.3

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 ELAM
Scenario #2: Am I receiving the ARP Request and Flooding it?
Spine-1
slot - 1, asic - 0, slice - 0
============================
<prev>
Incoming Interface: Eth1/1
Lo1 Eth1/47 Eth1/3 Src Idx : 0x1, Src BD : 10 Target Hardware address:
192.168.100.1
Eth1/47 Eth1/3 Outgoing Interface Info: dst_ptr 10, FF:FF:FF:FF:FF:FF
dst_ptr_is_flood 1 Sender Hardware address:
Leaf-1 Leaf-2 10:B3:D6:A4:BA:17
Packet Type: ARP Target Protocol address: 10.10.10.3
Sender Protocol address: 10.10.10.1
VLAN 10
SRC IF VLAN 20 Dst MAC address: FF:FF:FF:FF:FF:FF ARP opcode: 1
10.10.10.1 10.20.20.1
Eth1/1 Eth1/3 Src MAC address:
VLAN 10 VLAN 10 10:B3:D6:A4:BA:17 Sup hit: 1, Sup Idx: 2648
.1q Tag0 VLAN: 10, cos = 0x6
<cont>

Leaf-1# show system internal access-list sup-redirect-stats | i 2648

Host 1 Host 3 2648 ARP REQUEST 3
10.10.10.2 10.10.10.3
2648 ARP REQUEST 0

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 ELAM
Decode Flood Interfaces Option #1: Consistency Checker
• Non-detailed output:
Leaf-1# show consistency-checker membership vlan 10
Checks: Port membership of Vlan in vifvlanmbr, rwepgstate and qsmt_ovtbl tables
Additional Checks: Fex port membership of Vlan in vifvlanmbrsearchtable table
Ports configured as "switchport monitor" will be skipped

Ports configured as "switchport mode private-vlan" will be skipped

For additional PVLAN membership vlan checks use:
"show consistency-checker membership vlan [vlan-id] private-vlan"

Checking hardware for Module 1 Unit 0

No FEX interfaces to validate
Consistency Check: PASSED
Vlan:10, Hardware state consistent for:
Ethernet1/1
Ethernet1/47
Ethernet1/48

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 ELAM
Decode Flood Interfaces Option #1: Consistency Checker
• Detailed output:
Leaf-1# show consistency-checker membership vlan 10 detail
"expectedDetails": {},
"actualDetails": {
"hwTableName": "tah_sug_qsmt_ovtbl",
"hwIndexName": "data",
"tableData": [
{
"number": 1,
"units": [
{
"number": 0,
"slices": [
{
"number": 0,
Translated Bitmap on
"bitmap": "0x00000000:0x000000c0:0x00010000“
<snip>
Slide 52

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
ELAM
Decode Flood Interfaces Option #2: ELAM and ASIC Hardware Tables

Leaf-1# attach module 1

module-1# debug platform internal tah elam
module-1(TAH-elam)# trig init slice 0 in-select 6 use-src-id 32
Slot 1: param values: start asic 0, slice 0, lu-a2d 1, in-select 6, out-select 0,
src_id 32
module-1(TAH-elam-insel6)# set outer arp source-ip 10.10.10.1 target-ip
10.10.10.254
module-1(TAH-elam-insel6)# start
module-1(TAH-elam-insel6)# report module-1# hex 100xa
SUGARBOWL ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 0
============================

Incoming Interface: Eth1/1

Src Idx : 0x1, Src BD : 10
Outgoing Interface Info: dst_ptr 10, dst_ptr_is_flood 1
<snip>

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
ELAM
Decode Flood Interfaces Option #2: ELAM and ASIC Hardware Tables

module-1(TAH-elam-insel6)# debug hardware internal sug dump asic 0 slice

0 table tah_sug_qsmt_ovtbl 0xa changed field-per-line
asic instance is 0
asic slice is 0
tbl name is tah_sug_qsmt_ovtbl
start entry is 10
change entries only
one field per line
Block base address: 0x01400000
1st table entry address: 0x016800a0
ENTRY[10] = {
data=0x00000000:0x000000c0:0x00010000
}

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
ELAM
Decode Flood Interfaces Option #2: ELAM and ASIC Hardware Tables
• Starting from right and moving to the left in the above hex string, each value holds 4 bits (starting from 0 on
the far right). Therefore, if we count the bits:

000000c0:00010000 = 0000 0000 0000 0000 0000 0000 1100 0000 : 0000 0000 0000 0001 0000 0000 0000 0000

39 38 16 3 0
• This bitmap corresponds to the Sport column values under “show interface hardware-mappings”

show interface hardware-mappings | i 16|38|39|Sport

Name Ifindex Smod Unit HPort FPort NPort VPort Slice SPort SrcId MacId MacSP VIF Block BlkSrcID
Eth1/1 1a000000 1 0 16 255 0 -1 0 16 32 4 0 1 0 32
<snip>
Eth1/47 1a005c00 1 0 38 255 184 -1 0 38 68 8 4 1537 0 68
Eth1/48 1a005e00 1 0 39 255 188 -1 0 39 70 8 6 1537 0 70
<snip>

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
 ELAM
ELAM for ARP Reply
• For the ARP Reply, you can just get the results from an ARP ELAM for the reverse flow:

Leaf-1(TAH-elam-insel6)# set outer arp source-ip-addr 10.10.10.254 target-ip-addr 10.10.10.1

EX3(TAH-elam-insel6)# start
EX3(TAH-elam-insel6)# report
SUGARBOWL ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 0
============================

Incoming Interface: Eth1/47

Src Idx : 0x601, Src BD : 10
Outgoing Interface Info: dmod 1, dpid 16
Dst Idx : 0x1, Dst BD : 10

Packet Type: ARP

<snip>

Target Hardware address: 10:B3:D6:A4:BA:17

Sender Hardware address: 00:00:0C:9F:F0:0A
Target Protocol address: 10.10.10.1
Sender Protocol address: 10.10.10.254
ARP opcode: 2

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
 ELAM
Scenario #3: VXLAN Flow – Is the fabric dropping the packet?
Spine-1
• Determine your SRC Interface and its slice number and source ID on each
switch:

Leaf-1# show system internal ethpm info interface eth1/1 | i dpid

Lo1 Eth1/47 Eth1/3 Lo1
192.168.100.1 192.168.100.2
IF_STATIC_INFO:
Eth1/47 Eth1/3 port_name=Ethernet1/1,if_index:0x1a000000,ltl=6144,slot=0,
SRC IF nxos_port=0,dmod=1,dpid=16,unit=0,queue=65535,xbar_unitbmp=0x0,ns_pid
Leaf-1 Leaf-2 =255,slice_num=0,port_on_slice=16,src_id=32

Leaf-2# show system internal ethpm info interface eth1/3 | i dpid

SRC IF VLAN 20
VLAN 10 IF_STATIC_INFO:
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1
port_name=Ethernet1/3,if_index:0x1a000400,ltl=6136,slot=0,
VLAN 10 VLAN 20 nxos_port=8,dmod=1,dpid=18,unit=0,queue=65535,xbar_unitbmp=0x0,ns_pid
=255,slice_num=0,port_on_slice=18,src_id=36

• Confirm that the packet is non-encapsulated on ingress for Leaf-1, but it

would be encapsulated on ingress for Leaf-2
Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
 ELAM
Scenario #3: VXLAN Flow – Is the fabric dropping the packet?
Spine-1

Leaf-1# debug platform internal Leaf-2# debug platform internal

tah elam tah elam
Lo1 Eth1/47 Eth1/3 Lo1
192.168.100.1 192.168.100.2 Leaf-1(TAH-elam)# trig init slice 0 Leaf-2(TAH-elam)# trig init slice 0
Eth1/47 Eth1/3 in-select 6 use-src-id 32 in-select 9 use-src-id 36
Leaf-1 Leaf-2
Slot 1: param values: start asic 0, Slot 1: param values: start asic 0,
slice 0, lu-a2d 1, in-select 6, out- slice 0, lu-a2d 1, in-select 9, out-
select 0, src_id 32 select 0, src_id 36
VLAN 10 VLAN 20
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1 Leaf-1(TAH-elam-insel6)# set Leaf-2(TAH-elam-insel6)# set
outer ipv4 src_ip 10.10.10.2 inner ipv4 src_ip 10.10.10.2
VLAN 10 VLAN 20
dst_ip 10.20.20.20 dst_ip 10.20.20.20
Leaf-1(TAH-elam-insel6)# start Leaf-2(TAH-elam-insel6)# start

Leaf-1(TAH-elam-insel6)# report Leaf-2(TAH-elam-insel6)# report

Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 ELAM
Scenario #3: VXLAN Flow – Is the fabric dropping the packet?
Spine-1
SIP: 192.168.100.1, DIP: Leaf-2# report
192.168.100.2 Leaf-1# report SUGARBOWL ELAM REPORT SUMMARY
{SIP: 10.10.10.2, DIP: SUGARBOWL ELAM REPORT slot - 1, asic - 0, slice - 0
10.20.20.20}
SUMMARY ============================
Lo1 Eth1/47 Eth1/3 Lo1 slot - 1, asic - 0, slice - 0
192.168.100.1 192.168.100.2 ============================ Incoming Interface: Eth1/3
Eth1/47 Eth1/3
Src Idx : 0x9, Src BD : 200
Leaf-2 Incoming Interface: Eth1/1 Outgoing Interface Info: dmod 1, dpid 17
Leaf-1
Src Idx : 0x1, Src BD : 10 Dst Idx : 0x5, Dst BD : 20
Outgoing Interface Info: dmod 1,
dpid 38 Outer Dst IPv4 address: 192.168.100.2
VLAN 10 VLAN 20 Dst Idx : 0xb9, Dst BD : 200 Outer Src IPv4 address: 192.168.100.1
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1

VLAN 10 VLAN 20 <snip> <snip>

Dst IPv4 address: 10.20.20.20 Inner Payload

Src IPv4 address: 10.10.10.2 Inner Dst IPv4 address: 10.20.20.20
Inner Src IPv4 address: 10.10.10.2
<snip>
Host 1 Host 2 <snip>
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 ELAM
Scenario #3: VXLAN Flow – Is the fabric dropping the packet?
Spine-1

Lo1 Eth1/47 Eth1/3 Lo1

192.168.100.1 192.168.100.2 Leaf-1# show system internal ethpm
Eth1/47 Eth1/3 Leaf-2# show system internal ethpm
info global | i dpid=38
info global | i dpid=17
Leaf-1 Leaf-2 IF_STATIC_INFO:
IF_STATIC_INFO:
port_name=Ethernet1/47,if_index:0x
port_name=Ethernet1/2,if_index:0x1
1a005c00,ltl=5960,slot=0,
a000200,ltl=6140,slot=0,
VLAN 10 VLAN 20 nxos_port=184,dmod=1,dpid=38,unit
nxos_port=4,dmod=1,dpid=17,unit=0,
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1 =0,queue=65535,xbar_unitbmp=0x0,
queue=65535,xbar_unitbmp=0x0,ns_
ns_pid=255,sli
VLAN 10 VLAN 20 pid=255,slice_
ce_num=0,port_on_slice=38,src_id=6
num=0,port_on_slice=17,src_id=34
8

Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 ELAM
Scenario #4: Are we dropping the packet?
Spine-1
Leaf-1# debug platform internal tah elam

Lo1 Eth1/47 Eth1/3 Lo1

Leaf-1(TAH-elam)# trig init
192.168.100.1 192.168.100.2 Slot 1: param values: start asic 0, start slice 0, lu-a2d 1, in-
Eth1/47 Eth1/3
select 6, out-select 0
Leaf-1 Leaf-2

Leaf-1(TAH-elam-insel6)# set outer ipv4 src_ip 10.10.10.2

VLAN 10 VLAN 20 dst_ip 10.20.20.20
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1

VLAN 20
VLAN 10
Leaf-1(TAH-elam-insel6)# start

Leaf-1(TAH-elam-insel6)# report
Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
 ELAM
Scenario #4: Are we dropping the packet?
Spine-1
Leaf-1# report
SUGARBOWL ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 0
============================
Lo1 Eth1/47 Eth1/3 Lo1 Incoming Interface: Eth1/1
192.168.100.1 192.168.100.2 Src Idx : 0x1, Src BD : 10
Eth1/47 Eth1/3 Outgoing Interface Info: dmod 1, dpid 38
Dst Idx : 0xb9, Dst BD : 200
Leaf-1 Leaf-2 <snip>
Dst IPv4 address: 10.20.20.20
Src IPv4 address: 10.10.10.2
<snip>
VLAN 10 VLAN 20 Drop Info:
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1 ----------

VLAN 20
Even though we show a
VLAN 10 LUA:
LUB: forwarding decision to another
LUC: interface, we are dropping the
LUD:
ACL_DROP packet due to this reason
Final Drops:
Host 1 Host 2 ACL_DROP
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
 ELAM
Scenario #4: Are we dropping the packet?
Spine-1

Leaf-1# show run interface vlan 10

!Command: show running-config interface Vlan10

Lo1 Eth1/47 Eth1/3 Lo1 !Running configuration last done at: Sun May 12 21:38:29 2024
192.168.100.1 192.168.100.2 !Time: Sun May 12 21:38:32 2024
Eth1/47 Eth1/3
version 10.3(4a) Bios:version 05.47
Leaf-1 Leaf-2
interface Vlan10
no shutdown
ip access-group ciscolive in
VLAN 10 VLAN 20 no ip redirects
10.10.10.1 Eth1/1 Eth1/2 10.20.20.1 ip address 10.10.10.1/24

VLAN 10 VLAN 20 Leaf-1# sh ip access-list ciscolive

IP access list ciscolive

10 deny ip 10.10.10.2/32 10.20.20.20/32
20 permit ip any Cisco live

Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
ELAM
Scenario #4: Are we dropping the packet?
• Most common drop codes for ELAM/codes that indicate bad forwarding:
• ACL_DROP: Frame/Packet matched on a deny entry for an ACL
• IP_MTU_CHECK_FAILURE: Frame failed the MTU check for the interface
• IP_SELF_FWD_FAILURE: IP Redirects enabled on the SVI
• ROUTING_DISABLED: Routing for particular vlan disabled
• SRC_VLAN_MBR: Packet/Frame received on an interface where the vlan is not configured/programmed
• TTL_EXPIRED: Packet received on an interface that causes the TTL to be decremented to zero, resulting in a
drop
• UC_DF_CHECK_FAILURE: vPC loop avoidance failure
• UC_PC_CFG_TABLE_DROP: No route in the VRF for the destination
• UC_RPF_FAILURE:
• UC_TENANT_MYTEP_BRIDGE_MISS: VXLAN leaf receiving traffic from a leaf for which it has not learned any
hosts/routes ; it does not have a peering with that VTEP
• UC_TENANT_MYTEP_ROUTE_MISS: VXLAN leaf in the particular tenant VRF does not have a route for the
given destination

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Tool #4:
Consistency-Checker
Consistency-Checker
Example Customer Scenario
Problem Description:
Host 2
192.168.0.167
You have devices in vlan 100 in a similar (simplified)
topology to that on the left. Some can resolve ARP and
VLAN 100
ping, while others cannot ping or even resolve ARP.
Eth1/49-50 Eth1/2

Core- Core-
9k-1 9k-2
Po100 How would you troubleshoot this?

VLAN 100 Eth1/1

Eth1/1

VLAN 100 VLAN 100

Host 1 Host 3
192.168.0.48 192.168.0.21

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 Consistency-Checker
Example Customer Scenario
Normal Troubleshooting Reveals:
Host 2
192.168.0.167 • Links from the switches to the affected devices (and
between the switches themselves) were tested and found to
VLAN 100 be clean
Eth1/49-50 Eth1/2 • MACs of affected devices are learned on relevant interfaces
Core- Core-
9k-1 9k-2 • STP found to be forwarding on all relevant interfaces
Po100

• No ACLs on the core switches

VLAN 100 Eth1/1
Eth1/1 • Host 2 can always resolve ARP and Host 3, and can resolve
VLAN 100 VLAN 100 ARP for Host 1, but not ping it
• Host 1 cannot resolve ARP for Host 2, but if Host 2 tries, it
successfully pings it
Host 1 Host 3
192.168.0.48 192.168.0.21

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Consistency-Checker
Example Customer Scenario
Troubleshooting Using Our Session’s Tools (So Far) Reveals:
Host 2
192.168.0.167 • No software switching of the traffic on either of the switches
for any flows, as per Ethanalyzer
VLAN 100

Eth1/49-50 Eth1/2 • Multiple ELAM captures reveal the following:

• For the ARP traffic between Hosts 1 and 3, the ARP Request is
Core- Core-
9k-1 9k-2
flooded and the Reply is seen to ingress and egress Eth1/49 on Core-
Po100
9k-1
• ICMP traffic between Hosts 1 and 3 shows the Request Entering
VLAN 100 Eth1/1
Core-9k-1 on Eth1/49, and the Reply is seen leaving Eth1/50
Eth1/1

VLAN 100 VLAN 100

• ARP traffic between Hosts 1 and 2 shows the Request makes it to
Host 2, and the Reply is sent by Core-9k-2 on Eth1/50
• When Host 3 attempts to Host 1, all traffic is seen on Eth1/49
between Eth1/49 on both switches?

Host 1 Host 3
Eth1/50 appears to have some issue, but what is it and how
192.168.0.48 192.168.0.21 can we figure it out?

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Consistency Checker
Forwarding State/Feature Consistency Verification Tool

• Validates consistency between software and hardware programming of many

features
• Grabs output directly from ASIC hardware tables and compares it to the existing
configuration
• Includes both L2 and L3 features and technologies such as Switching, Unicast/
Multicast Routing, VXLAN config checks, Interface Programming Checks
• Can complement troubleshooting done with Ethanalyzer and ELAM
• Syntax:
• Show consistency-checker [feature/technology] [feature options] [detail]

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Consistency-Checker
Example Customer Scenario
• Using the consistency checker for our problem:
Host 2
192.168.0.167 Core-9k-1# show consistency-checker member vlan 100
<snip>
VLAN 100

Eth1/49-50
Checking hardware for Module 1 Unit 0
Eth1/2
<snip>
Core- Core- Consistency Check: FAILED
9k-1
Po100
9k-2 Vlan:10, Hardware state consistent for:
Ethernet1/1
Ethernet1/49
VLAN 100 Eth1/1 Vlan:10, Hardware state inconsistent for:
Eth1/1
Ethernet1/50
VLAN 100 VLAN 100
• Other commands that would work in this instance:
show consistency-checker stp-state vlan 100
Host 1 Host 3 show consistency-checker l2 switchport int port-channel 100
192.168.0.48 192.168.0.21

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Consistency-Checker
Example Customer Scenario: Resolution Steps
• You’ve identified the underlying issue. Now what?
Host 2
192.168.0.167 • Steps :
• Grab a [show tech detail] and/or relevant feature show techs
VLAN 100
• Flap the relevant SVI/interface/route
Eth1/49-50 Eth1/2
• If flapping doesn’t work, perform [reload ascii]
Core- Core-
9k-1 9k-2
• Regardless of flapping/reloads working, take a show tech
Po100
detail/feature show tech after
• For further analysis, you can send your troubleshooting results and
VLAN 100 Eth1/1
the show techs to TAC
Eth1/1

VLAN 100 VLAN 100

Host 1 Host 3
192.168.0.48 192.168.0.21

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Tool #5:
Show Troubleshoot
Troubleshoot Command
• Helps us check MAC and IP route programming in hardware
• Gathers hardware and software table commands
• Nests and organizes commands and their output for more coherent viewing
Syntax
• L2:

Leaf-1#show troubleshoot l2 [mac|port-channel] [interface|vlan]

• L3

Leaf-1#show troubleshoot l3 [ipv4|ipv6] [src-ip|vrf] [IP|vrf_name]

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Troubleshoot Command
Example of L3 Troubleshoot Output
Leaf-1# show troubleshoot l3 ipv4 10.20.20.1 src-ip 10.10.10.1 vrf default
************************************ Check Route in RIB
CHECK ROUTE IN PI RIB
************************************
show ip route 10.20.20.20 vrf default

10.20.20.20/32, ubest/mbest: 1/0

*via 10.1.1.2, Eth1/47, [110/5], 00:17:13, ospf-1, intra
************************************
CHECK ROUTE IN PD FIB
************************************ Check Route in FIB
show forwarding route 10.20.20.20/32 vrf default
slot 1
=======
IPv4 routes for table default/base

------------------+-----------------------------------------+----------------------+-----------------+----------------- Notice this entry in

Prefix | Next-hop | Interface | Labels | Partial Install FIB matches with RIB
------------------+-----------------------------------------+----------------------+-----------------+-----------------
10.20.20.20/32 10.1.1.2 Ethernet1/47

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Troubleshoot Command
Example of L3 Troubleshoot Output

************************************
Check prefix
CHECK ROUTE IN UFIB
************************************ learning history
show forwarding internal trace v4-pfx-history module 1
Check Host
PREFIX 10.20.20.20/32 TABLE_ID 0x1
route entry in
Time usecs ha_handle next_obj next_obj_HH NH_cnt epoch operation
2024/04/24 13:59:57.239 58555 0xca9071fc V4 adj 0x7a127 1 1 Create
v4 host table

************************************
CHECK HOST ROUTE IN HARDWARE
************************************
show hardware internal tah l3 v4host | grep 10.20.20.20 Adjacency
HW Loc | Ip Entry | VRF | MPath | NumP | Base/L2ptr |CC|SR|DR|TD|DC|DE|LI|HR| Index Checks
-----------|------------------------|---------|-----------|-----------|----------------|--|--|--|--|--|--|--|
3/5 | 10.20.20.20 | 1 | No |0 | 0xd0004 | | | | | | | | |

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Troubleshoot Command
Example of L3 Troubleshoot Output

************************************ Checks if adjacency

CHECK FOR THE ADJACENCY is correct
************************************
show hardware internal tah l3 adjacency 0xd0004

AdjId | FP | BD | DMac | DstIdx | DstIsPtr |

-------------- |-------------------- |----------|------------------------ |---------- |------------|
0xd0004 | 13/0/0x4 | 4100 | e4:1f:7b:2f:a6:87 | 1 | No |

Leaf-1# sh ip arp | in 10.1.1.2 Entry matches with

10.1.1.2 00:06:24 e41f.7b2f.a687 Ethernet1/47 RIB and FIB
Leaf-1#

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Troubleshoot Command
Example of L3 Troubleshoot Output

************************************ Starting to check

RUNNING CONSISTENCY CHECKER consistency
************************************
************************************
CHECKING HARDWARE ASIC TYPE
************************************ Consistency Check
Please wait, consistency checker may take a while... Passed

Consistency checker passed for 10.20.20.20/32

Leaf-1#

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Troubleshoot Command
Example of L2 Troubleshoot Output

Leaf-1# show troubleshoot l2 mac 689e.0b8b.0327 vlan 10 detail

MAC: 689e.0b8b.0327 Vlan: 10
Show spanning-tree VLAN 10
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
10 VLAN0010 active Eth1/1

VLAN Type Vlan-mode

---- ----- ----------
10 enet CE

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Troubleshoot Command
Example of L2 Troubleshoot Output
Leaf-1# show troubleshoot l2 mac 689e.0b8b.0327 vlan 10 detail
MAC: 689e.0b8b.0327 Vlan: 10
Show spanning-tree VLAN 10
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
10 VLAN0010 active Eth1/1

VLAN Type Vlan-mode

---- ----- ----------
10 enet CE

show mac address-table address 689e.0b8b.0327 vlan 10

Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 689e.0b8b.0327 dynamic 0 F F Eth1/1

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Troubleshoot Command
Example of L2 Troubleshoot Output

show system internal l2fm l2dbg macdb address 689e.0b8b.0327 vlan 10

Legend
------
Db: 0-MACDB, 1-GWMACDB, 2-SMACDB, 3-RMDB, 4-SECMACDB 5-STAGEDB
Src: 0-UNKNOWN, 1-L2FM, 2-PEER, 3-LC, 4-HSRP
5-GLBP, 6-VRRP, 7-STP, 8-DOTX, 9-PSEC 10-CLI 11-PVLAN
12-ETHPM, 13-ALW_LRN, 14-Non_PI_MOD, 15-MCT_DOWN, 16 - SDB
17-OTV, 18-Deounce Timer, 19-AM, 20-PCM_DOWN, 21 - MCT_UP
22-VxLAN, 23-L2RIB 24-CTRL, 25-UFDM
Slot:0 based for LCS 31-MCEC 20-OTV/ORIB

VLAN: 10 MAC: 689e.0b8b.0327 FE ID: 0

Time If/swid Db Op Src Slot FE-BMP Count Detail
Apr 24 13:56:37 2024:176320 0x0901000a 0 REGDB_ADD 0 0 0xffff --
Apr 24 13:56:37 2024:176334 0x1a000200 0 MAC_NOTIF_AM_MOVE 1 0 0xffff --

VLAN: 10 MAC: 689e.0b8b.0327

Time If/swid Db Op Src Slot FE-BMP Count Detail
Apr 24 13:56:32 2024:729 0x1a000200 0 INSERT 3 0 0x1 --

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Tool #6: iCAM
Intelligent CAM Analytics and Machine-learning (iCAM)
• iCAM provides resource monitoring and analytics.
• You can obtain traffic, scale and resource (usage level) monitoring for the following resources and
functions:

Enable Feature icam and iCAM monitoring

monitoring
ACL TCAM entries ACL TCAM resource
Leaf-1(config)# feature icam utilization
Leaf-1(config)#icam monitor scale Forward information base (FIB) TCAM resource Layer 2 Switching
utilization
Unicast routing Multicast Routing
Beginning with 9.3(5)
VXLAN Process memory
This feature is enabled by default
Shared memory

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Intelligent CAM Analytics and Machine-learning (iCAM)
Using iCAM to check L2-Switching Scale
VLAN
Utilization Level
Leaf-1# show icam scale l2-switching Critical

-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Scale Limits for L2 Switching
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Feature Verified Config Cur Cur Threshold Polled
Scale Scale Scale Util Exceeded Timestamp
------------------------------------------------------------------------------------------------------------------------------------------------------------------
MAC Addresses - - - - - -
Mod:1,FE:0) 92000 92000 16 0.01 None 2024-05-09 13:54:50
VLANs 3967 3000 3839 127.96 Critical 2024-05-09 13:54:50
(VDC:1) - - 3839 127.96 Critical 2024-05-09 13:54:50
Isolated Port*Vlan 190000 190000 0 0 None 2024-05-09 13:54:50
(VDC:1) - - 0 0 None 2024-05-09 13:54:50
Leaf-1#

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Intelligent CAM Analytics and Machine-learning (iCAM)
Using iCAM to check Unicast Routing Scale
LPM Routes
Scale High
Leaf-1# show icam scale unicast-routing

-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Scale Limits for Unicast Routing
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Feature Verified Config Cur Cur Threshold Polled
Scale Scale Scale Util Exceeded Timestamp
------------------------------------------------------------------------------------------------------------------------------------------------------------------
IPVv4 LPM Routes - - - - - -
Mod:1) 6000 6000 5468 91.13 Warning 2024-05-09 13:54:50
IPVv6 LPM Routes - - - - - -
Mod:1) 1900 1900 03 0.15 None 2024-05-09 13:54:50

Leaf-1#. High Utilization

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Tool #7: PIE
PIE
Platform Telemetry Tool

• Platform Insights Engine

• Can help to identify or RCA issues with different
Layer 1 issues
• Supported on Cloudscale ToR switches starting on
10.2(1)
• EoR Cloudscale (not Fretta) linecards supported as of 10.2(2)

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
 PIE
Scenario #1: Link Failure
Spine-1

Leaf-1# show pie interface ethernet 1/47 link-down-rca

2024-05-01 04:56:42 Event Id: 00003377 Ethernet1/50

Eth1/47 Eth1/3
Source Id: 436232704 RCA Code: 1
Eth1/47 Eth1/3

Leaf-2
Reason: No Signal from peer is detected .Please check peer
Leaf-1
Eth1/47 shut configuration.
on Spine-1

Eth1/1 Eth1/2

• Apart of Link Debug Telemetry to assist with more granular L1 link issues
related to signaling

• Show pie interface ethernet X [link-down-rca|link-flap-rca] [detail]

Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
 PIE
Scenario #2: Optic Health
Spine-1

Leaf-1# show pie interface e1/47 transceiver-insights

2024-05-01 01:03:26 Event Id: 00008608 Event Class: xcvr DOM DB

Eth1/47 Eth1/3
Event Interface: Ethernet1/1
Eth1/47 Eth1/3

Leaf-2
Health Metric: --------BAD------- Mod: 01
Leaf-1
Leaf-1 showing
CRCs on E1/47

Eth1/1 Eth1/2

• Provides metrics for optics to indicate source of link flaps or optic health
with respect to current/voltage/power

• Only supported when DOM is supported on the transceiver

Host 1 Host 2 • Show pie interface ethernet X [transceiver-insights] [detail]

10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
 PIE
Scenario #3: Environment Monitoring (PSUs and FANs)
Spine-1

Leaf-1# show pie envmon fan

2024-05-01 05:36:32 Event Id: 00003419 Event Class: Fan insights
Source Id: 0x0102 Mod: 01
Eth1/47 Eth1/3
fan_health : PIE_ENTITY_HEALTH_BAD
Eth1/47
Leaf-1 showing
Eth1/3 Leaf-1# show pie envmon psu all
alerts
Leaf-2
2024-05-01 05:36:32 Event Id: 00003418 Event Class: PSU insights
Leaf-1
Source Id: 0x0101 Mod: 01
psu_health : PIE_ENTITY_HEALTH_GOOD

Eth1/1 Eth1/2

• Provides metrics for all fan and PSU statuses

• Show pie envmon fan [detail]

• Show pie envmon psu [#|all] [detail]

Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
 PIE
Scenario #4: CPU and Memory Top Talkers
Spine-1

Leaf-2# show pie envmon cpu-usage

2024-05-01 06:06:35 Event Id: 00003453 Event Class: CPU usage
insights Source Id: 0x0102 Mod: 01
Eth1/47 Eth1/3
Cpu_Health : Normal
Eth1/47 Eth1/3
Want to verify
CPU/memory
Leaf-2
Leaf-2# show pie envmon mem-usage
Leaf-1 status of Leaf-2
2024-05-01 06:06:35 Event Id: 00003454 Event Class: MEM usage
insights Source Id: 0x0102 Mod: 01
Memory_Health : OK
Eth1/1 Eth1/2

• Shows overall CPU/memory status and top 10 talkers

• Show pie env cpu-usage [detail]

• Show pie env mem-usage [detail]

Host 1 Host 2
10.10.10.2 10.20.20.20

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Summary
 Summary
Tool #1: Ethanalyzer – Captures packet going to and from CPU

Tool #2: SPAN-to-CPU – Captures Passthrough Traffic

Tool #3: ELAM – Captures 1 packet at a time

Tool #4: Consistency Checker – Checks consistency of various component

Tool #5: Show troubleshoot - Check hardware programming for l2 and l3

entries

Tool #6: iCAM - iCAM provides resource monitoring and analytic

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Conclusion
In conclusion, NXOS is a feature rich OS with lots of inbuilt tools.

If you know how to use them, it will expand your tool and empower you in your
troubleshooting.

#CiscoLive
Question & Answer
Additional Reference Resources
ELAM
https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/213848-nexus-9000-cloud-
scale-asic-tahoe-nx-o.html
https://www.youtube.com/watch?v=s0PSHN2Qxhc

Ethanalyzer

https://www.cisco.com/c/en/us/support/docs/switches/nexus-7000-series-switches/116136-trouble-ethanalyzer-
nexus7000-00.html

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
 • Visit the Cisco Showcase
for related demos

• Book your one-on-one

Meet the Engineer meeting

Continue • Attend the interactive education with

DevNet, Capture the Flag, and Walk-in
your education Labs

• Visit the On-Demand Library

for more sessions at
www.CiscoLive.com/on-demand

Contact me at: vinsawan@cisco.com, crissoto@cisco.com

TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
Complete Your Session Evaluations

Complete a minimum of 4 session surveys and the Overall Event Survey to be entered in
a drawing to win 1 of 5 full conference passes to Cisco Live 2025.

Earn 100 points per survey completed and compete on the Cisco Live Challenge
leaderboard.

Level up and earn exclusive prizes!

Complete your surveys in the Cisco Live mobile app.

#CiscoLive TACDCN-2010 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Thank you

#CiscoLive