OcNOS SP : Troubleshooting Guide : Layer 2 Switching
Layer 2 Switching
This chapter contains steps to resolve Layer 2 switching issues.
Spanning Tree Protocol
This section shows how to resolve Spanning Tree Protocol (STP) issues.
 
Symptom/Cause
Solution
STP convergence
 
Interface might be shutdown.
Incorrect command syntax
BPDU is getting dropped
Use these commands:
show spanning-tree
Check the Root Id and Bridge ID. Non Root bridge should have the bridge id of the peer bridge
show spanning-tree statistics bridge 1
Check the bpdu transmitting and receiving.
If receiving is zero means packet is not receiving.
In this issue check the port state. Whether its up or down.
show interface description
Check the interface status.
show running-config
Verify the whether the command configured successfully.
Port flapping
 
Might be receiving the superior and inferior BPDU simultaneously
Handling of the BPDU is improper
Use these commands:
debug mstp all
debug mstp cli
debug mstp packet rx
debug mstp packet tx
debug mstp protocol detail
debug mstp timer detail
BPDU sending and receiving
 
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show running-config
Verify if the interface is not shut down by the user.
show spanning-tree statistics bridge 1
Check the bpdu transmitting and receiving. If receiving is zero, it means packet is not receiving.
show spanning-tree statistics interface p8p1 bridge 1
Check the packets send and receive per interface wise.
RSTP convergence
 
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show running-config
Verify if the interface is not shut down by the user.
show spanning-tree statistics bridge 1
Check the bpdu transmitting and receiving.
If receiving is zero means packet is not receiving.
show spanning-tree
In the output part please check the Root Id and Bridge ID. A non root bridge should have the bridge id of the peer bridge.
show spanning-tree statistics interface p8p1 bridge 1
Check the packets send and receive per interface wise.
MSTP convergence
 
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show running-config
Verify if the interface is not shut down by the user.
show spanning-tree statistics bridge 1
Check the bpdu transmitting and receiving.
If receiving is zero means packet is not receiving.
show spanning-tree mst detail
Check the Root Id and Bridge ID. A non Root bridge should have the bridge id of the peer bridge.
show spanning-tree statistics interface p8p1 bridge 1
Check the packets send and receive per interface wise.
RPVST convergence
 
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show running-config
Verify if the interface is not shut down by the user.
show spanning-tree statistics bridge 1
Check the bpdu transmitting and receiving.
If receiving is zero means packet is not receiving.
show spanning-tree rpvst detail
Check the Root Id and Bridge ID. A non Root bridge should have the bridge id of the peer bridge.
show spanning-tree statistics interface p8p1 bridge 1
Check the packets send and receive per interface wise.
BPDU Guard
 
Symptom/Cause
Solution
BPDU is not handled properly
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show spanning-tree
This command have bpdu-guard configured a field.
If its not showing configured (on) then its not get configured.
show running-config
Check the configuration of the root-guard. If it showing off means the bpdu guard is not configured. If it configured successfully then upon receiving the superior BPDU the port state will down. Once the port admin goes down then need to bring it up manually.
BPDU Filter
 
Symptom/Cause
Solution
BPDU is not handled properly
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show spanning-tree
This command have bpdu-filter configured a field.
If its not showing configured (on) then its not get configured.
show running-config
Check the configuration of the root-guard. If configured successfully then upon receiving the superior BPDU the bpdu-filter configured port will loose its bpdu-filter and it will become normal port.
Root Guard
 
Symptom/Cause
Solution
BPDU is not handled properly
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show spanning-tree
This command have root guard configured a field.
If its not showing configured then its not get configured.
show running-config
Check the configuration of the root-guard. On receiving superior BDPU then root guard port should move to the root-inconsistent. If its not behaving like this then handling of the packet is improper in the control plan.
VLANs
 
Symptom/Cause
Solution
Unable to create/delete VLAN with a bridge
 
Bridge is not VLAN aware
Incorrect command syntax
Maximum number of vlans created already
VLAN created or deleted already
Use these commands:
show vlan brief
Confirm if the VLAN is created.
show running-config
Verify the type of bridge configured and the number of VLANS already configured on the bridge.
VLAN tagged/untagged packets are not egressed/ingressed properly
 
Incorrect configuration on interface
Interface may be in down state
Interface may be in spanning-tree disabled state
Use these commands:
show vlan brief
Confirm if the VLAN is created.
show running-config
Verify the type of bridge configured and the number of VLANS already configured on the bridge.
show interface IF_NAME
View the input/output/dropped packet counters.
clear counters IF_NAME
Use this command to clear counter statistics for the interface.
VLAN interface is not operational
 
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show running-config
Verify if the interface is not shut down by the user.
show vlan brief
Check if the VLAN is bound to physical interface
show interface
Verify if the member interface is UP and RUNNING
Interfaces
Link Light for the Port does not come on
 
Symptom/Cause
Solution
No cable connected.
Connect cable from switch to a known good device.
Wrong Port
Make sure that both ends of the cable are plugged into the correct ports.
Device has no power
Ensure that both devices have power.
Wrong cable type
Verify the cable selection.
Bad cable
Swap suspect cable with known good cable. Look for broken or missing pins on connectors.
Loose connection
Check for loose connections. Sometimes a cable appears to be seated in the jack, but is not. Unplug the cable and reinsert it.
Patch Panels
Bypass the patch panel if possible to rule it out.
Media Convertors
Bypass the media convertor (e.g fiber-to-copper) if possible to rule it out.
Other
 
Symptom/Cause
Solution
Bad Port or Module Port or Interface or Module not enabled
Move the cable to a known good port to troubleshoot a suspect port or module.
Interface is down
Use this command:
show interface brief
Look for “Status” and “Reason”. If the “Status” is down and “Reason” is AD (Administratively Down), it means that the administrator has disabled the interface. Bring the interface up:
no shutdown
VLAN interface is down
Use these commands:
show interface brief
Check if the corresponding VLAN interface “Status” is up or down. Also check the value in the “Reason” column.
show vlan brief
Check if the “State” field to see if it is in “SUSPEND”. Enable the van:
vlan <vlan-name> bridge <bridge-id> state enable”
Possible cause might be no switchport given on the interface attached to VLAN due to which van is detached from the interface. Enable switchport on that particular interface and attach the van and bridge again.
show running-config interface <interface-name>
View the configuration.
Link Aggregation
 
Symptom/Cause
Solution
LACP packets not transmitted properly
 
Interface is down
Channel group is configured as passive at both the nodes
Use these commands:
show lacp-counter
Verify if there are any packets are sent out of interface.
show interface IF_NAME description
Check if the interface is up
show running config
Check if Active mode is configured at one
LACP packets not received properly
 
Interface is down
Channel group is configured as passive at both the nodes
Use these commands:
show lacp-counter
Verify if there are any packets are sent out of interface.
show interface IF_NAME description
Check if the interface is up
show running config
Check if Active mode is configured at one
LACP sync is not up
 
Interface is down
Channel group is configured as passive at both the nodes
Link bandwidth mismatch
MTU mismatch
VLAN mismatch
DUPLEX mismatch
Use these commands:
show etherchannel detail
Check the Actor system ID, partner system ID, interface coming under the aggregation and sync bit
show etherchannel summary
Check the sync bit.
show interface IF_NAME description
Check if the interface is up, Bandwidth, duplex and MTU. Check if Active mode is configured at one
MLAG
 
Symptom/Cause
Solution
MLAG Domain adjacency between TORs to determine active / standby during switchover
Use these commands:
show mlag domain summary
Verify domain and MLAG sync configuration.
show lagd mlag 1
To list the properties / Neighbor partner information show lagd stp-config.
STP MLAG Configurations Present in LAGD
To list the MCCPDU received on the system/ MCEC debugs
Use these commands:
show mcec statistics
clear mcec statistics
Command to display /clear the MCC PDU statistics
show mlag stp-synchronization status
To check STP MLAG Configuration / Sync status
debug mcec (timer | event | hello | info | cli | mac-sync | stp |all )
Command to enable debug messages for MCEC module
LLDP
 
Symptom/Cause
Solution
LLDP packets not transmitted properly
 
Interface is down
LLDP agent not enabled
LLDP tx not enabled
Use these commands:
show lldp interface IF_NAME
Verify if there are any packets discarded.
show interface IF_NAME description :
Check if the interface is up
LLDP packets not received properly
 
Interface is down
LLDP agent not enabled
LLDP rx not enabled
Use these commands:
show lldp interface IF_NAME
Verify if there are any packets received in error.
show interface IF_NAME description :
Check if the interface is up
LLDP system name, system description not seen in the peer
 
Interface is down
LLDP agent not enabled
LLDP rx/tx not enabled
Basic management capabilities are not enabled.
Use these commands:
show lldp interface IF_NAME neighbor
Check the neighbor information
show running config
Check if system name/description are configured at the sending side
Note: This applies to many other lldp capabilities like ieee-8021-org-specific, ieee-8023-org-specific, and med capabilities. The needed capabilities must be enabled on the device using lldp tlv-select.
All VLANs configured not displayed in the remote VLAN details of the peer device
 
VLAN not configured for the interface
This is a special case where there are more than 82 vlans and the TLV size is increasing the MTU. As vlan details are encoded after encoding all the other TLVs. So whenever the buffer becomes full, the end of TLV is sent automatically and the rest of the vlan information is not displayed. This is the expected behavior.
LLDP med capabilities are not displayed in the neighbor
 
Interface is not up
LLDP agent not enabled
Tx/rx not enabled
LLDP tlv-type med capabilities not enabled
LLDP med dev-type has to be end point class3 at least on one side. It doesn’t sent its med capabilities if net-connect is enabled until it receives med info from the neighbor
Use these commands:
show lldp interface IF_NAME
Verify if tx/rx, med is enabled.
show interface IF_NAME descriptio
Check if the interface is up
show lldp interface IF_NAME neighbor
Verify the med related information of neighbor
DCB
 
Symptom/Cause
Solution
show data-center-bridging (admin-details | operational-details | remote-details)
interface IFNAME is not showing any output even after configuring DCB
 
Interface is not up
LLDP agent is not enabled
TxRx is not enabled
Use these commands:
lldp-agent
set lldp enable txrx
lldp tlv ieee-8021-org-specific
Correct PFC operational details not shown in ON-AUTO mode
 
Interface is not up
LLDP agent is not enabled
Capabilities of peer are less than that of device
Set the cap of PFC:
priority-flow-control cap <0-8>
 
Note: When the device’s local port willing bit is NOT set (ON mode) and peer’s willing bit is set (AUTO mode):
-- Local machine.
Operational details == Admin details
-- Peer machine
Operational details == Remote details
Correct PFC operational details not shown in AUTO-AUTO mode
 
Interface is not up.
LLDP agent is not enabled.
Capabilities of port with lower MAC is lesser than that of the one with higher MAC
Set the cap of PFC:
priority-flow-control cap <0-8>
 
Note: When the device’s local port willing bit is set (AUTO mode) and peer’s willing is also set (AUTO mode):
for device’s MAC address < Peer’s MAC address
Operational details == Remote details
for device’s MAC address > Peer’s MAC address
Operational details == Admin details
Proper PFC operational details are not shown in ON-ON mode
 
Interface is not up.
LLDP agent is not enabled.
Capabilities of port with lower MAC is lesser than that of the one with higher MAC
Set the cap of PFC:
priority-flow-control cap <0-8>
 
Note: When the device’s local port willing bit is set (AUTO mode) and peer’s willing is also set (AUTO mode):
-- for device’s MAC address < Peer’s MAC address
Operational details == Remote details
-- for device’s MAC address > Peer’s MAC address
Operational details == Admin details
Not able to add any more traffic class groups
 
Interface is not up.
LLDP agent is not enabled.
Max traffic class group may be lesser than the number of traffic class groups added.
Check max traffic class group:
max-traffic-class-group <0-7>
Note: Use the no form of this command to set the maximum number of traffic classes to 0 which is the default.
If the value is 0, you can add priorities for all 8 traffic classes with the traffic-class-group command. When the number
of traffic classes is not 0, you can add priorities with the traffic-class-group command until the maximum is reached.
no max-traffic-class-group
802.1x
 
Symptom/Cause
Solution
Packets does not reach to device
 
Loose connection between device and XSUPPLICANT
Following XSUPPLICANT scripts need to be modified based upon the connections (on which interface we need to allow EAPOL packets and on which interfaces we have to deny) and the IP address assigned.
/usr/local/etc/1x/ md5-example.conf
/usr/local/etc/1x/startup.sh
/usr/local/etc/1x/startup2.sh
The very first packet sent to device should be EAPOL packet. Data packets will be dropped if the first packet is not EAPOL.
Packets do not reach to that interface of device which is connected to radius server.
 
Packets must be getting dropped at kernel level due to some malformed fields.
The kernel must not be lifting those packets to the protocol level.
The decoding of the packet could have some issue.
The other interface is down.
 
Packets do not reach to Radius Server
 
Loose connection between device and RADIUS SERVER.
IP Configured at device for RADIUS SERVER must not match the IP configured at the RADIUS SERVER’s interface connected with device.
 
Radius Server does not reply back to device
 
The RADIUS SERVER must not be having the XSUPPLICANT details with it
Check following files:
/usr/local/etc/raddb/users
The entry corresponding to the mac address of the xsupplicant interface which is connected to device should be updated.
 
00:02:A5:4E:FF:83 Auth-Type := eap, User-Password == "00:02:A5:4E:FF:83"
Tunnel-type:0 = 13,
Tunnel-Medium-Type:0 = 6,
Tunnel-Private-Group-ID:0 = 201,
Reply-Message = "Hello, %u"
 
/usr/local/etc/raddb/clients.conf
Ip should be updated
 
client 10.10.10.40/24 {
secret = authd
shortname = device
}
Radius Server replies back to device with “access challenge”
 
Authentication issues at RADIUS SERVER
Credentials must be verified properly.
XSUPPLICANT should retrigger the EAPOL packets
FDB does not get updated at device accordingly
 
Problem with authentication of the particular MAC or port.
Retrigger the EAPOL packet and check above mentioned scenarios one by one to find whether packets are getting dropped somewhere or there is some problem with the credential matching.
IGMP Snooping
 
Symptom/Cause
Solution
IGMP Report not learned
 
IGMP Snooping disabled globally.
IGMP Snooping disabled on vlan interface.
Port on which report is sent in discarding state (xSTP).
Destination MAC of the frame does not correspond to Destination IP multicast address.
TTL of the IP packet not 1.
For IGMPv3 report destination address of IP packet not 224.0.0.22.
Use these commands:
show igmp snooping interface
Check if IGMP snooping is enabled globally and on interface.
If underlying L2 interface is in forwarding state in spanning tree then it will be listed below vlan interface.
show running-config
Confirm if igmp snooping is disabled globally or on interface.
IGMP reports forwarded by switch
 
Snooping switch sending reports only when it receives an IGMP query
Use these commands:
show igmp snooping mrouter VLAN-IFNAME
Confirm mrouter interface learned
IGMP Queries sent with Source IP as zero
 
Switch sending queries with SIP as 0 when it is configured as querier, otherwise queries received from router will be forwarded as is to all interfaces bound to that VLAN.
Use these commands:
show igmp snooping interface VLAN-IFNAME
Check if querier is enabled on the interface
show running-config
Confirm the configuration of querier.
Traffic not forwarded from the source specified in the IGMP membership Report.
 
The resulting state of the source might be in exclude list OR
The group might have been expired
Use these commands:
show igmp snooping groups detail
Confirm if the source is in exclude list or include list of the Group and to check expiry timer, if it is dynamic group then it will not be shown in this command after expiry.
PVLAN
 
Symptom/Cause
Solution
Private VLAN cannot be created
 
VLAN is not created.
Incorrect command Syntax
Given Bridge Group is incorrect.
 
 
 
 
 
 
 
With the issue of “no shutdown” command, vlan interface is not activated
 
Due to the nature of Private VLANs, you cannot activate the VLAN interface for isolated or community VLANs. You can only activate the VLAN interface that belongs to primary VLAN.
Use these commands:
show vlan brief
Confirm if the VLAN is created.
show running-config
Confirm the bridge type created. Private Vlan cannot be created for provider bridges
 
 
 
 
 
Use this command:
show vlan private-vlan bridge BRIDGE_GROUP
Confirm the type of Private vlans configured
Not able to associate a Secondary PVLAN to the Primary PVLAN
 
Secondary Private VLAN type is Isolated and one isolated PVLAN is already associated with the particular Primary PVLAN.
Secondary PVLAN is already associated with some other Primary PVLAN.
Use this command:
show vlan private-vlan bridge BRIDGE_GROUP
Confirm the types of PVLANs configured and associations of Secondary PVLANs with Primary PVLANs.
Not able to configure an interface as private-vlan host-port
 
Interface is not access-port
Use this command:
show interface IFNAME
Verify the port mode. Only Access ports can be configured as host-ports.