OcNOS-DC : Troubleshooting Guide : Layer 2 Switching
Layer 2 Switching
This chapter contains steps to resolve Layer 2 switching issues.
Spanning Tree Protocol
This section shows how to resolve Spanning Tree Protocol (STP) issues.
 
Symptom/Cause
Solution
STP convergence
 
Interface might be shutdown.
Incorrect command syntax
BPDU is getting dropped
Use these commands:
show spanning-tree
Check the Root Id and Bridge ID. Non Root bridge should have the bridge id of the peer bridge
show spanning-tree statistics bridge 1
Check the bpdu transmitting and receiving.
If receiving is zero means packet is not receiving.
In this issue check the port state. Whether its up or down.
show interface description
Check the interface status.
show running-config
Verify the whether the command configured successfully.
Port flapping
 
Might be receiving the superior and inferior BPDU simultaneously
Handling of the BPDU is improper
Use these commands:
debug mstp all
debug mstp cli
debug mstp packet rx
debug mstp packet tx
debug mstp protocol detail
debug mstp timer detail
BPDU sending and receiving
 
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show running-config
Verify if the interface is not shut down by the user.
show spanning-tree statistics bridge 1
Check the bpdu transmitting and receiving. If receiving is zero, it means packet is not receiving.
show spanning-tree statistics interface p8p1 bridge 1
Check the packets send and receive per interface wise.
RSTP convergence
 
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show running-config
Verify if the interface is not shut down by the user.
show spanning-tree statistics bridge 1
Check the bpdu transmitting and receiving.
If receiving is zero means packet is not receiving.
show spanning-tree
In the output part please check the Root Id and Bridge ID. A non root bridge should have the bridge id of the peer bridge.
show spanning-tree statistics interface p8p1 bridge 1
Check the packets send and receive per interface wise.
MSTP convergence
 
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show running-config
Verify if the interface is not shut down by the user.
show spanning-tree statistics bridge 1
Check the bpdu transmitting and receiving.
If receiving is zero means packet is not receiving.
show spanning-tree mst detail
Check the Root Id and Bridge ID. A non Root bridge should have the bridge id of the peer bridge.
show spanning-tree statistics interface p8p1 bridge 1
Check the packets send and receive per interface wise.
RPVST convergence
 
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show running-config
Verify if the interface is not shut down by the user.
show spanning-tree statistics bridge 1
Check the bpdu transmitting and receiving.
If receiving is zero means packet is not receiving.
show spanning-tree rpvst detail
Check the Root Id and Bridge ID. A non Root bridge should have the bridge id of the peer bridge.
show spanning-tree statistics interface p8p1 bridge 1
Check the packets send and receive per interface wise.
BPDU Guard
 
Symptom/Cause
Solution
BPDU is not handled properly
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show spanning-tree
This command have bpdu-guard configured a field.
If its not showing configured (on) then its not get configured.
show running-config
Check the configuration of the root-guard. If it showing off means the bpdu guard is not configured. If it configured successfully then upon receiving the superior BPDU the port state will down. Once the port admin goes down then need to bring it up manually.
BPDU Filter
 
Symptom/Cause
Solution
BPDU is not handled properly
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show spanning-tree
This command have bpdu-filter configured a field.
If its not showing configured (on) then its not get configured.
show running-config
Check the configuration of the root-guard. If configured successfully then upon receiving the superior BPDU the bpdu-filter configured port will loose its bpdu-filter and it will become normal port.
Root Guard
 
Symptom/Cause
Solution
BPDU is not handled properly
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show spanning-tree
This command have root guard configured a field.
If its not showing configured then its not get configured.
show running-config
Check the configuration of the root-guard. On receiving superior BDPU then root guard port should move to the root-inconsistent. If its not behaving like this then handling of the packet is improper in the control plan.
VLANs
 
Symptom/Cause
Solution
Unable to create/delete VLAN with a bridge
 
Bridge is not VLAN aware
Incorrect command syntax
Maximum number of vlans created already
VLAN created or deleted already
Use these commands:
show vlan brief
Confirm if the VLAN is created.
show running-config
Verify the type of bridge configured and the number of VLANS already configured on the bridge.
VLAN tagged/untagged packets are not egressed/ingressed properly
 
Incorrect configuration on interface
Interface may be in down state
Interface may be in spanning-tree disabled state
Use these commands:
show vlan brief
Confirm if the VLAN is created.
show running-config
Verify the type of bridge configured and the number of VLANS already configured on the bridge.
show interface IF_NAME
View the input/output/dropped packet counters.
clear counters IF_NAME
Use this command to clear counter statistics for the interface.
VLAN interface is not operational
 
Interface is admin shutdown
No associated physical interfaces
All associated physical interfaces are down
Use these commands:
show running-config
Verify if the interface is not shut down by the user.
show vlan brief
Check if the VLAN is bound to physical interface
show interface
Verify if the member interface is UP and RUNNING
Interfaces
Link Light for the Port does not come on
 
Symptom/Cause
Solution
No cable connected.
Connect cable from switch to a known good device.
Wrong Port
Make sure that both ends of the cable are plugged into the correct ports.
Device has no power
Ensure that both devices have power.
Wrong cable type
Verify the cable selection.
Bad cable
Swap suspect cable with known good cable. Look for broken or missing pins on connectors.
Loose connection
Check for loose connections. Sometimes a cable appears to be seated in the jack, but is not. Unplug the cable and reinsert it.
Patch Panels
Bypass the patch panel if possible to rule it out.
Media Convertors
Bypass the media convertor (e.g fiber-to-copper) if possible to rule it out.
Other
 
Symptom/Cause
Solution
Bad Port or Module Port or Interface or Module not enabled
Move the cable to a known good port to troubleshoot a suspect port or module.
Interface is down
Use this command:
show interface brief
Look for “Status” and “Reason”. If the “Status” is down and “Reason” is AD (Administratively Down), it means that the administrator has disabled the interface. Bring the interface up:
no shutdown
VLAN interface is down
Use these commands:
show interface brief
Check if the corresponding VLAN interface “Status” is up or down. Also check the value in the “Reason” column.
show vlan brief
Check if the “State” field to see if it is in “SUSPEND”. Enable the van:
vlan <vlan-name> bridge <bridge-id> state enable”
Possible cause might be no switchport given on the interface attached to VLAN due to which van is detached from the interface. Enable switchport on that particular interface and attach the van and bridge again.
show running-config interface <interface-name>
View the configuration.
Link Aggregation
 
Symptom/Cause
Solution
LACP packets not transmitted properly
 
Interface is down
Channel group is configured as passive at both the nodes
Use these commands:
show lacp-counter
Verify if there are any packets are sent out of interface.
show interface IF_NAME description
Check if the interface is up
show running config
To list the LACP running configuration
LACP packets not received properly
 
Interface is down
Channel group is configured as passive at both the nodes
Use these commands:
show lacp-counter
Verify if there are any packets are sent out of interface.
show interface IF_NAME description
Check if the interface is up
LACP sync is not up
 
Interface is down
Channel group is configured as passive at both the nodes
Link bandwidth mismatch
MTU mismatch
VLAN mismatch
DUPLEX mismatch
Use these commands:
show etherchannel detail
Check the Actor system ID, partner system ID, interface coming under the aggregation and sync bit
show etherchannel summary
Check the sync bit.
show interface IF_NAME description
Check if the interface is up, Bandwidth, duplex and MTU. Check if Active mode is configured at one
Multi-Chassis Link Aggregation
 
Symptom/Cause
Solution
MLAG Domain adjacency between TORs to determine active / standby during switchover
Use these commands:
show mlag domain summary
Verify domain and MLAG sync configuration.
show lagd mlag 1
To list the properties / Neighbor partner info
show lagd stp-config
STP MLAG Configurations Present in LAGD
To list the MCCPDU received on the system/ mcec debugs
Use these commands:
show mcec statistics
clear mcec statistics
Command to display /clear the mccpdu statistics
show mlag stp-synchronization status
To check STP MLAG Configuration / Sync status
debug mcec (timer | event | hello | info | cli | mac-sync | stp |all )
Command to enable debug messages for MCEC module
LLDP
 
Symptom/Cause
Solution
LLDP packets not transmitted properly
 
Interface is down
LLDP agent not enabled
LLDP tx not enabled
Use these commands:
show lldp interface IF_NAME
Verify if there are any packets discarded.
show interface IF_NAME description
Check if the interface is up
LLDP packets not received properly
 
Interface is down
LLDP agent not enabled
LLDP rx not enabled
Use these commands:
show lldp interface IF_NAME
Verify if there are any packets received in error.
show interface IF_NAME description :
Check if the interface is up
LLDP system name, system description not seen in the peer
 
Interface is down
LLDP agent not enabled
LLDP rx/tx not enabled
Basic management capabilities are not enabled.
Use these commands:
show lldp interface IF_NAME neighbor
Check the neighbor information
show running config
Check if system name/description are configured at the sending side
Note: This applies to many other lldp capabilities like ieee-8021-org-specific, ieee-8023-org-specific, and med capabilities. The needed capabilities must be enabled on the device using lldp tlv-select.
All VLANs configured not displayed in the remote VLAN details of the peer device
 
VLAN not configured for the interface
This is a special case where there are more than 82 vlans and the TLV size is increasing the MTU. As vlan details are encoded after encoding all the other TLVs. So whenever the buffer becomes full, the end of TLV is sent automatically and the rest of the vlan information is not displayed. This is the expected behavior.
LLDP med capabilities are not displayed in the neighbor
 
Interface is not up
LLDP agent not enabled
Tx/rx not enabled
LLDP tlv-type med capabilities not enabled
LLDP med dev-type has to be end point class3 at least on one side. It doesn’t sent its med capabilities if net-connect is enabled until it receives med info from the neighbor
Use these commands:
show lldp interface IF_NAME
Verify if tx/rx, med is enabled.
show interface IF_NAME descriptio
Check if the interface is up
show lldp interface IF_NAME neighbor
Verify the med related information of neighbor
802.1x
 
 
Symptom/Cause
Solution
Packets does not reach to device
 
Loose connection between device and XSUPPLICANT
Following XSUPPLICANT scripts need to be modified based upon the connections (on which interface we need to allow EAPOL packets and on which interfaces we have to deny) and the IP address assigned.
/usr/local/etc/1x/ md5-example.conf
/usr/local/etc/1x/startup.sh
/usr/local/etc/1x/startup2.sh
The very first packet sent to device should be EAPOL packet. Data packets will be dropped if the first packet is not EAPOL.
Packets do not reach to that interface of device which is connected to radius server.
 
Packets must be getting dropped at kernel level due to some malformed fields.
The kernel must not be lifting those packets to the protocol level.
The decoding of the packet could have some issue.
The other interface is down.
 
Packets do not reach to Radius Server
 
Loose connection between device and RADIUS SERVER.
IP Configured at device for RADIUS SERVER must not match the IP configured at the RADIUS SERVER’s interface connected with device.
 
Radius Server does not reply back to device
 
The RADIUS SERVER must not be having the XSUPPLICANT details with it
Check following files:
/usr/local/etc/raddb/users
The entry corresponding to the mac address of the xsupplicant interface which is connected to device should be updated.
 
00:02:A5:4E:FF:83 Auth-Type := eap, User-Password == "00:02:A5:4E:FF:83"
Tunnel-type:0 = 13,
Tunnel-Medium-Type:0 = 6,
Tunnel-Private-Group-ID:0 = 201,
Reply-Message = "Hello, %u"
 
/usr/local/etc/raddb/clients.conf
Ip should be updated
 
client 10.10.10.40/24 {
secret = authd
shortname = device
}
Radius Server replies back to device with “access challenge”
 
Authentication issues at RADIUS SERVER
Credentials must be verified properly.
XSUPPLICANT should retrigger the EAPOL packets
FDB does not get updated at device accordingly
 
Problem with authentication of the particular MAC or port.
Retrigger the EAPOL packet and check above mentioned scenarios one by one to find whether packets are getting dropped somewhere or there is some problem with the credential matching.
VLAN Cross-Connect
 
Symptom/Cause
Solution
Cross-connect is not up
Check cross-connect status
 
Interface is down
Use these commands:
show cross-connect
Verify cross-connect is up.
show interface IF_NAME
Check if the interface is up
Cross-connect packets not received properly
Interface is down
 
Check cross-connect status
 
Check interface stats
Use these commands:
show interface IF_NAME
Check if the interface is up
show cross-connect
Verify cross-connect is up.
show interface IF_NAME counters rate mbps
Verify interface stats are incrementing properly
 
 
 
 
 
 
 
 
 
 
IGMP Snooping
 
Symptom/Cause
Solution
IGMP Report not learned
 
IGMP Snooping disabled globally.
IGMP Snooping disabled on vlan interface.
Port on which report is sent in discarding state (xSTP).
Destination MAC of the frame does not correspond to Destination IP multicast address.
TTL of the IP packet not 1.
For IGMPv3 report destination address of IP packet not 224.0.0.22.
Use these commands:
show igmp snooping interface
Check if IGMP snooping is enabled globally and on interface.
If underlying L2 interface is in forwarding state in spanning tree then it will be listed below vlan interface.
show running-config
Confirm if igmp snooping is disabled globally or on interface.
IGMP reports forwarded by switch
 
Snooping switch sending reports only when it receives an IGMP query
Use these commands:
show igmp snooping mrouter VLAN-IFNAME
Confirm mrouter interface learned
IGMP Queries sent with Source IP as zero
 
Switch sending queries with SIP as 0 when it is configured as querier, otherwise queries received from router will be forwarded as is to all interfaces bound to that VLAN.
Use these commands:
show igmp snooping interface VLAN-IFNAME
Check if querier is enabled on the interface
show running-config
Confirm the configuration of querier.
Traffic not forwarded from the source specified in the IGMP membership Report.
 
The resulting state of the source might be in exclude list OR
The group might have been expired
Use these commands:
show igmp snooping groups detail
Confirm if the source is in exclude list or include list of the Group and to check expiry timer, if it is dynamic group then it will not be shown in this command after expiry.
PVLAN
 
Symptom/Cause
Solution
Private VLAN cannot be created
 
VLAN is not created.
Incorrect command Syntax
Given Bridge Group is incorrect.
 
 
 
 
 
 
 
With the issue of “no shutdown” command, vlan interface is not activated
 
Due to the nature of Private VLANs, you cannot activate the VLAN interface for isolated or community VLANs. You can only activate the VLAN interface that belongs to primary VLAN.
Use these commands:
show vlan brief
Confirm if the VLAN is created.
show running-config
Confirm the bridge type created. Private Vlan cannot be created for provider bridges
 
 
 
 
 
Use this command:
show vlan private-vlan bridge BRIDGE_GROUP
Confirm the type of Private vlans configured
Not able to associate a Secondary PVLAN to the Primary PVLAN
 
Secondary Private VLAN type is Isolated and one isolated PVLAN is already associated with the particular Primary PVLAN.
Secondary PVLAN is already associated with some other Primary PVLAN.
Use this command:
show vlan private-vlan bridge BRIDGE_GROUP
Confirm the types of PVLANs configured and associations of Secondary PVLANs with Primary PVLANs.
Not able to configure an interface as private-vlan host-port
 
Interface is not access-port
Use this command:
show interface IFNAME
Verify the port mode. Only Access ports can be configured as host-ports.