In Data centers, where Leaf and Spine CLOS network topology is deployed, L2 and L3 VxLAN overlay packets gets forwarded over IP underlay. Traditional monitoring tools such as ping, trace-route could only provide information related to underlay path. For both reactive and proactive fault management of overlay networks, there arises a need to have Ethernet OAM like functions available for the overlay networks.

Like traditional Ethernet, the VxLAN networks need OAM (Operations, Administration, and Maintenance) services to verify connectivity, continuity, and fault isolation between Leaf switches. To achieve this, VxLAN OAM functionality is implemented in the OcNOS 7.0 release. It defines MEPs (Maintenance End Points) and MIPs (Maintenance Intermediate Points) within the VxLAN Leaf and Spine architecture.

Leafs are the nodes where the VxLAN tunnel endpoints (VTEPs) are logically located. It encapsulates standard Ethernet traffic from hosts into VxLAN packets for transport across the IP underlay and decapsulates them upon arrival. The document uses Leaf and VTEP terms interchangeably.

The Spine switch operates purely at the IP underlay level, utilizing routing protocols like OSPF and BGP to route encapsulated packets to the destination VTEPs. So, it is viewed as a transit node in the underlay path.

A VxLAN MEP is a logical endpoint of an OAM Maintenance Entity (ME) located at a VTEP. It is the OAM anchor point at the VTEP edge of a VxLAN segment. It is responsible for generating, transmitting, and terminating VxLAN OAM frames (like continuity check message, ping and pathtrace). VxLAN MEPs are uniquely identified using the VxLAN Network Identifier (VNI) and remote VTEP IP. The OAM feature works on the Down MEP.

A VxLAN MIP is a passive point inside the VxLAN underlay path (typically spine node). It does not originate OAM frames but can respond to certain OAM requests (like pathtrace). It is an intermediate transit point that allows operators to trace the underlay path hop by hop. The MIPs placement is conceptually inside the VxLAN tunnel between VTEPs.

In Leaf and Spine CLOS network topology, Equal-Cost Multi-Path (ECMP) routes exist to and from each leaf and spine switch, ensuring multiple redundant paths for traffic forwarding. Thus the VxLAN data packets traverse through multiple ECMP paths. Identifying the exact point of failure in the path is often complex. The CLI commands introduced in VxLAN OAM feature are designed to overcome these challenges. This feature provides the following OAM services to verify loopback, pathtrace and continuity check message (CCM) exchanges between MEPs:

In the VxLAN network topology, server nodes are connected to leaf switches on the access side, while the uplink ports of the leaf switches are connected to spine nodes.

Following is the flow of VxLAN OAM Packet communication:

The CFM objects (Maintenance Domain (MD) or Maintenance Association (MA) and Maintenance Endpoint (MEP) / Maintenance Intermediate Point (MIP)) required for the VxLAN OAM operations are created by default or explicitly through the configurations. The default CFM objects are created when the feature is enabled. And the explicit objects are created as part of the CFM module configurations. Both the implicit and explicit configurations can co-exist in the system at the same time. However, it is the VxLAN OAM operations which decides the implicit or explicit mode functioning, based on the supplied CLI parameters.

Below Table 1-1 explains the VxLAN OAM mode, configurations and the allowed operations.

For accurate packet path debugging, OAM packets (ping and pathtrace) must follow the same forwarding logic as host data packets. The OAM packets are generated directly on the VTEP node while VxLAN data typically originates from a host. Since the VTEP generates OAM packets locally, users can supply the flow parameters (such as ingress port, destination MAC, and VLAN) in the ping and pathtrace CLIs manually by copying the parameters of the host device to simulate a specific host's traffic.

Following are the example that shows the usage of flow parameters in ping and pathtrace CLIs.

 

It provides the capability of OAM operation for VxLAN networks.

Ensure the following prerequisites are met before configuring VxLAN OAM functionality:

This section details the systematic procedures required to establish the VxLAN OAM feature, starting with the Leaf-Spine (CLOS) network topology and providing step-by-step Command Line Interface (CLI) instructions to configure VTEPs, Spines, and access switches for OAM operation, BGP routing, and CFM maintenance points.

The VxLAN OAM feature is demonstrated using a standard 2-Tier Leaf-Spine (CLOS) topology. This topology ensures redundant paths and is fundamental to the EVPN architecture used for VxLAN control.

The topology shown in Figure 1-6 consists of:

The configuration includes the steps listed in the below table.

 

The following configuration sets the underlay router and an EVPN Route Reflector (RR) in a VxLAN/EVPN network.

The L2 access switch configuration sets up the VLAN trunking and uplinks toward VTEPs.

Execute following show commands to verify VxLAN OAM.

Verify the names and configuration details of the Maintenance Domain (MD) and Maintenance Association (MA) configured for VxLAN OAM.

Display the traffic statistics for all CFM OAM messages, including CCM sent/received, Loopback, Link Trace, and Path Trace counts.

Execute the following show commands to verify the connectivity status of MEP on the domain mdom2 associated with MA name ma2.

Verify the Connectivity Status (e.g., Fully Connected) for all configured local MEPs within MD mdom2/MA ma2.

Check for active OAM defects (e.g., defRDICCM, defRemoteCCM) present on MEPs in the MD mdom2.

Shows the total count of configured (MIPs, MEPs (Up/Down/Active), and active CCM sessions.

Display the details of the local MEPs (ID, direction, CCM status/interval, Peer-Addr) on MD mdom2/MA ma2..

Display the state of the remote MEPs (ID, Rx CCM status, RDI, Peer-IP) detected or expected on MD mdom2/MA ma2.

Display the detailed CCM status (State, Remote MEP IP and ID, VNID, last flap/recovery time) for a specific local MEP ID 12.

Display the detailed CCM status (State, Remote MEP IP and ID, VNID, last flap/recovery time) for a specific local MEP ID 13.

 

Display the detailed CCM status (State, Remote MEP IP and ID, VNID, last flap/recovery time) for a specific local MEP ID 14.

Execute the following show commands to verify the name and other detailed information about MA and MD.

Execute the following show commands to verify the connectivity status of MEP on the domain mdom2 associated with MA name ma2.

Verify the defects present in MD named mdom2.

Execute the following show commands to verify the information about the total CFM sessions count.

Execute the following show commands to verify the information about the local MEP on the MD name mdom2 associated with MA name ma2.

 

Execute the following show commands to verify the information about the remote MEPs on the MD named mdom2 associated with MA name ma2.

Execute the following show commands to verify the information about the CCM for the MEP ID 23 on the MD named mdom2 associated with the MA name ma2.

Execute the following show commands to verify the information about the CCM for the MEP ID 24 on MD named mdom2 associated with the MA name ma2.

Execute the following show commands to verify the name and other detailed information about MA and MD.

Execute the following show commands to verify the detailed information about CFM statistics of CCM, LBM, LBR, LTM, LTR and PTM.

Execute the following show commands to verify the connectivity status of MEP on the domain mdom2 associated with MA name ma2.

Execute the following show commands to verify the information about the total CFM sessions count.

Execute the following show commands to verify the information about the local MEP on the MD name mdom2 associated with MA name ma2.

Execute the following show commands to verify the information about the remote MEPs on the MD named mdom2 associated with MA name ma2.

Execute the following show commands to verify the information about the CCM for the MEP ID 31 on the MD named mdom2 associated with MA name ma2.

Execute the following show commands to verify the information about the CCM for the MEP ID 32 on the MD named mdom2 associated with the MA name ma2.

Execute the following show commands to verify the information about the CCM for the MEP ID 34 on MD named mdom2 associated with the MA name ma2.

Execute the following show commands to verify the name and other detailed information about MA and MD.

Execute the following show commands to verify the detailed information about CFM statistics of CCM, LBM, LBR, LTM, LTR and PTM.

Execute the following show commands to verify the information about the CCM for the MEP ID 41 on the MD domain named mdom2 associated with MA name ma2.

Execute the following show commands to verify the information about the CCM for the MEP ID 43 on the MD named mdom2 associated with the MA name ma2.

Verify the names, level, and MIP permission of the Maintenance Domain (MD) and Maintenance Association (MA) configured for VxLAN OAM spine mode.

Verify the names, level, and MIP permission of the Maintenance Domain (MD) and Maintenance Association (MA) configured for VxLAN OAM spine mode.

The example configuration shared below ensures that VTEPs (Virtual Tunnel Endpoints) can proactively detect connectivity failures across the fabric. It is a proactive monitoring framework for four VTEPs (Leaf1 through Leaf4) to ensure 100% reachability of the EVPN instance in an overlay network. The goal is to detect underlay path failures or VTEP service interruptions within one second, prior to host-level application timeouts.

Following are the sample CFM configuration running on all the VTEPs.

The following show output ensures that all MEPs connectivity status are fully connected.

The following show output ensures that no defects found in any MEPs.

The following output displays the MEPs count and the total CCM sessions count on each MEP.

The following output displays the MEP information of local and remote interface on each VTEPs. Also the CFM objects created on each VTEPs.

Note that the CCM feature is not supported for default MEP.

The following output shows the CFM statistics: CCM sent and received, LBM sent and LBR received, LTM sent and LTR received.

Note that the CCM send and received count is not considered for inactive MEP.

The below output verifies the connectivity from Leaf1 (1.1.1.1) to Leaf3 (3.3.3.3).

Identifies the specific underlay path in the transit hop from Leaf1 (1.1.1.1) to Spine1 (11.10.1.2 - IP configured on the ingress interface ce2) before reaching the target VTEP (Leaf3).

The following output shows the statics for all the local meps (meps 12,13,14)

After disabling cc multicast state on leaf3, the MEP state becomes down.

Leaf1 detects the loss of connectivity when cc multicast is disabled on Leaf3.

After enabling the cc multicast state on Leaf 3 (mep 31), the connectivity is recovered.

Following are the new CLIs introduced:

Use this VxLAN OAM ping command to test the reachability of VTEPs involved in the VxLAN tunnel. This CLI provides flow parameters to simulate the host packet arriving from the access port.

Privilege execute mode

This command was introduced before OcNOS version 7.0.0.

Use this command to find the exact path of a flow between leaf nodes. This CLI provides flow parameters to simulate the host packet arriving from the access port.

Specify the PTM packet outgoing interface name.

Privilege execute mode

This command is introduced in OcNOS version 7.0.0.

To verify pathtrace with outgoing interface

Use this command to display the CCM status between the VTEPs.

Privilege execute mode

This command was introduced before OcNOS version 7.0.0.

Use this command to enable the VxLAN OAM feature. Executing this command installs the necessary hardware filters.

None

Configuration Mode

This command was introduced before OcNOS version 7.0.0.

Executing the CLI on the leaf node creates implicit mode CFM objects.

Executing the CLI on the spine node creates Implicit mode CFM objects.

Verifying ping in implicit mode.

Executing the following CLIs on the spine node creates explicit mode CFM objects.

Verifying pathtrace in explicit mode

Use this command to display the CFM objects created, including the CFM objects.

None

NA

Configuration Mode

This command was introduced in OcNOS version 7.0.0.

The following existing CLIs have been modified:

Use this command to create a CFM Maintenance Domain (MD) and enter into Ethernet CFM mode. The CLI is modified to add a static option to the MD.

Configure mode

This command was introduced in OcNOS version 3.0. The parameter static is introduced in OcNOS version 7.0.0.

To create a static MD.

The CLI is modified to add a remote-vtep <VTEP-IP> parameter.

Ethernet CFM MA mode

is introduced in OcNOS version 7.0.0.

To configure a Down MEP for the connectivity status of a remote VTEP.

Use this command to display CFM statistics: CCM sent and received, LBM sent and LBR received, LTM sent and LTR

received. This CLI is modified to include pathtrace messages PTM and PTR statistics.

This command was introduced before OcNOS version 3.0. The output is modified to add path trace in OcNOS version 7.0.0

To display CCM, Ping and Pathtrace statistics using CFM command.

When troubleshooting the VxLAN OAM feature, focus on verifying the core EVPN and CFM components, as these form the foundation for OAM messages. Here are some suggested troubleshooting tips, grouped by logical area:

Confirm that VxLAN OAM is enabled on both the VTEPs and Spines.

Recognize the functionality differences based on mode.

Before troubleshooting the VxLAN overlay, ensure the underlay network can route IP traffic between all VTEP-IP addresses (loopbacks).

Use OAM Ping Command: Use the specific VxLAN OAM ping command to test reachability over the L2 VxLAN tunnel.

Use the detailed CCM status command to find recent flaps, the remote MEP ID, and RDI (Remote Defect Indication) status.

Use the OAM pathtrace command to determine the exact path the VxLAN packet takes, identifying which Spine (MIP) or VTEP is the point of failure.

If pathtrace stops at a Spine, confirm that the Spine has the matching MD and MA configured with mip-creation static.

If troubleshooting a specific flow, the flow-inport, flow-dst, and flow-vlan parameters are mandatory to select the correct ECMP path.

A local MEP must be configured as active true for CCM, ping and pathtrace to work.

Ensure the MD name, MA name, level, and the linked EVPN ID are configured identically on all VTEPs and spines.

The following provides definitions for key terms or abbreviations and their meanings used throughout this document: