This document describes the functionalities of ECN and PFC over VxLAN network for Artificial intelligence applications and other workloads that unitize RDMA over Converged Ethernet (RoCEv2) transport protocol.

Artificial intelligence and machine learning (AI/ML) workloads in modern data centers demand ultra-high bandwidth and low-latency network fabrics, typically utilizing RDMA over Converged Ethernet (RoCE) v2 transports to enable efficient Graphics Processing Unit (GPU)-to-GPU communication for distributed training. Because these applications are extremely sensitive to packet loss, even small amounts of dropped traffic can cause retransmissions, add latency, and significantly delay training completion.

To address this, data center fabrics are designed to be lossless, with:

The combination of Explicit Congestion Notification (ECN) based congestion signaling and PFC is fundamental to achieving predictable, efficient, and scalable network performance required for AI/ML workload in large-scale data centers.

When the ingress buffer for a lossless priority on a port reaches its limit, the device immediately transmits an XOFF (pause) frame to the link partner. This action halts traffic transmission for the Classes of Service (CoS) mapped to that congested priority, preventing packet loss while in-flight packets are buffered.

OcNOS facilitates this process by supporting CoS-to-queue and DSCP-to-queue mapping profiles, which route incoming traffic to one of up to eight available lossless priority queues..

ECN - Marking

When an egress queue exceeds its configured buffering threshold, ECN manages traffic flow by marking packets with congestion bits instead of dropping them. This marking strategy helps reduce the occurrence of PFC pauses, thereby improving overall network throughput and latency.

ECN and PFC work together to deliver lossless and efficient transport in the VxLAN network. While PFC provides a mechanism to pause incoming traffic when the buffering limit is reached, ECN complements it by signaling congestion before the buffers become full. Marking packets at early stages of congestion, allows endpoints to reduce their sending rate gracefully, rather than abruptly pauses traffic. This proactive approach lowers the frequency of PFC pause events, helping maintain higher throughput and reducing latency.

OcNOS supports the following for ECN to work in L3 VxLAN transport:

OcNOS supports the following for PFC to work in L2 and L3 VxLAN transport:

This feature delivers lossless Ethernet fabric required for transport of AI/ML and other workloads that utilize RoCEv2 transport protocol.

Ensure the IP underlay provides reachability between all VTEP loopback addresses. This connectivity, established via routing protocols like OSPF or BGP (both present in the base configurations), is fundamental for the VxLAN tunnels where PFC and ECN will function.

This section describes the configuration procedures for Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) over VxLAN. It involves applying PFC on underlay and access links, and enabling ECN within the Lossless queuing policy map applied to the hardware.

The following topology shows

Configuring PFC and ECN over VxLAN involves the following steps:

It includes the following sections:

This section describes the configuration of PFC on MH Access Switch.

 

 

This section shows the state of the VTEP1 switch under normal operation before any congestion was introduced.

Verify the status of established VxLAN tunnels. The output confirms that tunnels from the local VTEP1 (3.3.3.3) to remote VTEPs 5.5.5.5 (VTEP2) and 4.4.4.4 (VTEP3) are established.

Verify the detailed information about each configured VNID.

Verify the status of BGP sessions used for underlay reachability.

Verify the BGP EVPN sessions used for overlay control plane (MAC/IP advertisement).

Display the current transmit (Tx) and receive (Rx) traffic rates on interfaces.

The above output indicates an active traffic flow through the VxLAN fabric.

Display the configured QoS settings.

Display both the configured (Admin) and current operational state of PFC on all interfaces.

Verify the counters for PFC pause frames sent and received per priority on each interface.

The above output indicates that no PFC pausing occurred before congestion as all counters are zero.

This section shows the state after congestion was induced by applying a 1 Gbps shaper on the egress network interface cd3/1.

Display the QoS configuration to verify the congestion applied on the interface cd3/1.

Verify the traffic is throttled on ingress as per egress shape configurations.

The above output traffic rates reflect the congestion. The transmit rate on the shaped interface cd3/1 is capped around 0.70 Gbps. Consequently, the receive rate on the access interface xe0 has reduced to ~1.00 Gbps (down from ~7.98 Gbps) as the PFC throttles the ingress traffic. The rate on the other uplink cd4/1 is also adjusted (~0.30 Gbps).

Verify the PFC pause frames counters on ingress interface xe0 is incremented.

The pause sent counter for interface xe0 (the ingress access port receiving traffic) on priority 7 is rapidly incrementing. This means VTEP1 is sending PFC pause frames out of interface xe0 towards the connected host (SH1) for priority 7 because the downstream path (specifically the shaped egress cd3/1) is congested for traffic mapped to that priority.

Verify that the PFC administrative and operational status remain unchanged (identical to before congestion).