The Switch packet buffer tuning, is a system aimed at enhancing network switch performance by avoiding congestion and packet drops. It refers to how a switch allocates packet buffer size based on traffic priority classes referred as Priority Groups (PG) instead of physical ports. Buffers temporarily store packets during congestion. Each PG competes for shared memory in the switch buffer pool.

In Data Center Bridging (DCB) or QoS (Quality of Service) implementations, traffic is grouped based on priority. Each group corresponds to a certain class of service, for example, storage traffic, control traffic, bulk data, best-effort, etc. Tuning the packet buffer size at the PG level ensures that each class of traffic gets the right amount of buffer space to avoid congestion and packet drops.

The section explains the configuration of packet buffers and CLIs introduced to implement the functionalities.

OcNOS can be customized to respond to Priority-based Flow Control (PFC) traffic to manage when to pause frames to prevent packet loss.

Custom Device Responses and Ease of Configuration: Switch Packet Buffer Tuning is indeed designed to allow for custom device responses to Priority-based Flow Control (PFC) pause storms. This feature grants users the ability to control precisely when the switch transmits pause frames to upstream devices. Furthermore, it simplifies configuration by enabling the global adjustment of buffer limits. Pausing the frames temporarily halt the traffic transmission with the link partner which effectively prevents the packet loss due to congestion.

Priority Group (PG) Configuration: Enables to configure each Priority Group (PG) port with a specific limit on the amount of shared memory it can receive. Priority groups are essential for classifying ingress packets based on their input priority.

PFC XOFF Threshold and XON Offsets: PGs can be configured using PFC XOFF threshold and XON offsets. XOFF threshold is an absolute value while XON limit is an offset from the XOFF limit.When the shared occupancy or receive buffer of an ingress PG reaches its configured PFC XOFF limit, the PG triggers the transmission of a PFC pause frame to its egress peer. This action signals the peer to stop sending packets, providing the device with the necessary time to alleviate traffic congestion. The primary purpose is to ensure that, during congestion at an egress port, the ingress port generates PFC frames based on these user-defined thresholds.

XOFF threshold default value is different for each platform, as follows:

Queue-Specific Buffer Limits with Dynamic Threshold (Alpha Value): Beyond PG configuration, Switch Packet Buffer Tuning also enables users to adjust the limit of buffers that each individual queue can consume from the buffer shared pool. This fine-grained control is managed by a dynamic threshold setting, often referred to as the alpha value. Users can configure this dynamic threshold for different queues, thereby regulating their access to the shared buffer.

The implementation of Switch Packet Buffer Tuning is intended for LTSW chipsets (Tomahawk4 (TH4) platforms, Tomahawk 5 (TH5) platforms, TD4) and DC chipsets (Tomahawk3 (TH3) platforms, TD3).

Configuring XOFF threshold imposes static limit of PG queue size which take precedence over dynamic threshold. Hence, users can configure either XOFF threshold or dynamic threshold at a time.

Performing add, delete, or update operations on PG profiles will cause a system-wide traffic interruption, resulting in configured session flaps.

QoS buffer configuration using flow-control buffer commands are only applicable to ports or priorities only when flow control is enabled. The flow control buffer configurations are optional. If the configurations are not done, defaults are used.

Enables user to:

Customize device responses to PFC pause storms.

Avoid packet loss by pausing the traffic transmission using the PFC.

Globally adjust the limits of the buffers, leading to an ease of configuration.

To adjust the PG queue size using dynamic threshold, standard traffic forwarding is sufficient. However, for modifying XON or XOFF thresholds, the interface must have PFC enabled.

This section explain how to configure ingress QoS profile and egress policy mapping.

The following topology shows PFC threshold values on priority group interfaces xe1 of Rotuer 1.

The following procedure shows tje QoS profile mapping to ingress traffic to Priority Groups (PGs) and sets the XON/XOFF thresholds for the desired lossless priority.

Verify the ingress (I) and egress (E) queue statistics, including queue size for interface xe33 pg-3.

Look for the configured Priority Group pg-3 (I) on xe33 and the corresponding egress queue q3 (E) on xe34. Ingress PG queue size (pg-3 on xe33) is non-zero. This shows the ingress buffer is holding traffic before the PFC is triggered.

Verify the number of PFC pause frames sent and received on xe33 - pg3 interface and priority.

The PFC pause frame increases on the ingress interface xe33 on pg-3 when congestion is detected. This indicates that the switch is signaling the sender to stop sending traffic for this priority.

Verify the XON and XOFF threshold value on interface xe33.

Verify the ingress (I) and egress (E) queue statistics, including queue size for interface xe33 pg-3.

Look for the configured Priority Group pg-3 (I) on xe33 and the corresponding egress queue q3 (E) on xe34. Ingress PG queue size (pg-3 on xe33) is non-zero. This shows the ingress buffer is holding traffic before the PFC is triggered.

Repeatedly execute the following show command during congestion. Focus on the Q-Size for the interface xe33

Verify the XON and XOFF threshold value on interface xe33.

The topology involves four VTEP routers, hosts and two spines.

This procedure outlines the steps to configure the switch packet buffer tuning settings, primarily focusing on Quality of Service (QoS) and Priority Flow Control (PFC) configurations.

Execute the following steps to configure the QoS profiles and policy maps globally on Spine and Leaf nodes.

Verify the QoS Priority Group (PG) settings, including the XON/XOFF and dynamic thresholds, for the defined profiles.

Verify that traffic is being correctly classified and processed by the queuing policies on the interfaces.

The non-zero counts for Total pkts and Total bytes for class-map q7 (Queue 7) on multiple interfaces indicate that the queuing policy is operational.

The Dropped pkts and Dropped Bytes columns show 0 for all monitored interfaces and q7. This indicates that the buffers were not congested enough to trigger packet drops.

Following are some the real time examples to create, attach, update PG profile.

For voice network traffic, consistent and predictable latency is critical. By configuring a low alpha value (e.g. 1/16 MMU alpha) on the relevant traffic classes, queues rely primarily on their reserved buffers rather than the shared pool. This ensures packets are forwarded with minimal jitter. Following is an ingress PG map profile example for low alpha value.

For workloads like backup traffic or bulk file transfers, the priority is maximizing throughput rather than minimizing latency. By setting a higher alpha value (e.g. 1/4 or 1/2 MMU alpha), queues can aggressively borrow from the shared buffer, allowing them to handle large bursts of data efficiently. Following is an ingress PG map profile example for high alpha value.

To attach a PG profile to a MLAG interface

The Switch Buffer Tuning introduces the following configuration commands.

 

Use this command to create a ingress priority group (PG) profile or update the default ingress PG profile.

Use no parameter of this command to remove a ingress PG profile.

Use no parameter of priority-group command to unset a PG mapping in the ingress PG profile.

Use qos map-profile to map the profile on interface.

Use this command to update the priority mapping in the PG profile.

 

Configure Mode

Introduced in OcNOS Version 7.0.0. This command is supported only on the Trident3 (TD3) platforms, Trident 4 (TD4) platforms, Tomahawk3 (TH3) platforms, Tomahawk4 (TH4) platforms, and Tomahawk 5 (TH5) platforms.

Command to create and attach user-defined PG profile.

Updating default global PG profile with XON-offset:

Updating default global PG profile with XOFF-threshold:

Updating default global PG profile with dynamic-threshold:

Updating default global PG profile with XON-offset and XOFF-threshold:

Updating default global PG profile with XON-offset and dynamic threshold:

Accessing default global PG profile:

Detaching a PG profile:

 

Use this command to configure egress dynamic-threshold values.

Use no parameter of this command to unconfigure .

None

config-pmap-c-que-def mode

policy-class-map-queue mode

Introduced in OcNOS Version 7.0.0. This command is supported only on the Trident3 (TD3) platforms, Trident 4 (TD4) platforms, Tomahawk3 (TH3) platforms, Tomahawk4 (TH4) platforms, and Tomahawk 5 (TH5) platforms.

Command to configure egress dynamic-threshold value:

 

The following existing CLI commands are updated to include ingress-pg-map attribute.

The CLI is modified to include ingress-pg-map attribute.

ingress-pg-map - This attribute displays the details of the PG profile attached to an interface or all.

 

This CLI is modified to include ingress-pg-map attribute.

ingress-pg-map - This attribute attach or detach an ingress priority group profile to and from an interface.

Following are some of the commands that can be used for troubleshooting:

The following provides definitions for key terms or abbreviations and their meanings used throughout this document: