by lunarg on October 3rd 2022, at 09:15

Dell OS10 (used on PowerSwitch S-series) supports RDMA over Converged Ethernet (RoCE). The SmartFabric manual has some good examples on how to configure it but here's the summary.

A few remarks first: you can choose which queue to place traffic in. The manual states Q3 which is the required queue if you also use iSCSI (which is by default placed in Q4), but if you don't use iSCSI, you can opt to place the traffic in Q4 instead of Q3.

Also, when entering the commands, be sure to check whether you're in the proper context in CONFIG mode. I did not add the necessary "exit" commands so don't blindly copy/paste this.

Global configuration (per switch)

First, enable DCB(X), which is required for RoCE (PFC and ETS):

dcbx enable

Create the inbound QoS class and policy:

class-map type network-qos pfcdot1p4
match qos-group 4
policy-map type network-qos policy_pfcdot1p4
class pfcdot1p4
pfc-cos 4

Create the necessary queues for outbound traffic:

class-map type queuing Q0
match queue 0
class-map type queuing Q4
match queue 4

If you are using RoCEv2, you can also enable ECN:

wred wred_ecn
random-detect ecn
random-detect color green minimum-threshold 1000 maximum-threshold 2000 drop-probability 100
random-detect color yellow minimum-threshold 500 maximum-threshold 1000 drop-probability 100
random-detect color red minimum-threshold 100 maximum-threshold 500 drop-probability 100

Create the policy for outbound traffic:

policy-map type queuing policy_2Q
class Q0 
bandwidth percent 40
class Q4
bandwidth percent 60
random-detect wred_ecn

In the code above, leave out the last line if you don't want to use ECN.

Finally, create the QoS map:

qos-map traffic-class 2Q 
queue 0 qos-group 0-3,5-7 
queue 4 qos-group 4

Port configuration

Note that certain items (such as enabling ETS) can also be done globally.

For each interface you wish to enable RoCE on, add these lines:

interface ethernet 1/1/x
priority-flow-control mode on
service-policy input type network-qos policy_pfcdot1p4
service-policy output type queuing policy_2Q
ets mode on
qos-map traffic-class 2Q
trust-map dot1p default

Note that if you want to use RDMA across switches, you need to configure the policies on the uplink ports as well. For LAGs, this means the member ports and not the LAG (port-channel) itself.