packet steering and bidirectional forwarding on ipq807x (was: Re: [PATCH] qualcommax: qca_edma: do not set a zero RX hash)

[John Macias]

Hello Julius,

My earlier reply to you did not reach the list, because it went out as
HTML and openwrt-devel rejected it. You and Robert received it directly.
I am repeating the result here in plain text, with one correction, and
then reporting the bidirectional loss I said I would look into. Felix,
I have added you because the second part concerns packet-steering.uc.

1. Retest of da1356d2, for the list

On main b5bc4bec on a RAX120 v1, qca_edma.c and qca_ppe were unmodified;
the only additions were the out-of-tree board support. UDP LAN->WAN at
900 Mb/s with 1200-byte datagrams lost 0.002% with 1 flow and 0.02% with
4 flows under the default steering, and TCP ran at 933-940 Mb/s. Before
da1356d2, the same test lost 18-21%.

I also set rps_cpus to CPU2 alone on all four eth0 rx queues. With NAPI
threads on CPUs 0, 1 and 3 receiving, every packet was still processed
on CPU2, so no frame carried a zero hash. The zero-hash patch is not
needed; please drop it.

Correction: I wrote that mask e on the rx queues gave 0% loss with
bidirectional traffic. That was one lucky run. Over 8 runs, the worse
direction lost between 0.1% and 40%.

2. Bidirectional UDP

The test is iperf3 -u -b 900M -l 1200 --bidir for 10 s, with one flow in
each direction, routed LAN->WAN with the default firewall and no flow
offload. I made eight runs per configuration. Run i uses client port
41000+13i, so every configuration sees the same eight flow pairs.

The hashes are fine. Tracing net:napi_gro_receive_entry shows a nonzero
hash with l4_hash=1 on every packet, a distinct hash per flow, and each
flow arriving on a single rx queue.

There are two separate effects.

a) Both directions on one ring. In about 1 pair in 4, RSS places both
   directions of the pair in the same rx ring: client port 41013 puts
   both on rx-3, and 41065 puts both on rx-1. That ring's NAPI thread
   then carries about 188 kpps alone, and one direction loses 9-41%
   (above 26% in all but one run) in every configuration below. This
   looks inherent to hashing each flow to one ring. I mention it so the
   numbers below make sense, not to ask for a change.

b) The steering rule. When a device has at least as many rx queues as
   there are CPUs, packet-steering.uc pins NAPI thread i to CPU i and
   points queue i's RPS at CPU (i+1) % ncpus. That rule came from
   acce25b789 ("improve packet steering on ipq40xx (and possibly
   others)"). On ipq807x, with 4 rx rings and 4 CPUs, it moves each
   flow's forwarding work onto a CPU that is also running another
   ring's NAPI thread.

The table gives the worse direction's loss in percent, over the 8 runs.
The first column covers the 6 pairs that use different rings, the second
covers the 2 pairs that share one ring.

  configuration                          different rings    same ring
  default (queue i RPS -> CPU i+1)       0.01-49.0 [1]      36.6-38.8
  RPS off on all queues                  0.35-1.61          37.2-38.6
  mask e on all queues                   0.14-17.5          26.5-39.5
  each queue: all CPUs but its own       0.07-20.6          9.4-40.5
  default + software flow offload        0.00-12.7          31.1-33.0
  RPS off + software flow offload        0.00-0.37          27.9-31.9
  RPS off, lan1 RPS CPU2, wan RPS CPU3   0.70-1.38 [2]      35.2-37.8

  [1] 4 of the 6 pairs lost 28.6-49.0%; the other two lost 0.01% and
      3.0%.
  [2] This behaved like RPS off. I have not worked out why setting RPS
      on the user ports did not move the work on main.

For reference, an older single-ring build of ours steers by ingress
port: RX NAPI on CPU0, TX completion on CPU1, LAN-port RPS on CPU2 and
WAN RPS on CPU3. On five of these pairs, including both pairs that share
a ring on main, it gives 0.00-0.46%. That build is a different tree, so
the comparison is only indicative.

On this hardware, the best main configuration I found for forwarding was
RPS off with software flow offload. The default next-CPU rule was the
worst.

3. Question

Is the next-CPU rule known to help on a target that already spreads
flows across one RSS ring per CPU? If not, would you accept leaving RPS
off in that branch, so each queue stays on its NAPI CPU, or a per-target
opt-out? I do not have an ipq40xx to check that such a change would
not regress the case the rule was written for. I am happy to test
patches on the RAX120 or to send the per-run data, softnet counters and
traces.

The tests and this write-up were done with AI assistance.

Thanks,
John