Enabling WiFi slows down NAT (bcm53xx)

[Rafał Miłecki]

After a lot of debugging & hacking I managed to make OpenWrt do 940 Mb/s
NAT on bcm53xx BCM47094:
http://lists.openwrt.org/pipermail/openwrt-devel/2022-June/038834.html

The problem I recently discovered is that NAT slowes down after enabling
WiFi. Without any actual traffic. I mean just enabling 2 APs:
uci set wireless.radio0.disabled=0
uci set wireless.radio1.disabled=0
(it's a BCM47094 + 2 x BCM4366).

For some reason NAT drops from 940 Mb/s to 660/715/775 Mb/s (it varies
between iperf sessions).

I don't use any offloading.


I was hoping to debug that issue using "perf". I did some recording and
used FlameGraph for folding and generating SVG graphs:
http://files.zajec.net/openwrt/bcm53xx-wifi-nat-impact/

I also generated diff graph which should be the most helpful:
http://files.zajec.net/openwrt/bcm53xx-wifi-nat-impact/wifi-off-to-on-diff.svg

Unfortunately I don't see anything obvious there.

I see v7_dma_* functions getting *less* CPU time. That is bad as those
are critical for processing network traffic buffers.

I see slightly higher bcma_host_soc_read32() usage which is a simple
readl() but apparently is slow on bcm53xx. It's being called by
bgmac_poll() however which isn't WiFi related at all.


Can anyone make anything out of this? Any idea how to avoid NAT slowing
down with WiFi on? Or at least debug what really slows it down?

-- 
Rafał