Testing network / NAT performance

[Rafał Miłecki]

On 12.06.2022 21:58, Rafał Miłecki wrote:
> 6. Organizing kernel symbols
> 
>     CPUs of home routers usually have small caches. The way kernel
>     symbols get organized during compilation may significantly affect
>     network performance [3]. It's especially annoying as network
>     unrelated changes may move / reorder symbols and affect cache hits &
>     misses.
> 
>     There isn't a reliable solution for that. It may help to add:
>     -falign-functions=32 or
>     -falign-functions=64 (depending on platform).
>     using e.g. KBUILD_CFLAGS.

I'll provide an example of a really annoying behaviour I've just
debugged. I noticed a NAT speed regression when switching from kernel
5.10 to 5.15. I narrowed it down to the 5.14 → 5.15 switch and then
started bisecting process.

I debugged that following commit:
4c00e1e2e58ee Merge tag 'linux-watchdog-5.15-rc1' of git://www.linux-watchdog.org/linux-watchdog
dropped NAT speed from ~938 Mb/s down to 907 Mb/s.

*****

Here comes interesting part: regression isn't present in the commit
41e73feb10249 ("dt-bindings: watchdog: Add compatible for Mediatek
MT7986") - the last commit in the merged branch (tag).

It means that merged code affects NAT performance only on top of the
previous commit 192ad3c27a489 ("Merge tag 'for-linus' of
git://git.kernel.org/pub/scm/virt/kvm/kvm").

I kept debugging and discovered that reverting dbe80cf471f94 ("watchdog:
Start watchdog in watchdog_set_last_hw_keepalive only if appropriate")
brings back high NAT speed.

*****

Another interesting part: cherry-picking above commit on top of the
192ad3c27a489 ("Merge tag 'for-linus' of
git://git.kernel.org/pub/scm/virt/kvm/kvm") does nothing (no NAT
regression). Further debugging revealed another commit required to
trigger regression: 60bcd91aafd22 ("watchdog: introduce
watchdog_dev_suspend/resume"). Cherry-picking both on top of kvm affects
NAT performance.

*****

Finally (even more fun):

1. Cherry picking both commits on top of v5.14 does nothing (does not
    break NAT performance).

2. Reverting both commits from v5.15 doesn't fix regression.

So all that watchdog thing is just some kind of a glitch. It makes
debugging an actual regression a really painful process. It breaks
reliability of automated testing.

All of that happens with -falign-functions=32 and I'm not aware of any
workaround for such issues.

FWIW: actual regression seems to be caused by one of commits introduced
by the 626bf91a292e2 ("Merge tag 'net-5.15-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net").