Network down on HEAD at reboot

[Philip Prindeville]

Hi,

I updated my APU6’s to HEAD about a week ago, and I’m seeing some very odd behavior.  Like catastrophically bad behavior.

If I reboot, then the interfaces are initialized, addresses, subnet masks, and routes installed.  And everything would seem to be normal… except.

Clients can get DHCP leases, and dumping their ARP table shows the default gateway.  But the default gateway won’t reply to pings to its address.

I see clients sending SYN packets trying to reach various servers on-net and off-, but none of the connections completes.

For the pings, I see them hitting the gateway just fine with tcpdump…. but it doesn’t respond.

If on a client, I delete the ARP entry for the router, it immediately reappears… so the ARP functionality in the router seems to be working fine.

But ping and ssh just timeout.

Again, tcpdump shows me the packets arriving but no outgoing response.

‘lsof -n -P -p $(pidof sshd)’ shows me listening on *:22 so it shouldn’t care what interfaces the packets come in on.

If I get onto the CLI on the serial console and restart the firewall, nothing seemingly changes: no ping and no ssh.

If I get onto the console and re-run ‘/etc/init.d/network restart’, then I can ping the router!

I see the following appear on the console from syslog spew:


Dec 12 17:23:04 OpenWrt2 kernel: [259728.378694] 8021q: adding VLAN 0 to HW filter on device eth1
Dec 12 17:23:04 OpenWrt2 kernel: [259728.457341] 8021q: adding VLAN 0 to HW filter on device eth3
Dec 12 17:23:07 OpenWrt2 kernel: [259731.116972] igb 0000:04:00.0 eth3: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Dec 12 17:23:07 OpenWrt2 kernel: [259731.354957] igb 0000:02:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Dec 12 17:23:07 OpenWrt2 kernel: [259731.471854] ntpd[19747]: segfault at 24 ip 00005621aaba8ef1 sp 00007ffe4704bc50 error 4 in ntpd[3fef1,5621aab70000+4e000] likely on CPU 2 (core 2, socket 0)
Dec 12 17:23:07 OpenWrt2 kernel: [259731.471921] Code: 89 c4 67 e8 20 ba ff ff 48 89 c3 48 85 ed 75 1a ba 80 00 00 00 48 8d 35 0a a5 01 00 48 89 c7 ff 15 a4 06 04 00 e9 a7 00 00 00 <0f> b7 4d 00 66 83 f9 02 74 0b 66 83 f9 0a 74 1e 66 85 c9 75 7b 48


which makes me think that the kernel I’m running somehow broke untagged frames in VLAN or possibly in the igb driver.  If there’s an ‘iproute2’ command that will show before/after state with VLAN tables, please share it and I’ll post results.

root at OpenWrt2:~# cat /etc/os-release 
NAME="OpenWrt"
VERSION="SNAPSHOT"
ID="openwrt"
ID_LIKE="lede openwrt"
PRETTY_NAME="OpenWrt SNAPSHOT"
VERSION_ID="snapshot"
HOME_URL="https://openwrt.org/"
BUG_URL="https://bugs.openwrt.org/"
SUPPORT_URL="https://forum.openwrt.org/"
FIRMWARE_URL="https://downloads.openwrt.org/"
BUILD_ID="r32233-a638a72a79"
OPENWRT_BOARD="x86/64"
OPENWRT_ARCH="x86_64"
OPENWRT_TAINTS="no-all busybox"
OPENWRT_DEVICE_MANUFACTURER="Redfish Solutions, LLC"
OPENWRT_DEVICE_MANUFACTURER_URL="http://www.redfish-solutions.com/"
OPENWRT_DEVICE_PRODUCT="Generic"
OPENWRT_DEVICE_REVISION="v0"
OPENWRT_RELEASE="OpenWrt SNAPSHOT r32233-a638a72a79"
OPENWRT_BUILD_DATE="1765045444"
root at OpenWrt2:~# uname -r
6.12.60
root at OpenWrt2:~# 


What am I missing?  I used to think that any problem could be solved with ‘arp’, ‘ping’, and ‘tcpdump’ but now I’m not so sure.

Bonus points if you can figure this one out.

Thanks,

-Philip