[FS#2628] Kernel warning: eth0 (mtk_soc_eth): transmit queue 0 timed out (Attachment added)

Thu Dec 10 03:35:44 EST 2020

THIS IS AN AUTOMATED MESSAGE, DO NOT REPLY.

The following task has a new comment added:

FS#2628 - Kernel warning: eth0 (mtk_soc_eth): transmit queue 0 timed out
User who did this - mrakotiq (mrakotiq)

----------
Hi,
I hit this issue too with:
SoC Type: MediaTek MT7621 ver:1 eco:3
MIPS: machine is UBNT-ERX
Kernel: 4.14.209
openwrt: 19.7 d83040313390f56c22ff020f0f20da8f05e98e35

I have this version deployed on approximately 50 routers in our network. It looks like the network traffic is the trigger, but I'm not sure. Before applying this patch https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=34a96529041d4e9502c490c66f8af0154187c6d2 I can reliably reproduce this in lab environment with sending trafic out "dd if=/dev/zero bs=4k |nc -u 10.19.19.194 887" and simultaneously send many small packet with random dst port through the router from packet generator.
This conditions raliably generate message like log below. With last version from 19.7 branch I can't reproduce this in lab environment but in production environment this occurs randomly several times a day.

Dec  6 16:26:58 doubrava711 kern.warn kernel: [21424.183592] ------------[ cut here ]------------
Dec  6 16:26:58 doubrava711 kern.warn kernel: [21424.183669] WARNING: CPU: 0 PID: 29230 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x324
Dec  6 16:26:58 doubrava711 kern.info kernel: [21424.183676] NETDEV WATCHDOG: eth1 (mtk_soc_eth): transmit queue 0 timed out
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.183680] Modules linked in: pppoe ppp_async pppox ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE ebtable_nat ebtable_filter ebtable_broute xt_time xt_tcpudp xt_tcpmss xt_s
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.183907]  nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amand
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184131]  ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_se
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184256] CPU: 0 PID: 29230 Comm: bash Not tainted 4.14.195 #0
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184262] Stack : 00000000 8fe06040 80580000 800720e4 805b0000 8054b71c 00000000 00000000
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184293]         805168bc 8fc09dc4 8e594ffc 80584907 80511608 00000001 8fc09d68 53261622
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184322]         00000000 00000000 806e0000 00004280 00000000 000000dc 00000000 00000000
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184350]         00000000 80590000 000000db 80710000 80000000 805b0000 00000000 80544604
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184379]         8036289c 00000140 00000000 8fe06040 00000000 80580000 00000000 806e0000
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184408]         ...
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184417] Call Trace:
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184445] [] show_stack+0x58/0x100
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184477] [] dump_stack+0xa4/0xe0
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184498] [] __warn+0xe0/0x138
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184506] [] warn_slowpath_fmt+0x30/0x3c
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184517] [] dev_watchdog+0x1ac/0x324
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184551] [] call_timer_fn.isra.25+0x24/0x84
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184561] [] run_timer_softirq+0x1b4/0x240
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184591] [] __do_softirq+0x128/0x2ec
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184607] [] irq_exit+0xac/0xc8
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184625] [] plat_irq_dispatch+0xfc/0x138
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184634] [] except_vec_vi_end+0xb8/0xc4
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184643] [] copy_process.part.91+0x1250/0x1554
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184652] [] _do_fork+0xe0/0x2fc
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184660] [] sys_fork+0x24/0x30
Dec  6 16:26:58 doubrava711 kern.alert kernel: [21424.184682] [] syscall_common+0x34/0x58
Dec  6 16:26:58 doubrava711 kern.warn kernel: [21424.184697] ---[ end trace 8d819b188bbe37e8 ]---
Dec  6 16:26:58 doubrava711 kern.err kernel: [21424.184712] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec  6 16:26:58 doubrava711 kern.info kernel: [21424.184722] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec  6 16:26:58 doubrava711 kern.info kernel: [21424.184736] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0e950000, max=0, ctx=308, dtx=273, fdx=273, next=308
Dec  6 16:26:58 doubrava711 kern.info kernel: [21424.184748] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0d8e0000, max=0, calc=2748, drx=2749
Dec  6 16:26:59 doubrava711 kern.info kernel: [21424.586508] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3560170c, 0x10c = 0x80818
Dec  6 16:26:59 doubrava711 kern.info kernel: [21424.586527] mtk_soc_eth 1e100000.ethernet: reset pse
Dec  6 16:26:59 doubrava711 kern.info kernel: [21424.591792] mtk_soc_eth 1e100000.ethernet: PPE started
Dec  6 16:27:08 doubrava711 kern.err kernel: [21434.183552] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec  6 16:27:08 doubrava711 kern.info kernel: [21434.183579] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec  6 16:27:08 doubrava711 kern.info kernel: [21434.183594] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0ef50000, max=0, ctx=1, dtx=0, fdx=0, next=1
Dec  6 16:27:08 doubrava711 kern.info kernel: [21434.183606] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0c8d0000, max=0, calc=722, drx=723
Dec  6 16:27:09 doubrava711 kern.info kernel: [21434.586354] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x4a600a0c, 0x10c = 0x80818
Dec  6 16:27:09 doubrava711 kern.info kernel: [21434.591715] mtk_soc_eth 1e100000.ethernet: PPE started
Dec  6 16:27:18 doubrava711 kern.err kernel: [21444.183543] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec  6 16:27:18 doubrava711 kern.info kernel: [21444.183570] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec  6 16:27:18 doubrava711 kern.info kernel: [21444.183584] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0edb0000, max=0, ctx=1, dtx=0, fdx=0, next=1
Dec  6 16:27:18 doubrava711 kern.info kernel: [21444.183596] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0c860000, max=0, calc=567, drx=568
Dec  6 16:27:19 doubrava711 kern.info kernel: [21444.586349] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3f60150c, 0x10c = 0x80818
Dec  6 16:27:19 doubrava711 kern.info kernel: [21444.586368] mtk_soc_eth 1e100000.ethernet: reset pse
Dec  6 16:27:19 doubrava711 kern.info kernel: [21444.591657] mtk_soc_eth 1e100000.ethernet: PPE started
Dec  6 16:27:28 doubrava711 kern.err kernel: [21454.183606] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec  6 16:27:28 doubrava711 kern.info kernel: [21454.183633] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec  6 16:27:28 doubrava711 kern.info kernel: [21454.183647] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0c9b0000, max=0, ctx=1, dtx=0, fdx=0, next=1
Dec  6 16:27:28 doubrava711 kern.info kernel: [21454.183659] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0ee10000, max=0, calc=687, drx=688
Dec  6 16:27:29 doubrava711 kern.info kernel: [21454.586350] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x5860000c, 0x10c = 0x80818
Dec  6 16:27:29 doubrava711 kern.info kernel: [21454.591654] mtk_soc_eth 1e100000.ethernet: PPE started
...

**And this sometimes leads to kernel panic in (skbuff: skb_over_panic:) and reboot.**
Crashlog from two routers in production is attached.

If anyone has an idea how to fix it. I can check in a very short time if the error is fixed. Or extract more debug information.

----------

One or more files have been attached.

More information can be found at the following URL:
https://bugs.openwrt.org/index.php?do=details&task_id=2628#comment9138

You are receiving this message because you have requested it from the Flyspray bugtracking system.  If you did not expect this message or don't want to receive mails in future, you can change your notification settings at the URL shown above.