ath10k_ct firmware crashes

Tue Jan 18 04:29:14 PST 2022

Hi Ben,

After stress testing some wave 1 2x2 radios on multiple boards using 
openwrt master I noticed this crash on 2 of them.
Any idea?

01:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac 
Wireless Network Adapter

Thanks,

Koen

[   16.441327] ath10k 5.15 driver, optimized for CT firmware, probing 
pci device: 0x3c.
[   16.468523] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode 1 
irq_mode 0 reset_mode 0
[   17.719409] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c 
chip_id 0x043222ff sub 19b6:d042
[   17.728819] ath10k_pci 0000:01:00.0: kconfig debug 1 debugfs 1 
tracing 0 dfs 1 testmode 0
[   17.740820] ath10k_pci 0000:01:00.0: firmware ver 
10.1-ct-8x-__fH-022-ecad3248 api 2 features 
wmi-10.x,mfp,txstatus-noack,wmi-10.x-CT,ratemask-CT,txrate-CT,get-temp-CT,tx-rc-CT,cust-stats-CT,retry-gt2-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT 
crc32 1b2a161c
[   18.063728] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A 
crc32 bebc7c08
[   18.993270] ath10k_pci 0000:01:00.0: 10.1 wmi init: vdevs: 16 peers: 
127  tid: 256
[   19.011087] ath10k_pci 0000:01:00.0: wmi print 'P 128 V 8 T 410'
[   19.017226] ath10k_pci 0000:01:00.0: wmi print 'msdu-desc: 1424  
sw-crypt: 0 ct-sta: 0'
[   19.025374] ath10k_pci 0000:01:00.0: wmi print 'alloc rem: 25000 
iram: 38944'
[   19.082086] ath10k_pci 0000:01:00.0: htt-ver 2.2 wmi-op 2 htt-op 2 
cal file max-sta 128 raw 0 hwcrypto 1
[   19.097374] ath10k_pci 0000:01:00.0: NOTE:  Firmware DBGLOG output 
disabled in debug_mask: 0x10000000
[   19.228812] ath: EEPROM regdomain sanitized
[   19.228825] ath: EEPROM regdomain: 0x64
[   19.228830] ath: EEPROM indicates we should expect a direct regpair map
[   19.228849] ath: Country alpha2 being used: 00
[   19.228854] ath: Regpair used: 0x64
[   19.282229] usbcore: registered new interface driver option
[   19.288009] usbserial: USB Serial support registered for GSM modem 
(1-port)
[   19.318937] usbcore: registered new interface driver qcserial
[   19.324852] usbserial: USB Serial support registered for Qualcomm USB 
modem
[   19.342601] qcserial 2-1:1.0: Qualcomm USB modem converter detected
[   19.349298] usb 2-1: Qualcomm USB modem converter now attached to ttyUSB0
[   19.359080] qcserial 2-1:1.2: Qualcomm USB modem converter detected
[   19.365777] usb 2-1: Qualcomm USB modem converter now attached to ttyUSB1
[   19.389826] qcserial 2-1:1.3: Qualcomm USB modem converter detected
[   19.396484] usb 2-1: Qualcomm USB modem converter now attached to ttyUSB2
[   19.429616] kmodloader: done loading kernel modules from /etc/modules.d/*
[   26.341428] eth0: link up (1000Mbps/Full duplex)
[   26.346711] br-wan: port 1(eth0) entered blocking state
[   26.352077] br-wan: port 1(eth0) entered disabled state
[   26.357631] device eth0 entered promiscuous mode
[   26.375076] br-wan: port 1(eth0) entered blocking state
[   26.380431] br-wan: port 1(eth0) entered forwarding state
[   32.696768] ag71xx: max retries for SGMII fixup exceeded
[   32.702172] eth1: link up (1000Mbps/Full duplex)
[   32.727999] br-wan: port 2(eth1) entered blocking state
[   32.733310] br-wan: port 2(eth1) entered disabled state
[   32.738882] device eth1 entered promiscuous mode
[   32.808555] br-wan: port 2(eth1) entered blocking state
[   32.813867] br-wan: port 2(eth1) entered forwarding state
[   37.660591] EXT4-fs (sda1): mounted filesystem with ordered data 
mode. Opts: (null)
[   85.293759] ath10k_pci 0000:01:00.0: 10.1 wmi init: vdevs: 16 peers: 
127  tid: 256
[   85.311655] ath10k_pci 0000:01:00.0: wmi print 'P 128 V 8 T 410'
[   85.317869] ath10k_pci 0000:01:00.0: wmi print 'msdu-desc: 1424  
sw-crypt: 0 ct-sta: 0'
[   85.326004] ath10k_pci 0000:01:00.0: wmi print 'alloc rem: 25000 
iram: 38944'
[   85.398523] ath10k_pci 0000:01:00.0: pdev param 0 not supported by 
firmware
[   85.413584] ath10k_pci 0000:01:00.0: rts threshold -1
[   85.428114] device wlan0 entered promiscuous mode
[   85.435115] device wlan0 left promiscuous mode
[ 5578.497296] ath10k_pci 0000:01:00.0: Failed to synchronize setup for 
vdev 0 restart 0: -145, will restart firmware
[ 5578.507831] ath10k_pci 0000:01:00.0: failed to start vdev 0 addr 
00:00:00:00:00:00 on freq 5320: -145
[ 5578.518005] ath10k_pci 0000:01:00.0: firmware crashed! (guid 
b696878d-15a0-434c-9d17-81052d629e31)
[ 5578.527126] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c 
chip_id 0x043222ff sub 19b6:d042
[ 5578.536571] ath10k_pci 0000:01:00.0: kconfig debug 1 debugfs 1 
tracing 0 dfs 1 testmode 0
[ 5578.548738] ath10k_pci 0000:01:00.0: firmware ver 
10.1-ct-8x-__fH-022-ecad3248 api 2 features 
wmi-10.x,mfp,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,get-temp-CT,tx-rc-CT,cust-stats-CT,retry-gt2-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT 
crc32 1b2a161c
[ 5578.578143] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A 
crc32 bebc7c08
[ 5578.585563] ath10k_pci 0000:01:00.0: htt-ver 2.2 wmi-op 2 htt-op 2 
cal file max-sta 128 raw 0 hwcrypto 1
[ 5578.595344] ath10k_pci 0000:01:00.0: firmware register dump:
[ 5578.601106] ath10k_pci 0000:01:00.0: [00]: 0x00940750 0x00400C00 
0x00980000 0x009AEFC0
[ 5578.609147] ath10k_pci 0000:01:00.0: [04]: 0x009AF02C 0x009AF148 
0x00941B6C 0x00941B20
[ 5578.617179] ath10k_pci 0000:01:00.0: [08]: 0x00941B00 0x009423A4 
0x009422DC 0x009422C4
[ 5578.625217] ath10k_pci 0000:01:00.0: [12]: 0x00941B90 0x009423F4 
0x009423D4 0x009423DC
[ 5578.633257] ath10k_pci 0000:01:00.0: [16]: 0x009423E4 0x009423EC 
0x00942888 0x00942520
[ 5578.641304] ath10k_pci 0000:01:00.0: [20]: 0x009424FC 0x0094241C 
0x00942540 0x00942498
[ 5578.649365] ath10k_pci 0000:01:00.0: [24]: 0x00942450 0x0094250C 
0x009424D0 0x00942750
[ 5578.657408] ath10k_pci 0000:01:00.0: [28]: 0x00942798 0x00942844 
0x009428C4 0x00942594
[ 5578.665438] ath10k_pci 0000:01:00.0: [32]: 0x0094258C 0x00942C28 
0x00942F4C 0x00942F7C
[ 5578.673475] ath10k_pci 0000:01:00.0: [36]: 0x00942F90 0x00942FE4 
0x00942FF8 0x00943040
[ 5578.681518] ath10k_pci 0000:01:00.0: [40]: 0x0094077C 0x00940790 
0x00943084 0x00942F00
[ 5578.689562] ath10k_pci 0000:01:00.0: [44]: 0x009AF174 0x00942A80 
0x00942D40 0x00942D64
[ 5578.697618] ath10k_pci 0000:01:00.0: [48]: 0x00942D78 0x00942D9C 
0x00957E18 0x00957E28
[ 5578.705654] ath10k_pci 0000:01:00.0: [52]: 0x00957E20 0x00940788 
0x009430EC 0x00000000
[ 5578.713694] ath10k_pci 0000:01:00.0: [56]: 0x00000000 0x00000000 
0x00000000 0x00000000
[ 5578.721732] ath10k_pci 0000:01:00.0: Copy Engine register dump:
[ 5578.727760] ath10k_pci 0000:01:00.0: [00]: 0x00057400   7   7 3   3
[ 5578.734302] ath10k_pci 0000:01:00.0: [01]: 0x00057800   9   9 107 108
[ 5578.740855] ath10k_pci 0000:01:00.0: [02]: 0x00057c00  55  55 54  55
[ 5578.747409] ath10k_pci 0000:01:00.0: [03]: 0x00058000  30  30 2  30
[ 5578.753958] ath10k_pci 0000:01:00.0: [04]: 0x00058400   3   3 43   3
[ 5578.760506] ath10k_pci 0000:01:00.0: [05]: 0x00058800  27  27 26  27
[ 5578.767047] ath10k_pci 0000:01:00.0: [06]: 0x00058c00   4   4 4   4
[ 5578.773598] ath10k_pci 0000:01:00.0: [07]: 0x00059000   0   0 0   0
[ 5578.780284] ath10k_pci 0000:01:00.0: debug log header, dbuf: 
0x411ab0  dropped: 0
[ 5578.787961] ath10k_pci 0000:01:00.0: [0] next: 0x411a98 buf: 0x40feec 
sz: 1500 len: 0 count: 0 free: 0
[ 5578.797491] ath10k_pci 0000:01:00.0: [1] next: 0x411ab0 buf: 0x40f8fc 
sz: 1500 len: 0 count: 0 free: 0
[ 5578.813085] ath10k_pci 0000:01:00.0: removing peer, cleanup-all, 
deleting: peer eb620328 vdev: 0 addr: 00:00:00:00:00:00
[ 5579.304173] irq 18: nobody cared (try booting with the "irqpoll" option)
[ 5579.310983] CPU: 0 PID: 2113 Comm: kworker/u2:2 Not tainted 5.10.90 #0
[ 5579.317689] Workqueue: ath10k_wq ath10k_core_napi_enable [ath10k_core]
[ 5579.324306] Stack : 80700000 827a3100 00011000 ffffffff ffffffff 
800b9cb0 80700000 00000017
[ 5579.332797]         827a313c 80c09ce4 80c09cb4 80137eec 00011000 
00000001 80c09c88 5cbe00d5
[ 5579.341289]         00000000 00000000 8065611c 80c09b08 ffffefff 
00000000 00000000 ffffffea
[ 5579.349781]         00000156 80c09b14 00000156 8070c4c8 00000000 
00000000 00000012 00011000
[ 5579.358273]         ffffffff ffffffff 00000100 808c371c 00000018 
803bbda4 00000000 808c0000
[ 5579.366765]         ...
[ 5579.369248] Call Trace:
[ 5579.371755] [<80066ea4>] show_stack+0x30/0x100
[ 5579.376266] [<800bf27c>] __report_bad_irq+0x58/0xf8
[ 5579.381211] [<800bf74c>] note_interrupt+0x2c8/0x318
[ 5579.386157] [<800bcb60>] handle_irq_event+0x9c/0xcc
[ 5579.391105] [<800c0128>] handle_level_irq+0xb4/0x1bc
[ 5579.396136] [<800bc1fc>] generic_handle_irq+0x3c/0x54
[ 5579.401271] [<805c29a4>] ar724x_pci_irq_handler+0x78/0x118
[ 5579.406836] [<800bc1fc>] generic_handle_irq+0x3c/0x54
[ 5579.411979] [<803579d4>] ath79_intc_irq_handler+0x9c/0xec
[ 5579.417449] [<800bc1fc>] generic_handle_irq+0x3c/0x54
[ 5579.422583] [<805c7c1c>] do_IRQ+0x1c/0x2c
[ 5579.426649] [<803578b0>] plat_irq_dispatch+0x94/0xc8
[ 5579.431683] [<800625f8>] handle_int+0x138/0x144
[ 5579.436275] [<803578b0>] plat_irq_dispatch+0x94/0xc8
[ 5579.441306] [<800bc1fc>] generic_handle_irq+0x3c/0x54

[ 5579.447936] handlers:
[ 5579.450252] [<0c284d85>] ath10k_pci_disable_and_clear_legacy_irq 
[ath10k_pci]
[ 5579.457485] Disabling IRQ #18
[ 5579.588854] ieee80211 phy0: Hardware restart was requested
[ 5580.887406] ath10k_pci 0000:01:00.0: 10.1 wmi init: vdevs: 16 peers: 
127  tid: 256
[ 5580.997346] ath10k_pci 0000:01:00.0: wmi print 'P 128 V 8 T 410'
[ 5581.003472] ath10k_pci 0000:01:00.0: wmi print 'msdu-desc: 1424  
sw-crypt: 0 ct-sta: 0'
[ 5581.011612] ath10k_pci 0000:01:00.0: wmi print 'alloc rem: 25000 
iram: 38944'
[ 5581.547400] ath10k_pci 0000:01:00.0: pdev param 0 not supported by 
firmware
[ 5581.767469] ath10k_pci 0000:01:00.0: set-coverage-class, phyclk: 88  
value: 0
[ 5581.877443] ath10k_pci 0000:01:00.0: rts threshold -1
[ 5581.987447] ath10k_pci 0000:01:00.0: device successfully recovered