[OpenWrt-Devel] ath10k-ct 4.19 and IBSS

Koen Vandeputte koen.vandeputte at ncentric.com
Tue Aug 6 05:26:07 EDT 2019


On 05.08.19 18:17, Koen Vandeputte wrote:
>
> On 05.08.19 17:47, Koen Vandeputte wrote:
>>
>> On 27.06.19 16:24, Ben Greear wrote:
>>> On 6/27/19 7:17 AM, Koen Vandeputte wrote:
>>>>
>>>> On 26.06.19 18:39, Ben Greear wrote:
>>>>> On 6/26/19 9:28 AM, Koen Vandeputte wrote:
>>>>>>
>>>>>> On 26.06.19 18:16, Ben Greear wrote:
>>>>>>> On 6/26/19 2:02 AM, Koen Vandeputte wrote:
>>>>>>>>
>>>>>>>> On 25.06.19 15:54, Ben Greear wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 06/25/2019 02:53 AM, Koen Vandeputte wrote:
>>>>>>>>>>
>>>>>>>>>> On 24.06.19 22:04, Ben Greear wrote:
>>>>>>>>>>> On 6/24/19 8:32 AM, Koen Vandeputte wrote:
>>>>>>>>>>>> Hi Ben,
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> So I'm going to give this another try ..
>>>>>>>>>>>> As the IBSS functionality is heavily advertised as a delta 
>>>>>>>>>>>> to mainline, it would be very nice to get it working also :)
>>>>>>>>>>>>
>>>>>>>>>>>> Testing the latest ath10k-ct driver and firmware seems to 
>>>>>>>>>>>> be a step back compared to roughly a month ago.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm currently seeing the firmware crashing, which was not 
>>>>>>>>>>>> the case before:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ath10k-ct + htt-fw:
>>>>>>>>>>>>
>>>>>>>>>>>> https://pastebin.com/raw/7Sy9yx6s
>>>>>>>>>>>
>>>>>>>>>>> Looks like firmware ran out of some WMI event buffers and 
>>>>>>>>>>> crashed instead of handling
>>>>>>>>>>> it more gracefully.
>>>>>>>>>>>
>>>>>>>>>>> Please try the attached (untested) firmware and see if it 
>>>>>>>>>>> behaves better.
>>>>>>>>>>>
>>>>>>>>>> Hi Ben,
>>>>>>>>>>
>>>>>>>>>> 1 step forward here.
>>>>>>>>>>
>>>>>>>>>> I'm not seeing crashes anymore using the test-firmware.
>>>>>>>>>>
>>>>>>>>>> https://pastebin.com/raw/4ZeXu7iw
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I've linked up 2 IBSS devices (wave 1, VHT80)
>>>>>>>>>>
>>>>>>>>>> OLSR traffic (UDP) works and packets here are nicely going 
>>>>>>>>>> back & forth.
>>>>>>>>>>
>>>>>>>>>> Simply pinging (ICMP) between the 2 devices does not work.
>>>>>>>>>>
>>>>>>>>>> When sending 100 pings, (64 byte large) sometimes 1 gets 
>>>>>>>>>> through .. but with a latency of > 500ms
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think if the splat and the beacon spam below could be fixed 
>>>>>>>>>> .. this would be a major step forward:
>>>>>>>>>>
>>>>>>>>>> [   30.328423] ------------[ cut here ]------------
>>>>>>>>>> [   30.333251] WARNING: CPU: 0 PID: 1578 at 
>>>>>>>>>> /mnt/ramdisk/koen/firmware/builds/generic_rb922/build_dir/target-mips_24kc_musl/linux-ar71xx_mikrotik/ath10k-ct-2019-05-08-f98b6dc4/ath10k-4.19/mac.c:6563 
>>>>>>>>>> ath10k_mac_vif_beacon_free+0xc7c/0x115c [ath10k_core]
>>>>>>>>>> [   30.355636] Modules linked in: mbt ath9k ath9k_common 
>>>>>>>>>> qcserial pppoe ppp_async option cdc_mbim ath9k_hw ath10k_pci 
>>>>>>>>>> ath10k_core ath usb_wwan sierra_net sierra rndis_host 
>>>>>>>>>> qmi_wwan pppox ppp_generic mac80211 iptable_nat 
>>>>>>>>>> iptable_mangle iptable_filter ipt_REJECT ipt_MASQUERADE 
>>>>>>>>>> ip_tables huawei_cdc_ncm ftdi_sio cfg80211 cdc_subset cdc_ncm 
>>>>>>>>>> cdc_ether xt_time xt_tcpudp xt_state xt_nat xt_multiport 
>>>>>>>>>> xt_mark xt_mac xt_limt
>>>>>>>>>> [   30.427331]  nls_utf8 nls_iso8859_1 nls_cp437 authenc 
>>>>>>>>>> ehci_platform sd_mod scsi_mod ehci_hcd gpio_button_hotplug 
>>>>>>>>>> ext4 mbcache jbd2 usbcore nls_base usb_common ptp pps_core 
>>>>>>>>>> mii aead crypto_null cryptomgr crc32c_generic crypto_hash
>>>>>>>>>> [   30.448017] CPU: 0 PID: 1578 Comm: wpa_supplicant Not 
>>>>>>>>>> tainted 4.14.129 #0
>>>>>>>>>
>>>>>>>>> Please look in your code and let me know the source around the 
>>>>>>>>> line in mac.c (6563) that is splatting.
>>>>>>>>>
>>>>>>>>> Also, you might grab the latest ath10k-ct repo, it has a tweak 
>>>>>>>>> that might fix the SWBA overrun
>>>>>>>>> messages.
>>>>>>>>>
>>>>>>>>> https://github.com/greearb/ath10k-ct
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Ben
>>>>>>>>>
>>>>>>>> Hi Ben,
>>>>>>>>
>>>>>>>> Here is the output based on the latest git HEAD of your ct 
>>>>>>>> tree, combined with the test firmware:
>>>>>>>>
>>>>>>>> https://pastebin.com/raw/kwC6c18J
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> The splat decode does not match the source code, so I'm not 
>>>>>>> which is correct.
>>>>>>>
>>>>>> OpenWrt seems to add custom patches to your source.
>>>>>>
>>>>>> Please find the complete source in subsequent mail as being build.
>>>>>
>>>>> I did look in that code, and that is where I saw the mismatch. 
>>>>> Please check your own local system
>>>>> and see if the splat matches your code?  Maybe I made some mistake 
>>>>> of course...
>>>>>
>>>>> You can paste ~20 lines of code around the proper splat line and 
>>>>> then I can find it in my
>>>>> source...
>>>>>
>>>>> Thanks,
>>>>> Ben
>>>>>
>>>>>
>>>> Hi Ben,
>>>>
>>>> Just retried again on a slightly older commit (2019-05-08) and the 
>>>> splat points to another location now.
>>>> When looking it up, it again points to the WARN_ON pointed below ..
>>>>
>>>> Checking shows that all calls to ath10k_mac_vif_beacon_free() calls 
>>>> are way above this line (highest line nr is around line1970)
>>>> I currently can't explain where the mismatch comes from ..
>>>>
>>>> Current build below is just the git HEAD of openwrt 19.07 branch, 
>>>> cloned, build and flashed without any modification.
>>>>
>>>>
>>>> [   31.956774] WARNING: CPU: 0 PID: 1725 at 
>>>> /mnt/ramdisk/koen/firmware/builds/generic_rb922/build_dir/target-mips_24kc_musl/linux-ar71xx_mikrotik/ath10k-ct-2019-05-08-f98b6dc4/ath10k-4.19/mac.c:6563 
>>>> ath10k_mac_vif_beacon_free+0xc7c/0x115c [ath10k_core]
>>>>
>>>>
>>>>
>>>>                  ret = ath10k_config_ps(ar);
>>>>                  if (ret)
>>>>                          ath10k_warn(ar, "failed to setup ps on 
>>>> vdev %i: %d\n",
>>>>                                      arvif->vdev_id, ret);
>>>>          }
>>>>
>>>>          if (changed & BSS_CHANGED_MCAST_RATE &&
>>>> --->          !WARN_ON(ath10k_mac_vif_chan(arvif->vif, &def))) {
>>>>                  band = def.chan->band;
>>>
>>> I think this might not be to serious of a bug, and probably does not 
>>> cause any real
>>> trouble.
>>>
>>> It is also probably a bug in mac80211 or similar, but not certain 
>>> about that.
>>>
>>> The general set of bugs related to IBSS seem to be inability to 
>>> transmit frames
>>> sometimes (though it usually works well in my lab, so I have not 
>>> been able to really
>>> debug it).
>>>
>>> The simpler the test case, the better.  So, if you can reproduce 
>>> some packet transmit
>>> issue, preferably:
>>>
>>> 2 peers
>>> slow speed traffic
>>> no encryption
>>> no funny routing (OLSR, batman, etc)
>>>
>>> Please let me know.
>>>
>>> And, if you cannot reproduce in this simple setup, then what
>>> is the thing closest to this that does show the issue?  I can build 
>>> firmware with
>>> verbose tx-path (and rx-path, for that matter) debugging and let you 
>>> run it if you can
>>> find a good test case, but it cannot gather useful logs at high 
>>> packet transmission rates.
>>>
>>> Thanks,
>>> Ben
>>>
>
>> Hi Ben,
>>
>> I finally managed to get to some time to properly take a look using a 
>> simple setup.
>>
>> Attached all required files to simulate the issue.
>>
>> I compiled the latest OpenWrt master state, (included a full 
>> wpa_supplicant and iperf tools) and ran the 2 starts.
>>
>> Attached also logs as seen from both boards simultaneously.
>>
>>
>> basically:
>>
>> - If the boards finally do link after lots of tries, it will have a 
>> >200ms latency and max speed of about 3Mbit.
>>
>> - The wpa_sup config file is the most basic RSN enabled config.
>>
>> - I also tried the current Master state with/without all custom 
>> pathes, but the result is the same.
>>
>> - wpa_supp also nags about some missing IE's
>>
>>
>> Hw used:
>>
>> - 2x RB-922UAGS containing a on-board ar988x, capable of 30dBm.
>>
>> - 2x standard 5GHz omni antennae
>>
>> - board seperation distance ca 6ft
>>
>>
>>
>> Kind regards,
>>
>> Koen
>>
> (Re-Send due to large size)


Hi Ben,

Enabling debug shed some light:

ath10k-4.19/wmi.c:


         spin_lock_bh(&ar->data_lock);

         bcn = arvif->beacon;

         if (!bcn)
                 goto unlock;

         cb = ATH10K_SKB_CB(bcn);

         switch (arvif->beacon_state) {
         case ATH10K_BEACON_SENDING:
         case ATH10K_BEACON_SENT:
                 break;
         case ATH10K_BEACON_SCHEDULED:
                 arvif->beacon_state = ATH10K_BEACON_SENDING;
                 spin_unlock_bh(&ar->data_lock);

                 dtim_zero = !!(cb->flags & ATH10K_SKB_F_DTIM_ZERO);
                 deliver_cab = !!(cb->flags & ATH10K_SKB_F_DELIVER_CAB);
                 ret = ath10k_wmi_beacon_send_ref_nowait(arvif,
bcn->data, bcn->len,
cb->paddr,
                                                         dtim_zero,
deliver_cab);
                 ath10k_dbg(ar, ATH10K_DBG_BEACON,
                            "wmi event beacon send, vdev-id: %u rv: %d\n",
                            arvif->vdev_id, ret);

                 spin_lock_bh(&ar->data_lock);

                 if (ret == 0)
                         arvif->beacon_state = ATH10K_BEACON_SENT;
                 else
                         arvif->beacon_state = ATH10K_BEACON_SCHEDULED;
         }

unlock:
         spin_unlock_bh(&ar->data_lock);
}

-----------

dmesg:


[ 1758.739939] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: 0
[ 1758.750522] ath10k_pci 0000:01:00.0: wmi event beacon-tx-complete, 
vdev-id: 0  completion-status: 0x0 (OK) tried: 1 failed: 0 ratecode: 0x3 
rateflags: 0x0  tsFlags: 0x0
[ 1758.772647] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 1758.842348] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: 0
[ 1758.853062] ath10k_pci 0000:01:00.0: wmi event beacon-tx-complete, 
vdev-id: 0  completion-status: 0x0 (OK) tried: 1 failed: 0 ratecode: 0x3 
rateflags: 0x0  tsFlags: 0x0
[ 1758.944759] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: 0
[ 1758.955501] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: 0
[ 1758.955526] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: 0
[ 1758.955546] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: -11
[ 1758.955559] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped 
old beacon
[ 1758.963032] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: -11
[ 1758.963048] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped 
old beacon
[ 1758.970468] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: -11
[ 1758.970481] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped 
old beacon
[ 1758.977918] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: -11
[ 1758.977933] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped 
old beacon
[ 1758.985370] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: -11
[ 1758.987837] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: 0
[ 1758.987971] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: -11
[ 1758.987986] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped 
old beacon
[ 1758.995459] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: -11
[ 1758.995475] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped 
old beacon
[ 1759.002910] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: -11
[ 1759.002926] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped 
old beacon
[ 1759.010342] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: -11
[ 1759.010360] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: 0
[ 1759.010378] ath10k_pci 0000:01:00.0: wmi event beacon send, vdev-id: 
0  rv: -11
[ 1759.010391] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped 
old beacon


_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel


More information about the openwrt-devel mailing list