[OpenWrt-Devel] [linux-mips] [openwrt] [sc16is7xx] [bitbang I2C] Any suggestions on board reboot caused by "sched: RT throttling activated"?

[Xuebing Wang]

Hi community,

Any suggestions, thanks.

Xuebing Wang

On 10/19/2017 01:40 PM, Xuebing Wang wrote:
> Hi community,
>
> Any suggestions on how should I proceed with my issue?
>
> Description of my issue:
> -  My board kernel prints: "sched: RT throttling activated", then 
> reboot (there is no back-trace in UART console).
> -  I have sc16is752 (I2C based UART) on my board.
> -  As Atheros AR9331 does not have dedicated I2C controller, I am 
> using bitbang I2C.
> -  This is likely related to below line of sc16is752 driver 
> (drivers//tty/serial/sc16is7xx.c)
>         ret = devm_request_threaded_irq(dev, irq, NULL, sc16is7xx_ist,
> -  './ps -eo pid,nice,policy,cmd' confirms that above is a realtime 
> kernel thread (with SCHED_FIFO scheduling policy)
> -  Occurrence rate is about once or several times a day, after I 
> stress I2C/UART traffic.
> -  Occurrence rate is much lower if I do *NOT* stress I2C/UART traffic.
>
> My setup:
> -  Atheros AR9331 chip, which is MIPS32 24kc
> -  OpenWRT tag 15.05
> -  Linux kernel v3.18.29
> -  Not preempt kernel (CONFIG_PREEMPT_NONE=y)
> -  As AR9331 does not have dedicated I2C controller, I am using 
> bitbang I2C as below:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/tree/drivers/i2c/algos/i2c-algo-bit.c?h=linux-3.18.y 
>
> -  source code of sc16is752 driver sc16is7xx.c is here:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/tree/drivers/tty/serial/sc16is7xx.c?h=linux-3.18.y 
>
>
> Question 1): There seems a fundamental question about OpenWRT mips 
> linux kernel?
> -  OpenWRT tag 15.05
> -  In OpenWRT file target/linux/ar71xx/Makefile, there is CPU_TYPE=34kc
> -  Atheros AR9331 datasheet says it is 24kc (rather than 34kc)
> -  With CPU_TYPE=34kc, it uses gcc options "-mips32r2 -mtune=34kc"
>
> Question 2): I have no idea what causes the reboot? It is a chip 
> hardware reboot, or function similar to emergency_restart() is called 
> somewhere in the kernel? Kernel console only shows "sched: RT 
> throttling activated" and reboot, there is no back-trace.
>
> If I enable preempt kernel (CONFIG_PREEMPT=y), "RT throttling 
> activated" still happens, and there is no reboot.
> -  After "RT throttling activated" happens, I use "top -d 1" to check 
> CPU usage, I observed at least once sc16is752 threaded-irq handler 
> kernel thread occupies 95% of CPU for some seconds (not sure how many 
> seconds), then sc16is752 threaded-irq handler CPU usage drops to 
> normal, and the whole OpenWRT/Linux system is normal.
> -  I did observe once that sc16is752 threaded-irq handler CPU usage 
> stays at 95% forever, it seems like a dead-lock inside sc16is752 
> threaded-irq handler.
> -  It seems a dead-lock similar (in some way?) to below patch:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-4.4.y&id=d78006d2345f87889918a8a7aa3764628ff84263 
>
>
> Question 3) Is there any explanation why board does not reboot when I 
> enable preempt kernel (CONFIG_PREEMPT=y)?
>
> Question 4) I am not sure if this is related to MIPS32r2 34kc (with 
> 9-stage pipeline multi-threaded processor core), while Atheros AR9331 
> is 24kc (with 8-stage pipeline processor core)?
> https://www.imgtec.com/mips/classic/
>
>
> I have a temporary work-around:
> -  In bitbang I2C driver, use hrtimer-backed usleep_range() (rather 
> than busywait udelay()), then this issue is gone (verified by stress 
> test).
> -  As I do not know how long it takes to arm hrtimer on AR9331 
> platform, I use usleep_range(10us, 20us), this greatly reduces bitbang 
> I2C throughput, and reduces sc16is752 (I2C based UART) baudrate.
>
> Thanks.
> Xuebing Wang
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel