Re: Easily providing test/debug images to end users? "experimental" tree?

[Felix Baumann]

Hi,

could it be interesting to devise a system where users can enter their contact data for devices they own, showing that they are willing to test these devices for devs?
Devs could send out messages through the system asking for certain PR/patches to be tested by devices to confirm they don't break anything. Devs after all don't have access to all kinds of devices out there.
This could speed up development by having immediate feedback. Also it would mean devs could get to know testers better and don't have to explain everything right from the start because they know what certain testers are capable of (if they worked with them in the past)

Since such data goes stale quickly, the system should be asking users each year during release candidate phase if they are still up for testing and to update the list of devices they still have available for testing.

The OpenWrt project itself could use the system to contact users to actively test new kernels on devices they own before they are set to default for a target in snapshots. Currently this is tedious (I think) since new kernels aren't active in snapshots and users are usually asked to build their own images or look for the right PR with built images.

Users willing to offer their devices for testing can't always keep up with what is going on in development or on github. it's a lot

Regards
Felix Baumann

PS: Tim your link is broken due to a space in it. Here it is again so everyone can click the link:
https://forum.openwrt.org/t/ethernet-cable-test-with-raw-tdr-data-reporting-bug-fix-testers-needed/216724

Am 8. Dezember 2024 15:31:42 MEZ schrieb Hauke Mehrtens <hauke at hauke-m.de>:
>On 11/30/24 11:33, Tim Small wrote:
>> Hi,
>> 
>> It seems that it's currently a bit tricky to get work-in-progress changes or debug builds tested by end-users.
>> 
>> Some sort of end-user test is often needed where the issues are occurring sporadically, or where the developer lacks hardware or some other necessary piece of the puzzle to reproduce the issue themselves (e.g. client devices, radio environment, network traffic, or other connected equipment etc. etc.).
>> 
>> It is of course possible to just publish a git fork, but this relies on the testers being sufficiently savvy (and with the time) to set up a dev environment.
>> 
>> Alternatively the developer could build images on-demand for particular testers, but this increases load on the developers, who have to host the builds in question somewhere for download and potentially rebuild several times to keep the images fresh.
>> 
>> End-users have to be happy to install an image from $random_website (instead of "official" OpenWrt builds). Even if they are, this isn't really setting a great example to end-users when OpenWrt developers have to tell them to rust and download a build from whatever site.
>> 
>> A little analogous to how there is a "linux-next" github repository, I wonder if there would be merit in an OpenWrt EXPERIMENTAL tree?  Rather than being targetted only to developers, this tree could be used as source to build installable images for end-users to download and try when debugging particular issues.
>> 
>> It could be used for patches which haven't been sufficiently tested to justify merging to main, and/or for debug builds, or code changes which deliberately produce debug output.
>> 
>> I think the existence of such an experimental tree would also reduce the number of regressions and associate reverted changes in main.
>> 
>> Potentially a "stable-experimental" tree could also be useful for allowing users to test such patches against the current stable tree too (for devices in production and to allow them to easily install their own choice of userspace packages etc.).
>> 
>> Images could potentially be build on-demand (for particular targets) rather than continuously for all targets to reduce resource consumption.
>> 
>> As a practical example, I have a debug patch for Ethernet cable test TDR raw data reporting (on Marvell PHYs) which I'd like to get users to run to derive some timings which are needed to produce a finalised patch to the mainline kernel and OpenWrt main.
>> 
>> The change itself is pretty trivial, but the issue is hardware specific and I want to make sure that I don't cause any regressions (either slowing down or breaking the functionality), but I only have access to 1 of the 10 or so relevant pieces of hardware.
>> 
>> Forum post here:
>> 
>> https://forum.openwrt.org/t/ethernet-cable-test-with-raw-tdr-data- reporting-bug-fix-testers-needed/216724
>> 
>> Cheers,
>> 
>> Tim.
>> 
>
>Hi Tim,
>
>I think a separate experimental branch would cause a lot of maintenance efforts. If we just merge all PRs into it, it will probably not build most of the time because some PR broke something. When something gets merged we have to rebase again which is also some effort. Linux-next does not contain all random patches posted on the mailing lists, but only the trees of the subsystem maintainers which will merge into Linus tree later. It will not contain patches someone posted on the mailing list so that someone else can test them.
>
>We are already using the github CI to build test all PRs. We can also upload the resulting images somewhere. github already supports to upload the result of a github action run. I do not know how much storage space we have there.
>
>If we run into a build failure we are already uploading the log files, see: https://github.com/openwrt/actions-shared-workflows/blob/main/.github/workflows/reusable_build.yml#L680
>
>I do not know who has access to these files. Maybe you have to login to gitbub to get access.
>
>
>Hauke
>
>_______________________________________________
>openwrt-devel mailing list
>openwrt-devel at lists.openwrt.org
>https://lists.openwrt.org/mailman/listinfo/openwrt-devel