Easily providing test/debug images to end users? "experimental" tree?

[Hauke Mehrtens]

On 11/30/24 11:33, Tim Small wrote:
> Hi,
> 
> It seems that it's currently a bit tricky to get work-in-progress 
> changes or debug builds tested by end-users.
> 
> Some sort of end-user test is often needed where the issues are 
> occurring sporadically, or where the developer lacks hardware or some 
> other necessary piece of the puzzle to reproduce the issue themselves 
> (e.g. client devices, radio environment, network traffic, or other 
> connected equipment etc. etc.).
> 
> It is of course possible to just publish a git fork, but this relies on 
> the testers being sufficiently savvy (and with the time) to set up a dev 
> environment.
> 
> Alternatively the developer could build images on-demand for particular 
> testers, but this increases load on the developers, who have to host the 
> builds in question somewhere for download and potentially rebuild 
> several times to keep the images fresh.
> 
> End-users have to be happy to install an image from $random_website 
> (instead of "official" OpenWrt builds). Even if they are, this isn't 
> really setting a great example to end-users when OpenWrt developers have 
> to tell them to rust and download a build from whatever site.
> 
> A little analogous to how there is a "linux-next" github repository, I 
> wonder if there would be merit in an OpenWrt EXPERIMENTAL tree?  Rather 
> than being targetted only to developers, this tree could be used as 
> source to build installable images for end-users to download and try 
> when debugging particular issues.
> 
> It could be used for patches which haven't been sufficiently tested to 
> justify merging to main, and/or for debug builds, or code changes which 
> deliberately produce debug output.
> 
> I think the existence of such an experimental tree would also reduce the 
> number of regressions and associate reverted changes in main.
> 
> Potentially a "stable-experimental" tree could also be useful for 
> allowing users to test such patches against the current stable tree too 
> (for devices in production and to allow them to easily install their own 
> choice of userspace packages etc.).
> 
> Images could potentially be build on-demand (for particular targets) 
> rather than continuously for all targets to reduce resource consumption.
> 
> As a practical example, I have a debug patch for Ethernet cable test TDR 
> raw data reporting (on Marvell PHYs) which I'd like to get users to run 
> to derive some timings which are needed to produce a finalised patch to 
> the mainline kernel and OpenWrt main.
> 
> The change itself is pretty trivial, but the issue is hardware specific 
> and I want to make sure that I don't cause any regressions (either 
> slowing down or breaking the functionality), but I only have access to 1 
> of the 10 or so relevant pieces of hardware.
> 
> Forum post here:
> 
> https://forum.openwrt.org/t/ethernet-cable-test-with-raw-tdr-data- 
> reporting-bug-fix-testers-needed/216724
> 
> Cheers,
> 
> Tim.
> 

Hi Tim,

I think a separate experimental branch would cause a lot of maintenance 
efforts. If we just merge all PRs into it, it will probably not build 
most of the time because some PR broke something. When something gets 
merged we have to rebase again which is also some effort. Linux-next 
does not contain all random patches posted on the mailing lists, but 
only the trees of the subsystem maintainers which will merge into Linus 
tree later. It will not contain patches someone posted on the mailing 
list so that someone else can test them.

We are already using the github CI to build test all PRs. We can also 
upload the resulting images somewhere. github already supports to upload 
the result of a github action run. I do not know how much storage space 
we have there.

If we run into a build failure we are already uploading the log files, 
see: 
https://github.com/openwrt/actions-shared-workflows/blob/main/.github/workflows/reusable_build.yml#L680

I do not know who has access to these files. Maybe you have to login to 
gitbub to get access.


Hauke