[RFC] Refactoring OpenWrt's build infra

[Christian Marangi]

On Sun, Oct 16, 2022 at 02:07:05PM +0200, Baptiste Jonglez wrote:
> Hi,
> 
> On 05-10-22, Thibaut wrote:
> > Hi,
> > 
> > Following an earlier conversation on IRC with Petr, I’m willing to work on refactoring our buildbot setup as follows:
> > 
> > - single master for each stage (images and packages)
> > - latent workers attached to either master, thus able to build opportunistically from either master or release branches as needed / as work becomes available
> 
> This is a good idea, but I see one main downside: we would probably have
> to use the same buildbot worker image for all releases.
> 
> From what I remember, when the worker image was updated from Debian 9 to
> Debian 10, this seriously broke 19.07 builds.  Maybe Petr or Jow will
> remember the details better.
> 
> I see two ways to address this:
> 
> - either buildbot can run latent workers with a different Docker image
>   depending on the build
>

IMHO, this would be the safest and better solution to the problem. But
this means that we will have to support 2 thing instead of having one
centrilized container.

> - otherwise, we have to think early about the update strategy.  Maybe use
>   the shared buildbot instance for master branch + most recent release
>   only, and move older releases back to a dedicated buildbot instance?
> 
> > The main upside is that all buildslaves could be pooled, improving overall throughput and reducing wasted « idle time », thus lowering build times and operating costs.
> > 
> > Petr also suggested that extra release workers could be spawned at will (through e.g. cloud VMs) when a new release is to be tagged; tagged release could be scheduled only to release workers: this would still work within this « single master » build scheme.
> > 
> > NB: I’m aware of the potential performance penalty of having buildslaves randomly switching between branches, so I would try to come up with a reasonably smart solution to this issue if it doesn’t conflict with the main goals.
> 
> One thing to look for is disk space usage.  Full disks is a common cause
> of build failures.  If a single worker goes through builds for different
> branches, I would expect disk usage to be higher (e.g. more different
> versions of software in dl/).
> 

Would be ideal to have one centrilized dl/ dir where each runner can go
and take the file. We already support that in openwrt (to have a
different dl dir) and there isn't any problem with having different
release tar for the same package.

-- 
	Ansuel