[RFC] Refactoring OpenWrt's build infra

[Baptiste Jonglez]

Hi,

On 05-10-22, Thibaut wrote:
> Hi,
> 
> Following an earlier conversation on IRC with Petr, I’m willing to work on refactoring our buildbot setup as follows:
> 
> - single master for each stage (images and packages)
> - latent workers attached to either master, thus able to build opportunistically from either master or release branches as needed / as work becomes available

This is a good idea, but I see one main downside: we would probably have
to use the same buildbot worker image for all releases.

From what I remember, when the worker image was updated from Debian 9 to
Debian 10, this seriously broke 19.07 builds.  Maybe Petr or Jow will
remember the details better.

I see two ways to address this:

- either buildbot can run latent workers with a different Docker image
  depending on the build

- otherwise, we have to think early about the update strategy.  Maybe use
  the shared buildbot instance for master branch + most recent release
  only, and move older releases back to a dedicated buildbot instance?

> The main upside is that all buildslaves could be pooled, improving overall throughput and reducing wasted « idle time », thus lowering build times and operating costs.
> 
> Petr also suggested that extra release workers could be spawned at will (through e.g. cloud VMs) when a new release is to be tagged; tagged release could be scheduled only to release workers: this would still work within this « single master » build scheme.
> 
> NB: I’m aware of the potential performance penalty of having buildslaves randomly switching between branches, so I would try to come up with a reasonably smart solution to this issue if it doesn’t conflict with the main goals.

One thing to look for is disk space usage.  Full disks is a common cause
of build failures.  If a single worker goes through builds for different
branches, I would expect disk usage to be higher (e.g. more different
versions of software in dl/).

Thanks,
Baptiste
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openwrt.org/pipermail/openwrt-devel/attachments/20221016/f7948612/attachment.sig>