Unbalanced prioritization in the images buildbot? (main branch deprioritized too much)

[Thibaut]

Hi,

> Le 13 nov. 2023 à 16:55, Hannu Nyman <hannu.nyman at iki.fi> a écrit :
> 
> Looks like the release branches might have a too strong priority in the combined image buildbot, so that release branches get always built before the development main/master.
> 
> Recently there has been a steady flow of mostly small/unessential fixes/improvements for 22.03 and 23.05, so buildbot has been focused on building 22.03-SNAPSHOT and 23.05-SNAPSHOT (and the ongoing 23.05.1 build), and the main branch builds do lag really badly.
> 
> There was a bug in netifd WLAN device / hotplug handling that Felix fixed already 3 days ago, but we are still offering faulty downloads for half of the targets in main, as buildbot is prioritising  22.03 and 23.05 due to new commits in them. Oldest (and faulty) main build downloads are currently from Wed 8th = 5 days ago. That affects the imagebuilder/auc/attendedsysupgrade users, too.
> 
> The combined queue for images looks nice in principle, but can apparently lead into discrimination of main/master, if there are steadily new commits to the dormant stable branches.

I think it’s working exactly as designed. The release branches are prioritized over main by virtue of being *release* branches.
AIUI the purpose of autobuilder on the main branch is a convenience facility mainly used for confirming that new commits build correctly(*), not (IMHO) a way to « offer » images [for end users]. Users running bleeding edge are likely to build their own images anyway.

Furthermore the turn around time for any one commit to be fully built on any one branch is about a couple days on the current infrastructure, so « 3 days » is actually a very short timeframe in that context.

> There should maybe be some sanity check that each target gets built at least after X days since the last build, even if there are newer commits in the prioritized release branches. The "time since last build" should have a growing weight.

I disagree. Having up to date release branches is more important than having up-to-date main snapshot, so that the release branches are always confirmed to be in a releasable state should the need arise.

In any case, another way to tackle this problem would be to switch from continuous builds that starve the build resources to periodic builds that don’t (e.g. once a week), in which case it becomes possible to predictably and consistently build all branches.

(*) considering the amount of CI now going on, there would be less need to run the builders continuously and the periodic proposal above could make more sense.

My 2c.

T