Phase2 packages buildbot fails - timeout with rsync.openwrt.org since 5 days ago

[Paul Spooren]

Hi,

>> Phase2 packages buildbot currently fails almost all buils due to timeout
>> with rsync.openwrt.org.
>> (some packages architectures are now 5 days without successfull builds.)
> 
> thanks for the report, indeed, seems like downlink is somehow broken:
> 
>  ynezz at osuosl-host-03:~$ rsync -4 --verbose --progress -a 'rsync.openwrt.org::downloads/snapshots/targets/armsr/armv8/openwrt-sdk-*.tar.*' .
>  receiving incremental file list
>  openwrt-sdk-armsr-armv8_gcc-14.3.0_musl.Linux-x86_64.tar.zst
>       1,474,560   0%  117.55kB/s    0:36:18  rsync: [receiver] write error: Broken pipe (32)
>       3,014,656   1%   96.32kB/s    0:44:02  rsync: [receiver] write error: Broken pipe (32)
>       3,276,800   1%   92.84kB/s    0:45:38  ^C
> 
> I've disabled the worker for the time being and reported the issue to OSUOSL.

I investigated a bit, thinking the server would be overloaded, but it turned out that the sponsor silently limited our bandwidth due to a (new?) traffic cap.

> Traffic quota has been reached, therefore the server is limited in its bandwidth.

We’re at 145,555 GiB after 6 days and since we’re behind a CDN, this must be a lot of rsync traffic. Curiously, the upload via rsync is still running at speeds of 20 - 50MB/s while the download (i.e. upload from buildbots) is capped at ~300KB/s. 

I wrote the sponsor a nice email asking for an unlock. To lower our daily uploads a bit; I’d suggest to turn off the SNAPSHOT mirrors, which produces some 200GB x <number of mirrors> a day.

If neither works out, we can look for an unmetered Hetzner box, some S3 sponsoring or something entirely different?

tl;dr Download server sponsor limited bandwidth due to high usage, my best guess is that the issue not OSUOSL

Paul