High volume of traffic on download server - move to a 2-tiers rsync setup for mirrors?

[Thibaut]

> Le 6 janv. 2026 à 15:27, Baptiste Jonglez <baptiste at bitsofnetworks.org> a écrit :
> 
> I like provocative thinking ;)

Then here’s another thought :)

> One thing that is maybe not clear: we don't manage the mirrors ourselves,
> and the mirrors are not used by downloads.openwrt.org.  All "official"
> traffic is already handled by the CDN and our single download server, with
> old releases handled by the archive server.  So, mirrors don't cost us
> much in term of time and infrastructure (except this traffic problem).
> So far, the machinery is just "the rsync server is open to all".
> 
> There are two kind of mirrors as far as I know: "far-flung institutions
> showing their support" as you say, and people running private mirrors
> (which I guess they have good reasons for, maybe they have large fleets of
> OpenWrt devices)

About that. I can give you an excellent reason to run a local mirror when you depend on packages repositories being available and « working », e.g. when frequently building lots of images using IB: when the CDN becomes out of sync (say Packages doesn’t match the actual files available for download - and it happens quite often), you’re screwed.

As for the crazy amount of traffic, as mentioned on IRC I believe it’s partly due to our (honestly broken) way of updating the packages repositories: updating the entire archive whenever a single commit that affects a single package is pushed cannot scale.

I believe your proposal simply delays the inevitable: it will push down this problem to tier-1 mirrors but eventually we will be caught back again as the number of packages and platforms grow.

I believe it would make more sense to work on having a saner build system that can (build - that would also save CPU cycles - and) update only actually modified packages. There’s been preliminary work in phase1 to address this (at least for the ‘update’ part). Then all forms of mirror traffic should naturally decrease as untouched bits remain valid in cache.

My provocative 2c.
T.