High volume of traffic on download server - move to a 2-tiers rsync setup for mirrors?

[Paul Spooren]

Hi,

As an immediate step I’d suggest to disable rsync mirroring of SNAPSHOTs. Secondly we can enable Shielding for Fastly again, meaning a single Fastly server downloads files once and then distributes it to other CDN nodes.

Maybe some globally distributed tier 1 mirrors should keep the SNAPSHOTs, but anyone else shouldn’t mirror that in my opinion.

We have/had some CDN inconsistencies, so some things (like our Docker builds) use mirror-03 directly instead of the CDN. Ideally someone with knowledge around Varnish aka VCL can help out.

On the side, our current mirror site is somewhat hard to use and hard to check which mirrors are up to date, I’d like to see something like Alpine offers[1].

I informed Netcup about the issue but have not received an answer yet.

[1]: https://mirrors.alpinelinux.org/

> On 6. Jan 2026, at 21:48, Rich Brown <richb.hanover at gmail.com> wrote:
> 
> Baptiste, all,
> 
> I am happy to throw rocks in the pond. :-)
> 
>> On Jan 6, 2026, at 09:27, Baptiste Jonglez <baptiste at bitsofnetworks.org> wrote:
>> 
>> That being said, I agree the current number of mirrors is kind of
>> unreasonable for this purpose.
>> 
>> The idea of the 2-tiers system is precisely to delegate mirror
>> distribution to a few trusted people/org that have the knowledge and
>> infrastructure for this.
> 
> Establishing one or two Tier-2 servers could make sense. That also sub-contracts all the traffic and coordination problems of serving Tier-3 servers to them.
> 
> And perhaps we can ask anyone who wants to become a Tier 2 server to tell us how much traffic they are saving us. They should do the same analysis that Baptiste did for our central servers, and fill out these stats (or others):
> 
> - rsync downloads from mirrors : 
> - HTTP downloads.openwrt.org : 
> - HTTP sources.cdn.openwrt.org : 
> - rsync downloads from buildbot workers : 
> - rsync sources : 
> 
> Thanks again.
> 
> Rich
> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-adm