High volume of traffic on download server - move to a 2-tiers rsync setup for mirrors?

[Baptiste Jonglez]

On 06-01-26, Eric wrote:
> On Tuesday, January 6th, 2026 at 07:43, Thibaut <hacks at slashdirt.org> wrote:
> > About that. I can give you an excellent reason to run a local mirror when you depend on packages repositories being available and « working », e.g. when frequently building lots of images using IB: when the CDN becomes out of sync (say Packages doesn’t match the actual files available for download - and it happens quite often), you’re screwed.
> ...
> > I believe it would make more sense to work on having a saner build system that can (build - that would also save CPU cycles - and) update only actually modified packages. There’s been preliminary work in phase1 to address this (at least for the ‘update’ part). Then all forms of mirror traffic should naturally decrease as untouched bits remain valid in cache.
> 
> Big thumbs up here.  This is such a problem with ASU builds that we've
> got detection mechanisms on the server that look for out-of-sync package 
> issues and then tell users "wait a while and try again"...  I don't know
> how many dozens (hundreds?) of times I've had to explain this on the forums
> over the past couple of years as ASU gains more traction with users.

I believe this is caused by the CDN cache that introduces inconsistencies
between Packages list and actual packages.

I checked and we are actively busting the CDN cache through the Fastly API
when updating data on the download server.  This is done since at least
August 2025 (but I don't know if it works well).

Have you experienced inconsistencies since August?

If this is a different issue, do you have a link with a description of the problem in details?

Baptiste