High volume of traffic on download server - move to a 2-tiers rsync setup for mirrors?

[Baptiste Jonglez]

On 06-01-26, Michael Richardson wrote:
> Baptiste Jonglez <baptiste at bitsofnetworks.org> wrote:
>     rich> Proposal: We should eliminate all mirrors.
> 
> If the mirrors are ad-hoc, and are not listed as alternate A/AAAA RR, then
> we might want to artificially rate-limit the rsync support.

Rate-limit can be complex to setup (it's not supported natively by rsyncd).
It's simpler to just allow a fixed list of mirrors to already have good
control on mirror traffic.

> Are there some pathologies here?  Peers that repeated pull the same
> content?

I don't think so.  Several mirrors are pulling around 6 TB / month, which
makes sense because snapshots are regenerated constantly.  Other mirrors
pull less, probably because they don't sync snapshots.

Here are the anonymized top rsync downloaders for the whole of December
2025, with estimated country from whois based on the IP address:

# Rsync details for December 2025
('2025-12', 'downloads') total 106241 GiB
-> (IPv4) Italy 6197 GiB
-> (IPv6) Netherlands 6177 GiB
-> (IPv4) Italy 6151 GiB
-> (IPv6) France 6047 GiB
-> (IPv6) Germany 5898 GiB
-> (IPv6) Morocco 5668 GiB
-> (IPv6) China 5651 GiB
-> (IPv6) China 5642 GiB
-> (IPv6) China 5610 GiB
-> (IPv4) Singapore 3683 GiB
-> (IPv4) Germany 3590 GiB
-> (IPv6) China 3472 GiB
-> (IPv6) China 3293 GiB
-> (IPv6) Germany 3256 GiB
-> (IPv6) Japan 2785 GiB
-> (IPv6) USA 2716 GiB
-> (IPv6) Brazil 2697 GiB
-> (IPv6) Kazakhstan 2601 GiB
-> (IPv6) Germany 2447 GiB
-> (IPv6) Belgium 2308 GiB
-> (IPv4) China 1867 GiB
-> (IPv4) Canada 1567 GiB
-> (IPv6) China 1526 GiB
-> (IPv6) China 1438 GiB
-> (IPv6) China 1433 GiB

>     > I think public mirrors can still be useful, it leaves alternative choices
>     > if we have a big outage of the download server and the archive server
>     > (unlikely, but still possible).  Another reason is better performance: the
>     > CDN helps but still needs to fetch content from the download server in
>     > Germany.
> 
> ...
> 
>     > That being said, I agree the current number of mirrors is kind of
>     > unreasonable for this purpose.
> 
> split out the uses onto different names/IPs (v6 makes this easy).
> I assume all the CDNs can use v6.

You mean we could point the HTTP CDN to some mirrors instead of the main
download server?

That's not a bad idea, especially if we formalize some quality assurance
with tier-1 mirror operators (good uptime, monitoring, enough disk space
and bandwith, frequent pull from main download server, etc)

I would still only do this in case of problems on the main download
server, but it's clearly something we need to anticipate so that we only
have to "flip a switch" when needed.

Baptiste