Re: High volume of traffic on download server - move to a 2-tiers rsync setup for mirrors?

[Felix Baumann]

Hi Baptiste,

have you also worked on downloads.openwrt.org?
Could you please verify whether your changes modified package files for older releases by accident?
Hashes for packages from releases 18.06.x and 17.01.x don't match the packages.gz anymore leading to opkg rejecting these files since they have been tampered with.

<https://github.com/openwrt/openwrt/issues/21604>

Regards
Felix Baumann

Am 19. Januar 2026 01:13:03 MEZ schrieb Baptiste Jonglez <baptiste at bitsofnetworks.org>:
>Hi,
>
>Following all the feedback, here are recent changes regarding mirrors:
>
>- we have a new rsync service dedicated to tier-1 mirrors.  It only
>  includes active releases, because several mirror operators complained
>  that disk usage was too high (4-5 TB).  Limiting to 23.05 + 24.10 +
>  25.12 and excluding -rc minor releases, it's now only 1.5 TB.
>
>- we now generate a timestamp of last update, see [1].  Paul, you could
>  use that to build a nice table that displays mirror freshness.  Mirrors
>  are also encouraged to generate a local sync timestamp [2] and upstream
>  indication [3].
>
>- new mirror documentation here: https://openwrt.org/mirrors#how_to_mirror
>
>- we could start redirecting some traffic to tier-1 mirrors, but it's not
>  done yet.  In fact, some tier-1 mirror operators are expecting to
>  receive traffic this way to justify the cost of mirroring us.
>  Simplest way would be to do it from the CDN config; otherwise we could
>  try to setup a mirrorbits instance.  But we would need good assurance
>  that mirrors are up-to-date, it's not that easy.
>
>So far, only a single mirror is using the new rsync documentation (I am
>the admin managing it), you can see how it looks at:
>
>  https://openwrt.tetaneutral.net/releases/
>
>I'm waiting for feedback before asking all tier-1 mirror operators to
>switch to the new config.  So, is it OK if mirrors only carry active
>releases?  Does somebody need to mirror older releases?
>
>In that case, maybe archive.openwrt.org should provide rsync service?
>
>Baptiste
>
>[1] https://openwrt.tetaneutral.net/releases/
>[2] https://openwrt.tetaneutral.net/lastsync
>[3] https://openwrt.tetaneutral.net/upstream-mirror
>
>On 06-01-26, Baptiste Jonglez wrote:
>> Hi,
>> 
>> The current download server is donated by Netcup in Germany [1].  It has
>> been working well for some time, but we are now hitting traffic limits.
>> According to their conditions, after 3 TB / day, we are rate-limited to
>> 300 Mbit/s.  This has been causing issues with some buildbot workers that
>> time-out when downloading the SDK to build packages.
>> 
>> I see three solutions:
>> 
>> - ask Netcup to increase the traffic limit
>> - reduce the amount of traffic we generate
>> - find a new hosting provider for the download server
>> 
>> One of the problem is that we mix many usages on the same server, some
>> critical (main HTTP downloads, buildbot workers pulling the SDK) and some
>> less critical (mirroring to third-parties).
>> 
>> I did some stats, here is the average amount of traffic we sent *per day*
>> from the download server in December 2025:
>> 
>> - HTTP downloads.openwrt.org : 1 TB / day (mostly origin traffic for Fastly CDN)
>> - HTTP sources.cdn.openwrt.org : 100 GB / day (mostly origin traffic for Fastly CDN)
>> - rsync downloads from mirrors : 4 TB / day
>> - rsync downloads from buildbot workers : 30 GB /day
>> - rsync sources : 5 GB /day
>> 
>> As you can see, we have many mirrors pulling from the main download
>> server, with some mirrors in particular generating a lot of traffic (up to
>> 300 GB / day for some mirrors).
>> 
>> I think we should move to a 2-tier mirror architecture: have only 2-3
>> tier-1 mirrors that are allowed to pull from us, and then ask all other
>> mirrors to become tier-2 and to pull from a tier-1 mirror.  Tier 1 mirrors
>> should mirror the entire archive and should commit to a long-term
>> relationship.  I can contact mirror operators I know, and if they agree,
>> set things up this way.
>> 
>> Of course buildbot workers and the archive server would still be allowed
>> to rsync from the download server.
>> 
>> Any thoughts about this?
>> 
>> Baptiste
>> 
>> [1] https://openwrt.org/infrastructure
>
>
>
>> _______________________________________________
>> openwrt-adm mailing list
>> openwrt-adm at lists.openwrt.org
>> https://lists.openwrt.org/mailman/listinfo/openwrt-adm
>
>
>_______________________________________________
>openwrt-devel mailing list
>openwrt-devel at lists.openwrt.org
>https://lists.openwrt.org/mailman/listinfo/openwrt-devel