High volume of traffic on download server - move to a 2-tiers rsync setup for mirrors?

[Baptiste Jonglez]

Hi,

Following all the feedback, here are recent changes regarding mirrors:

- we have a new rsync service dedicated to tier-1 mirrors.  It only
  includes active releases, because several mirror operators complained
  that disk usage was too high (4-5 TB).  Limiting to 23.05 + 24.10 +
  25.12 and excluding -rc minor releases, it's now only 1.5 TB.

- we now generate a timestamp of last update, see [1].  Paul, you could
  use that to build a nice table that displays mirror freshness.  Mirrors
  are also encouraged to generate a local sync timestamp [2] and upstream
  indication [3].

- new mirror documentation here: https://openwrt.org/mirrors#how_to_mirror

- we could start redirecting some traffic to tier-1 mirrors, but it's not
  done yet.  In fact, some tier-1 mirror operators are expecting to
  receive traffic this way to justify the cost of mirroring us.
  Simplest way would be to do it from the CDN config; otherwise we could
  try to setup a mirrorbits instance.  But we would need good assurance
  that mirrors are up-to-date, it's not that easy.

So far, only a single mirror is using the new rsync documentation (I am
the admin managing it), you can see how it looks at:

  https://openwrt.tetaneutral.net/releases/

I'm waiting for feedback before asking all tier-1 mirror operators to
switch to the new config.  So, is it OK if mirrors only carry active
releases?  Does somebody need to mirror older releases?

In that case, maybe archive.openwrt.org should provide rsync service?

Baptiste

[1] https://openwrt.tetaneutral.net/releases/
[2] https://openwrt.tetaneutral.net/lastsync
[3] https://openwrt.tetaneutral.net/upstream-mirror

On 06-01-26, Baptiste Jonglez wrote:
> Hi,
> 
> The current download server is donated by Netcup in Germany [1].  It has
> been working well for some time, but we are now hitting traffic limits.
> According to their conditions, after 3 TB / day, we are rate-limited to
> 300 Mbit/s.  This has been causing issues with some buildbot workers that
> time-out when downloading the SDK to build packages.
> 
> I see three solutions:
> 
> - ask Netcup to increase the traffic limit
> - reduce the amount of traffic we generate
> - find a new hosting provider for the download server
> 
> One of the problem is that we mix many usages on the same server, some
> critical (main HTTP downloads, buildbot workers pulling the SDK) and some
> less critical (mirroring to third-parties).
> 
> I did some stats, here is the average amount of traffic we sent *per day*
> from the download server in December 2025:
> 
> - HTTP downloads.openwrt.org : 1 TB / day (mostly origin traffic for Fastly CDN)
> - HTTP sources.cdn.openwrt.org : 100 GB / day (mostly origin traffic for Fastly CDN)
> - rsync downloads from mirrors : 4 TB / day
> - rsync downloads from buildbot workers : 30 GB /day
> - rsync sources : 5 GB /day
> 
> As you can see, we have many mirrors pulling from the main download
> server, with some mirrors in particular generating a lot of traffic (up to
> 300 GB / day for some mirrors).
> 
> I think we should move to a 2-tier mirror architecture: have only 2-3
> tier-1 mirrors that are allowed to pull from us, and then ask all other
> mirrors to become tier-2 and to pull from a tier-1 mirror.  Tier 1 mirrors
> should mirror the entire archive and should commit to a long-term
> relationship.  I can contact mirror operators I know, and if they agree,
> set things up this way.
> 
> Of course buildbot workers and the archive server would still be allowed
> to rsync from the download server.
> 
> Any thoughts about this?
> 
> Baptiste
> 
> [1] https://openwrt.org/infrastructure



> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-adm