High volume of traffic on download server - move to a 2-tiers rsync setup for mirrors?

[Baptiste Jonglez]

On 06-01-26, Rich Brown wrote:
> Thank you Baptiste, for this analysis.
> 
> I re-sorted your list to by data consumption. it’s clear that the mirror traffic is 4x the second place consumer, and 3x ALL the traffic. In fact, the mirror traffic on its own seems to exceed the traffic cap.
> 
> - rsync downloads from mirrors : 4 TB / day
> - HTTP downloads.openwrt.org : 1 TB / day (mostly origin traffic for Fastly CDN)
> - HTTP sources.cdn.openwrt.org : 100 GB / day (mostly origin traffic for Fastly CDN)
> - rsync downloads from buildbot workers : 30 GB /day
> - rsync sources : 5 GB /day
> 
> I understand (and would support) your multi-tier suggestion below, but I am going to suggest a heretical solution, not because I advocate for it, but simply to start a discussion.
> 
> Proposal: We should eliminate all mirrors. 
> 
> Background: 20 years ago (more or less) mirror download sites for popular projects were essential. Primary hosts didn’t have the horsepower to handle all the download requests and traffic. In addition, it allowed “far flung” institutions to show their camaraderie and support for the project. They were a good thing. Today, in the presence of CDNs and much higher speed hosts, it’s not clear that all this “mirror machinery” is justified by the complexity.
> 
> Analysis: We should determine whether these mirrors actually help our central traffic problem. How many mirror sites do we support? Do we know the outbound traffic for In the mirror sites? What kind of data do they provide (sources? binary images?)
> 
> It is thinkable to me (although I have no information either way), that our central servers + Fastly CDN could easily handle the requests from individuals who otherwise would go to the mirrors.
> 
> If that’s the case, we could eliminate all these mirrors, eliminate the (human) communication and whatever infrastructure we support, save us some bandwidth, and some brainpower.
> 
> Of course, I may totally misunderstand the problem, or be proven to be totally wrong - that’s OK. But still I felt I need to ask the question.

I like provocative thinking ;)

One thing that is maybe not clear: we don't manage the mirrors ourselves,
and the mirrors are not used by downloads.openwrt.org.  All "official"
traffic is already handled by the CDN and our single download server, with
old releases handled by the archive server.  So, mirrors don't cost us
much in term of time and infrastructure (except this traffic problem).
So far, the machinery is just "the rsync server is open to all".

There are two kind of mirrors as far as I know: "far-flung institutions
showing their support" as you say, and people running private mirrors
(which I guess they have good reasons for, maybe they have large fleets of
OpenWrt devices)

There is an incomplete list here: https://openwrt.org/downloads#mirrors

I think public mirrors can still be useful, it leaves alternative choices
if we have a big outage of the download server and the archive server
(unlikely, but still possible).  Another reason is better performance: the
CDN helps but still needs to fetch content from the download server in
Germany.

That being said, I agree the current number of mirrors is kind of
unreasonable for this purpose.

The idea of the 2-tiers system is precisely to delegate mirror
distribution to a few trusted people/org that have the knowledge and
infrastructure for this.

Baptiste

> > On Jan 6, 2026, at 06:13, Baptiste Jonglez <baptiste at bitsofnetworks.org> wrote:
> > 
> > I think we should move to a 2-tier mirror architecture: have only 2-3
> > tier-1 mirrors that are allowed to pull from us, and then ask all other
> > mirrors to become tier-2 and to pull from a tier-1 mirror.  Tier 1 mirrors
> > should mirror the entire archive and should commit to a long-term
> > relationship.  I can contact mirror operators I know, and if they agree,
> > set things up this way.
> > 
> > Of course buildbot workers and the archive server would still be allowed
> > to rsync from the download server.
> > 
> 
> 
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel