Removing outdated releases from downloads.o.o

[Baptiste Jonglez]

Hi,

On 03-01-24, Paul Spooren wrote:
> Hi, I’m currently checking what needs to remove mirror-01/02 and replace them by a both faster and cheaper machine.
> 
> While we store all old releases (e.g. 5 year old 17.01), all links on the website point to archive.o.o. I suggest that we only serve snapshots and supported releases on the main download server and serve all other releases via the archive server only.
> 
> This would break very old forum links but for those cases we could either say such old shouldn’t be used in the first place or add a forwarding rule to the archive server.

I agree that the current amount of storage is becoming unmanageable.
I put some storage numbers for mirror operators on [1], the TL;DR is that
we generate around +700 GB of new release files per year and it's increasing.
This is really a lot.

However, changing the model needs some thinking, around several axes:

- determining relevant classes of data.  We could have three logical
  repositories: snapshots, active releases, old releases.  They might
  still technically be hosted on the same server, but they would be easier
  to separate.

- catering for mirror operators: currently some operators decide to mirror
  only part of the data for space reason, but it's not consistent.  The
  classes of data above would help: we could ask every mirror to carry at
  least "active releases" as a common denominator for consistency.

- delivery: keeping stable URLs over time, ensuring HTTP stays available,
  taking into account limited clients like uclient-fetch for stuff like
  TLS config and redirection

- disaster recovery: backups, recovery

- server cost vs. reliability vs. stability over time.  My feeling is that
  you can get only two out of three.  For instance, donations or free
  credits usually don't last for a long time.  The current download server
  is mostly reliable and stable, but expensive.  And so on.

From here, there are many solutions: put everything on object storage and
be done with it; keep an expensive and reliable server for active releases
only and rely on the community for the rest of the data; manage a second
cheaper server for the rest as part of the OpenWrt infra; use object
storage for the rest.

I would go with the lowest maintenance option whenever possible, like a
beefy VM for active releases and snapshots, and move to object storage for
the rest (not Amazon of course, but a EU provider such as Infomaniak [2]).
Then have a frontend configuration such as the current CDN that would
transparently proxy to the right place.

Baptiste

[1] https://openwrt.org/downloads#how_to_mirror
[2] https://www.infomaniak.com/en/hosting/public-cloud/prices
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openwrt.org/pipermail/openwrt-adm/attachments/20240105/4271968a/attachment.sig>