Disk usage increase on download server / mirrors

[Baptiste Jonglez]

Hi,

On 21-07-19, Baptiste Jonglez wrote:
> Hi,
> 
> I manage an OpenWrt download mirror that rsyncs from the main download
> server, and the mirror recently filled up its disk.
> 
> Disk usage seems to have increased rather quickly over time:
> 
> - one year ago: 210 GB
> - one month ago: 420 GB
> - now: more than 540 GB

We are now at almost 900 GB, and I noticed some mirrors started dropping
the snapshots altogether (and we might do the same from our mirror, we are
running out of space).

There was the idea to drop old kmod archives from snapshots after 6
months, is it still possible?  It would help a lot with disk usage.

It also needs to be applied to stable snapshots (maybe even more
aggressively?) because it is also starting to take lots of space:

   67.2 GiB [##########] /18.06-SNAPSHOT
   66.7 GiB [######### ] /19.07-SNAPSHOT
   27.0 GiB [####      ] /packages-19.07
   22.4 GiB [###       ] /19.07.0-rc1
   22.2 GiB [###       ] /19.07.3
   22.1 GiB [###       ] /19.07.2
   22.1 GiB [###       ] /19.07.4
   22.1 GiB [###       ] /19.07.1
   22.0 GiB [###       ] /19.07.0

Thanks,
Baptiste

> The upcoming 19.07 release explains part of the increased usage, but it's
> not actually that much (45 GB for targets + packages)
> 
> However, I noticed that all kernel modules are kept for each kernel bump,
> including for snapshots.  From what I can tell:
> 
> - there are 62 subtargets (although not all of them are built)
> - the kernel modules for each subtarget take around 17 MB of disk space
> - there are currently around 100 different kernel versions in snapshots
> 
> From these numbers, I estimate that each kernel bump takes 1 GB of space
> on the download server, and there are thus 100 GB of disk space "wasted"
> on old snapshot kernel versions.
> 
> I understand the rationale of keeping old kernel modules around, but maybe
> they could be cleaned up after a delay, especially for snapshots?
> 
> Thanks,
> Baptiste
> 
> PS: there may be other reasons for the disk usage increase, but this one
> looks like the most likely reason



> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> http://lists.infradead.org/mailman/listinfo/openwrt-adm

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openwrt.org/pipermail/openwrt-adm/attachments/20201124/403d6871/attachment.sig>