Disk usage increase on download server / mirrors

[Baptiste Jonglez]

Hi,

On 25-11-20, Hannu Nyman wrote:
> Baptiste Jonglez kirjoitti 25.11.2020 klo 1.37:
> > On 21-07-19, Jo-Philipp Wich wrote:
> > > I agree that we should start purging the oldest kmod archives. I'd
> > > propose dropping a kmod archive directory whenever it exceeds the age of
> > > 6 months.
> > This looks like a good idea.
> > 
> > To expand one my previous email, here are some numbers for snapshots:
> > 
> > - current total space (targets + packages): 208.0 GiB
> > - cleaning up kmods from > 6 months ago: 144.1 GiB
> > - cleaning up kmods from > 3 months ago: 120.0 GiB
> > - cleaning up kmods from > 1 month  ago: 95.7 GiB
> > 
> > So, snapshot kmods are responsible for about +10 GB of storage each month.
> > 
> > And for releases (18.06-SNAPSHOT and 19.07-SNAPSHOT):
> > 
> > - current total space (targets + packages): 682.9 GiB
> > - cleaning up kmods from > 6 months ago: 610.3 GiB
> > - cleaning up kmods from > 1 months ago: 597.4 GiB
> > 
> > If we clean up [snapshot kmods > 6 months] and [stable snapshots kmods > 1 months],
> > we gain 150 GB.
> > 
> > Here is what I used on a mirror to test:
> > 
> >    $ find snapshots/targets/ -type d -wholename '*/kmods/*' -ctime +180 -exec rm -r '{}' \;
> >    $ find releases/*SNAPSHOT -type d -wholename '*/kmods/*' -ctime +30 -exec rm -r '{}' \;
> 
> I would shorten the times quite much more.
> 
> I feel that nobody should be using release-branch snapshots and expect to be
> able to download kmods later. The role of release-branch snapshots is mainly
> for continuous compile testing. I think that the kmod lifetime should be at
> maximum 1 month, but might be just two weeks quite as well. If somebody want
> to use the HEAD of a release branch, he could just as well compile firmware
> by himself.
> 
> Regarding master snapshots, the reasonable time might be something like 3
> months (instead of 6). Master develops continuously, and I don't see the
> need to support 6-month old bleeding edge... That edge has dulled quite much
> by then :-(

This makes sense.  Petr has implemented a cleanup cron with the following
parameters:

- delete kmods for release snapshots after 30 days
- delete kmods for snapshots after 90 days
- in any case, the 5 newest kmods for each subtarget are always kept

This freed between 150 and 200 GB of disk space for mirrors.

Thanks Petr for handling this!

Baptiste
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openwrt.org/pipermail/openwrt-adm/attachments/20210221/672e659e/attachment.sig>