[PATCH] build: SDK/IB reproducible and faster compression

[Matthias Schiffer]

On 8/22/20 1:45 AM, Paul Spooren wrote:
> 
> On 21.08.20 06:47, Matthias Schiffer wrote:
>> Did you have a look at the zstd patches I sent a while ago? zstd is
>> superior to xz in most cases - the only reason I didn't merge the patches
>> yet was that all of our phase2 buildbots would need to have zstd installed
>> to extract the new SDK before we can actually switch (and I'm not sure who
>> can take care of that - jow?).
> 
> I'm aware of the patches and like the step forward to zstd! However I see
> the downstream concerns as well. How about applying my patches and you send
> a rebased version of your zstd patches, including the changes for
> reproducibility?

Fine with me.

> 
> Our buildbot containers are based on debian:9. When installing the zstd
> package it works fine with tar (tar -I zstd -xf foobar.tar.zst). If desired
> I can update the containers to contain zstd.
> 
> buildbot at 00801fffe9e2:~$ tar --version
> tar (GNU tar) 1.29
> 
> Starting with tar 1.31  we could also use `--zstd`, which is however not
> even part of Debian 10.
> 
> I have no access to the ansible.git repository, wouldn't it be enough to
> just do the following change?
> 
> -  sdk_pattern = openwrt-sdk-*.tar.xz
> + sdk_pattern = openwrt-sdk-*.tar.*
> 
> The following line in buildbot.git/phase2 could be duplicated and run based
> on which branch is used:
> https://git.openwrt.org/?p=buildbot.git;a=blob;f=phase2/master.cfg;h=5ed7eff5728da299dee7e01ff416fc11533bf13a;hb=HEAD#l409

It might make sense to make the unpack command an explicit configuration
variable and pipe the archive through that before passing it to tar - we'll
have to do that anyways as long as we need to support tar versions that
don't handle zstd themselves.

I can try to come up with patches for that, but I'm not familiar with our
buildbot scripts.


> 
> 
> For an easier migration maybe I'd be nice to create both zstd and xz for a
> month (week?), that could be implemented in an updated version of you
> patchset.

Seems like overkill - getting zstd installed in the container and preparing
the builtbot scripts can be done beforehand, so when we actually switch, we
only need to adjust the per-branch configuration. I don't see a problem
with breaking the builds for 2-3 days during the transition - it will only
affect phase2 for master snapshots anyways.


> 
>> The mtime part of this patch makes sense.
> Thanks


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openwrt.org/pipermail/openwrt-devel/attachments/20200822/ca0c7d29/attachment-0001.sig>