[PATCH] build: SDK/IB reproducible and faster compression

[Matthias Schiffer]

On 8/21/20 3:47 AM, Paul Spooren wrote:
> As friends of reproducible builds please have a look at this patch.
> 
> On 13.08.20 13:46, Paul Spooren wrote:
>> Both IB and SDK now use the same logic for packing.
>>
>> This commit add reproducible multithread compression to the SDK and
>> corrects the file mtime for both. Previously all files where just copied
>> over from the build system, generating random mtimes.
>>
>> Signed-off-by: Paul Spooren <mail at aparcar.org>
>> ---
>> This speeds up SDK building on my machine by 4x.
>>
>>   target/imagebuilder/Makefile | 5 ++++-
>>   target/sdk/Makefile          | 3 ++-
>>   2 files changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/target/imagebuilder/Makefile b/target/imagebuilder/Makefile
>> index b463feb456..ad19ab2b53 100644
>> --- a/target/imagebuilder/Makefile
>> +++ b/target/imagebuilder/Makefile
>> @@ -86,7 +86,10 @@ endif
>>       (cd $(PKG_BUILD_DIR); find staging_dir/host/bin/
>> $(IB_LDIR)/scripts/dtc/ -type f | \
>>           $(XARGS) $(SCRIPT_DIR)/bundle-libraries.sh
>> $(PKG_BUILD_DIR)/staging_dir/host)
>>       STRIP=sstrip $(SCRIPT_DIR)/rstrip.sh
>> $(PKG_BUILD_DIR)/staging_dir/host/bin/
>> -    $(TAR) -cf - -C $(BUILD_DIR) $(IB_NAME) | xz -T$(if $(filter
>> 1,$(NPROC)),2,0) -zc -7e > $@
>> +    (cd $(BUILD_DIR); \
>> +        tar -I '$(STAGING_DIR_HOST)/bin/xz -7e -T$(if $(filter
>> 1,$(NPROC)),2,0)' -cf $@ $(IB_NAME) \
>> +        --mtime="$(shell date --date=@$(SOURCE_DATE_EPOCH))"; \
>> +    )
>>     download:
>>   prepare:
>> diff --git a/target/sdk/Makefile b/target/sdk/Makefile
>> index d3552b47eb..022a791ebf 100644
>> --- a/target/sdk/Makefile
>> +++ b/target/sdk/Makefile
>> @@ -152,7 +152,8 @@ $(BIN_DIR)/$(SDK_NAME).tar.xz: clean
>>       find $(SDK_BUILD_DIR) -name CVS | $(XARGS) rm -rf
>>       -make -C $(SDK_BUILD_DIR)/scripts/config clean
>>       (cd $(BUILD_DIR); \
>> -        tar -I 'xz -7e' -cf $@ $(SDK_NAME); \
>> +        tar -I '$(STAGING_DIR_HOST)/bin/xz -7e -T$(if $(filter
>> 1,$(NPROC)),2,0)' -cf $@ $(SDK_NAME) \
>> +        --mtime="$(shell date --date=@$(SOURCE_DATE_EPOCH))"; \
>>       )
>>     download:

Did you have a look at the zstd patches I sent a while ago? zstd is
superior to xz in most cases - the only reason I didn't merge the patches
yet was that all of our phase2 buildbots would need to have zstd installed
to extract the new SDK before we can actually switch (and I'm not sure who
can take care of that - jow?).

The mtime part of this patch makes sense.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openwrt.org/pipermail/openwrt-devel/attachments/20200821/f47321f3/attachment.sig>