xz inadequate for a long term tarball reproducibility? [Was: [openwrt/openwrt] unetd: fix PKG_MIRROR_HASH]

[Felix Fietkau]

On 03.04.24 15:41, Petr Štetiar wrote:
> LEDE Commits <lede-commits at lists.infradead.org> [2024-04-03 07:29:21]:
> 
> Hi,
> 
> thanks a lot for a great commit message, really appreciate it! :-) Just to get
> a complete picture, I've additional questions, sorry.
> 
>> nbd pushed a commit to openwrt/openwrt.git, branch main:
>> https://git.openwrt.org/2070049c1cafa52224c946a6c334bf9fea4f549b
>> 
>> commit 2070049c1cafa52224c946a6c334bf9fea4f549b
>> Author: Paul Spooren <mail at aparcar.org>
>> AuthorDate: Wed Apr 3 13:04:36 2024 +0200
>> 
>>     unetd: fix PKG_MIRROR_HASH
>>     
>>     Our CI on GitHub as well as my local machine generates a different
>>     PKG_MIRROR_HASH from what Felix uploaded the other day.
> 
> Felix, can you provide more details about the host OS/compiler/version of the
> xz used for this tarball creation?

I didn't use xz from my host to generate this tarball. I simply hadn't 
built tools/ again in my tree when I made the update, so it was missing 
the xz downgrade.

> 
>>     After receiving Felix file, both have indeed different hashes, however
>>     when unpackaged via `xz -d` both have the same tarball content.
> 
> Paul, can you be more specific which `xz -d` is that? From the OpenWrt tools
> `staging_dir/host/bin/xz` or from your host? For example:
> 
>    $ staging_dir/host/bin/xz --version
>    xz (XZ Utils) 5.4.6
>    liblzma 5.4.6
> 
>>     Below the checksums to compare:
>>     
>>     a62bef497078c7b825f11fc8358c1a43f5db3e6d4b97812044f7653d60747d5b  dl/unetd-2024.03.31~80645766.tar.xz
>>     fbdac59581742bf208c18995b1d69d9848c93bfce487e57ba780d959e0d62fc4  dl/unetd-2024.03.31~80645766_felix.tar.xz
>>     
>>     After unpacking:
>>     
>>     a7189cae90bc600abf3a3bff3620dc17a9143be8c27d27412de6eb66a1cf1b7d  dl/unetd-2024.03.31~80645766.tar
>>     a7189cae90bc600abf3a3bff3620dc17a9143be8c27d27412de6eb66a1cf1b7d  dl/unetd-2024.03.31~80645766_felix.tar
>>     
>>     The tarball with the wrong hash was accidentally generated without the xz
>>     revert to version 5.4.6
> 
> interesting, would it be possible to upload `unetd-2024.03.31~80645766_felix.tar.xz`
> somewhere, so anyone interested could take a look?

Jonas pointed out that the tarball can be reproduced by compressing the 
.tar with -T 0.
https://github.com/openwrt/openwrt/pull/15057#issuecomment-2034437153

- Felix