[PATCH] image: let mksquashfs4 use all processors

[Paul Spooren]

> On 28. Feb 2022, at 12:35, Stijn Tintel <stijn at linux-ipv6.be> wrote:
> 
> On 28/02/2022 13:22, Paul Spooren wrote:
>> Hi team,
>> 
>>> On 19. Feb 2022, at 19:21, Phillip Lougher <phillip.lougher at gmail.com> wrote:
>>> 
>>> On Sat, Feb 19, 2022 at 4:01 PM Felix Fietkau <nbd at nbd.name> wrote:
>>>> On 19.02.22 16:54, Stijn Tintel wrote:
>>>>> Drop the -processors argument from the mksquashfs4 call, so it will use
>>>>> all available processors. This dramatically reduces the time to create
>>>>> squashfs filesystems.
>>>>> 
>>>>> The times below are observed when building an image for my main router,
>>>>> the WatchGuard Firebox M300 (qoriq target):
>>>>> 
>>>>> Before:
>>>>> real    4m45,973s
>>>>> 
>>>>> After:
>>>>> real    0m23,497s
>>>>> 
>>>>> Signed-off-by: Stijn Tintel <stijn at linux-ipv6.be>
>>>> We need to verify that this doesn't break reproducible builds...
>>>> 
>>> It won't break reproducible builds.  Since Mksquashfs version 4.4 the
>>> ordering is guaranteed to be the same, irrespective of how many
>>> threads or processors are used.
>> I think OpenWrt uses `squashfskit`[1] which is a fork some squashfs-tools version. I think it happened due to stalled development some years ago. Anyway, if squashfs-tools is actively maintained we should consider moving back to the upstream version.
>> 
>> Some time ago I already wanted to make the move but squashfskit uses some downstream patches which expose more compression options, are those now part of squashfs-tools[2]? If not, do we need those anyway?
> Not aware if we do or don't.
>> 
>> Thanks for all further information, I’d be happy to have upstream squashfs-tools with reproducible multithread support within OpenWrt!
>> 
>> Sunshine,
>> Paul
>> 
>> [1]: https://github.com/squashfskit/squashfskit
>> [2]: https://github.com/squashfskit/squashfskit/commit/5568b6a7d7db01c8f60a502d549a431a2b7219ae
> 
> If you decide to switch back to squashfs-tools, and incorporate the
> change to let it use all processors, we should probably respect the
> number of jobs the user passes to make, and limit mksquashfs to use at
> most that number of processors. This was suggested on #openwrt-devel,
> after my initial submission. Unfortunately I have not found a way to do
> so, as MAKE_JOBSERVER is exported in include/toplevel.mk, which is not
> included in include/image.mk, and I am not familiar enough with this
> part of the code to understand the impact of adding that include.

I’m happy to take care of that once we figured out how to proceed with our fork.

> 
> Stijn