[PATCH] image: let mksquashfs4 use all processors

[Stijn Tintel]

On 28/02/2022 13:22, Paul Spooren wrote:
> Hi team,
>
>> On 19. Feb 2022, at 19:21, Phillip Lougher <phillip.lougher at gmail.com> wrote:
>>
>> On Sat, Feb 19, 2022 at 4:01 PM Felix Fietkau <nbd at nbd.name> wrote:
>>> On 19.02.22 16:54, Stijn Tintel wrote:
>>>> Drop the -processors argument from the mksquashfs4 call, so it will use
>>>> all available processors. This dramatically reduces the time to create
>>>> squashfs filesystems.
>>>>
>>>> The times below are observed when building an image for my main router,
>>>> the WatchGuard Firebox M300 (qoriq target):
>>>>
>>>> Before:
>>>> real    4m45,973s
>>>>
>>>> After:
>>>> real    0m23,497s
>>>>
>>>> Signed-off-by: Stijn Tintel <stijn at linux-ipv6.be>
>>> We need to verify that this doesn't break reproducible builds...
>>>
>> It won't break reproducible builds.  Since Mksquashfs version 4.4 the
>> ordering is guaranteed to be the same, irrespective of how many
>> threads or processors are used.
> I think OpenWrt uses `squashfskit`[1] which is a fork some squashfs-tools version. I think it happened due to stalled development some years ago. Anyway, if squashfs-tools is actively maintained we should consider moving back to the upstream version.
>
> Some time ago I already wanted to make the move but squashfskit uses some downstream patches which expose more compression options, are those now part of squashfs-tools[2]? If not, do we need those anyway?
Not aware if we do or don't.
>
> Thanks for all further information, I’d be happy to have upstream squashfs-tools with reproducible multithread support within OpenWrt!
>
> Sunshine,
> Paul
>
> [1]: https://github.com/squashfskit/squashfskit
> [2]: https://github.com/squashfskit/squashfskit/commit/5568b6a7d7db01c8f60a502d549a431a2b7219ae

If you decide to switch back to squashfs-tools, and incorporate the
change to let it use all processors, we should probably respect the
number of jobs the user passes to make, and limit mksquashfs to use at
most that number of processors. This was suggested on #openwrt-devel,
after my initial submission. Unfortunately I have not found a way to do
so, as MAKE_JOBSERVER is exported in include/toplevel.mk, which is not
included in include/image.mk, and I am not familiar enough with this
part of the code to understand the impact of adding that include.

Stijn