[PATCH] image: let mksquashfs4 use all processors
mail at aparcar.org
Mon Feb 28 03:49:22 PST 2022
> On 28. Feb 2022, at 12:35, Stijn Tintel <stijn at linux-ipv6.be> wrote:
> On 28/02/2022 13:22, Paul Spooren wrote:
>> Hi team,
>>> On 19. Feb 2022, at 19:21, Phillip Lougher <phillip.lougher at gmail.com> wrote:
>>> On Sat, Feb 19, 2022 at 4:01 PM Felix Fietkau <nbd at nbd.name> wrote:
>>>> On 19.02.22 16:54, Stijn Tintel wrote:
>>>>> Drop the -processors argument from the mksquashfs4 call, so it will use
>>>>> all available processors. This dramatically reduces the time to create
>>>>> squashfs filesystems.
>>>>> The times below are observed when building an image for my main router,
>>>>> the WatchGuard Firebox M300 (qoriq target):
>>>>> real 4m45,973s
>>>>> real 0m23,497s
>>>>> Signed-off-by: Stijn Tintel <stijn at linux-ipv6.be>
>>>> We need to verify that this doesn't break reproducible builds...
>>> It won't break reproducible builds. Since Mksquashfs version 4.4 the
>>> ordering is guaranteed to be the same, irrespective of how many
>>> threads or processors are used.
>> I think OpenWrt uses `squashfskit` which is a fork some squashfs-tools version. I think it happened due to stalled development some years ago. Anyway, if squashfs-tools is actively maintained we should consider moving back to the upstream version.
>> Some time ago I already wanted to make the move but squashfskit uses some downstream patches which expose more compression options, are those now part of squashfs-tools? If not, do we need those anyway?
> Not aware if we do or don't.
>> Thanks for all further information, I’d be happy to have upstream squashfs-tools with reproducible multithread support within OpenWrt!
>> : https://github.com/squashfskit/squashfskit
>> : https://github.com/squashfskit/squashfskit/commit/5568b6a7d7db01c8f60a502d549a431a2b7219ae
> If you decide to switch back to squashfs-tools, and incorporate the
> change to let it use all processors, we should probably respect the
> number of jobs the user passes to make, and limit mksquashfs to use at
> most that number of processors. This was suggested on #openwrt-devel,
> after my initial submission. Unfortunately I have not found a way to do
> so, as MAKE_JOBSERVER is exported in include/toplevel.mk, which is not
> included in include/image.mk, and I am not familiar enough with this
> part of the code to understand the impact of adding that include.
I’m happy to take care of that once we figured out how to proceed with our fork.
More information about the openwrt-devel