[PATCH] image: let mksquashfs4 use all processors

Paul Spooren mail at aparcar.org
Tue Mar 29 05:34:19 PDT 2022


I extended the reasoning in the commit message and merged this as discussed in IRC

> On 28. Feb 2022, at 11:49, Paul Spooren <mail at aparcar.org> wrote:
> 
> 
> 
>> On 28. Feb 2022, at 12:35, Stijn Tintel <stijn at linux-ipv6.be> wrote:
>> 
>> On 28/02/2022 13:22, Paul Spooren wrote:
>>> Hi team,
>>> 
>>>> On 19. Feb 2022, at 19:21, Phillip Lougher <phillip.lougher at gmail.com> wrote:
>>>> 
>>>> On Sat, Feb 19, 2022 at 4:01 PM Felix Fietkau <nbd at nbd.name> wrote:
>>>>> On 19.02.22 16:54, Stijn Tintel wrote:
>>>>>> Drop the -processors argument from the mksquashfs4 call, so it will use
>>>>>> all available processors. This dramatically reduces the time to create
>>>>>> squashfs filesystems.
>>>>>> 
>>>>>> The times below are observed when building an image for my main router,
>>>>>> the WatchGuard Firebox M300 (qoriq target):
>>>>>> 
>>>>>> Before:
>>>>>> real 4m45,973s
>>>>>> 
>>>>>> After:
>>>>>> real 0m23,497s
>>>>>> 
>>>>>> Signed-off-by: Stijn Tintel <stijn at linux-ipv6.be>
>>>>> We need to verify that this doesn't break reproducible builds...
>>>>> 
>>>> It won't break reproducible builds. Since Mksquashfs version 4.4 the
>>>> ordering is guaranteed to be the same, irrespective of how many
>>>> threads or processors are used.
>>> I think OpenWrt uses `squashfskit`[1] which is a fork some squashfs-tools version. I think it happened due to stalled development some years ago. Anyway, if squashfs-tools is actively maintained we should consider moving back to the upstream version.
>>> 
>>> Some time ago I already wanted to make the move but squashfskit uses some downstream patches which expose more compression options, are those now part of squashfs-tools[2]? If not, do we need those anyway?
>> Not aware if we do or don't.
>>> 
>>> Thanks for all further information, I’d be happy to have upstream squashfs-tools with reproducible multithread support within OpenWrt!
>>> 
>>> Sunshine,
>>> Paul
>>> 
>>> [1]: https://github.com/squashfskit/squashfskit
>>> [2]: https://github.com/squashfskit/squashfskit/commit/5568b6a7d7db01c8f60a502d549a431a2b7219ae
>> 
>> If you decide to switch back to squashfs-tools, and incorporate the
>> change to let it use all processors, we should probably respect the
>> number of jobs the user passes to make, and limit mksquashfs to use at
>> most that number of processors. This was suggested on #openwrt-devel,
>> after my initial submission. Unfortunately I have not found a way to do
>> so, as MAKE_JOBSERVER is exported in include/toplevel.mk, which is not
>> included in include/image.mk, and I am not familiar enough with this
>> part of the code to understand the impact of adding that include.
> 
> I’m happy to take care of that once we figured out how to proceed with our fork.
> 
>> 
>> Stijn




More information about the openwrt-devel mailing list