[OpenWrt-Devel] [PATCH 2/5] build: image: Add pad-to and pad-rootfs-squashfs helpers

[Nishant Sharma]

Hi,

Bumping this thread after years.

On 01/04/19 11:03, Jo-Philipp Wich wrote:
>>>> I would like to avoid adding generating padded images by default.
>>>
>>>> I just came up with this simple script, which takes an existing image and
>>>> pads it to full size: http://nbd.name/pad-image.pl With this, shipping
>>>> padded images should be unnecessary.
>>>
>>> Ok, if that is preferred, it's fine with me. But we should probably add some
>>> note somewhere, that in order to test this images in QEMU (x86, armvirt,
>>> malta), the images should be padded with this script in order to provide
>>> usable images.
> 
> Most other targets ship image artifacts which are usable ootb, requiring
> one extra step to pad the combined images is a waste of user resources
> every single time. It also causes recurring confusion among users
> wanting to use x86 builds since the need for padding is neither
> documented, nor obvious while a combined.img.gz makes it obvious that it
> is an image file which requires decompression.
> 
> I really don't see the huge problem with conservatively padding the
> combined image artifacts to something like 32 or 48MB, it must not even
> be 256M or more.

I build large squashfs images (16GB, 32GB) for x86_64 running on Xeon 
processors for high workload gateways running VPN, Squid, Snort etc.

Till OpenWrt 19.07, I was able to build 32GB images on a machine with 
only 8GB of RAM and 8GB of swap. sysupgrade also worked fine on those 
images.

But since 21.02, I am getting the *memory exhausted* error.

Below is the build log snippet for 23.05.2:

<log>

dd 
if=/home/devuser/auto-build/hopbox-os/hopbox-openwrt/build_dir/target-x86_64_musl/linux-x86_64/root.squashfs 
 >> 
/home/devuser/auto-build/hopbox-os/hopbox-openwrt/build_dir/target-x86_64_musl/linux-x86_64/tmp/hopbox-arthur-23.05.2.1-x86-64-generic-squashfs-rootfs.img.gz

39891+1 records in
39891+1 records out
20424365 bytes (20 MB, 19 MiB) copied, 0.0445292 s, 459 MB/s

dd 
if=/home/devuser/auto-build/hopbox-os/hopbox-openwrt/build_dir/target-x86_64_musl/linux-x86_64/tmp/hopbox-arthur-23.05.2.1-x86-64-generic-squashfs-rootfs.img.gz 
of=/home/devuser/auto-build/hopbox-os/hopbox-openwrt/build_dir/target-x86_64_musl/linux-x86_64/tmp/hopbox-arthur-23.05.2.1-x86-64-generic-squashfs-rootfs.img.gz.new 
bs=16642998272 conv=sync

dd: memory exhausted by input buffer of size 16642998272 bytes (16 GiB)

</log>

Here the block size is set to 16GiB to pad rootfs and there are not 
enough system resources available on the build host to be able honour that.

I don't see any special case to handle x86/x86_64 in image-commands.mk

Is there a way I can disable padding for x86_64?

Thanks in advance.

Regards,
Nishant