Objective of OpenWRT/x86?

[Elliott Mitchell]

On Fri, May 05, 2023 at 02:43:08AM +0100, Joseph Mullally wrote:
> On Thu, May 4, 2023 at 7:35 AM Elliott Mitchell <ehem+openwrt at m5p.com> wrote:
> >
> > The last generic OpenWRT x86 5.15 image I built less than a week ago
> > produced a roughly 28MB vmlinux.bin file.  Some of that disappears since
> > initialization portions will be freed after boot, but this is fairly
> > representative of runtime use (additional will be used for what is found
> > during boot).
> >
> > My most recent OpenWRT based image which was moderately adapted to a
> > specific target (running in a Xen VM) was 18MB.  *That* is substantial
> > enough to go after.
> 
> Here are memory figures for a fully initialized OpenWrt x86/64 QEMU
> systems. Very easy for anyone capable of proposing kernel
> configuration patches to try themselves.
> 
> $ qemu-system-x86_64 -nodefaults -smp 2 -m 128 -vga std -nic
> user,model=virtio -nic user,model=virtio -drive
> file=openwrt.img,format=raw,if=virtio
> # opkg update && opkg install luci && reboot
> 
> https://downloads.openwrt.org/releases/22.03.5/targets/x86/64/openwrt-22.03.5-x86-64-generic-ext4-combined.img.gz
> Kernel: 5.10.176
> /boot/vmlinuz size: 5.0M
> Memory used after boot settles: ~44M
> 
> https://downloads.openwrt.org/snapshots/targets/x86/64/openwrt-x86-64-generic-ext4-combined.img.gz
> Kernel: 5.15.110
> /boot/vmlinuz size: 5.6M
> Memory used after boot settles: ~46M
> (Caveat: snapshot build)

Which says all the numbers in the documentation are completely out of
whack.  Interesting to get another data point though.

Citing the size of /boot/vmlinuz isn't too useful.  That is compressed
and will be uncompressed first thing during kernel boot.  The
uncompressed image then remains in memory taking up whatever size it is
during the entire runtime of the system (minus the size of __init
symbols).

This is why I've been citing the size of vmlinuz.bin
(obj/arch/x86/boot/compressed/vmlinux.bin) since that reflects the
actual memory used by the kernel.  Since I've been doing test builds,
28MB of the 46MB used by the 5.15.10 image is the kernel executable
itself.


> > Is OpenWRT/x86 a networking device Linux distribution?
> > Yet again, I'm asking the question:  What is THE goal of OpenWRT/86?
> 
> > Is OpenWRT/x86 aiming to be a desktop Linux distribution?
> 
> No, but everything else is as modular as possible through packages etc
> so people can do whatever they want with it. Even routing packets.
> Good container support will mean the OpenWrt distribution itself won't
> ever need to grow into something like Debian and can stay focused on
> embedded/networking use cases.

Except the x86 kernels really don't look like embedded kernels.  Too many
non-generic desktop options are enabled.


> I'd say this is the single most important feature of the x86 target,
> that it just works out of the box. Its now 2023. No-one has time to be
> recompiling kernels when they just need a reliable router now while
> juggling work and family. If you handed over a work-critical
> networking appliance with hand-rolled kernels to your work colleagues
> to maintain, you might as well have handed them a barnfire. So its
> good the x86 targets just work without any messing due to being too
> lean.

Indeed, my goal is to either have a proper modular kernel for x86 or else
have some kernels with single targets.  I do not seek to get rid of the
generic kernel, merely supplement it with some alternatives.

There is a Geode kernel, why not a virtio kernel?


> The original issue raised in these threads is centered around the
> perception that the default build/install of the x86 targets have
> excessive memory usage in different scenarious, along with various
> patches and suggestions for cutting this down. However simple testing
> like above shows this isn't really a problem.

One-third of memory of the image was used to boot with no activity.
Start doing things handled by 128MB APs and it could run out quickly.


> > I can state my goal/hope for OpenWRT/x86.  The *WRT Linux distributions
> > were so named for originally targeting the LinkSys WRT54G.  This was a
> > small AP, so one might expect builds to be for small APs.  By today's
> > standards 128MB RAM and 16MB of storage is relatively small.
> >
> > Problem is instead of the recommended 128MB memory, 16MB of storage
> > (https://openwrt.org/supported_devices/432_warning) the virtualization
> > examples (https://openwrt.org/docs/guide-user/virtualization/start) are
> > suggesting massively more memory.  256MB (small Xen example), 512MB
> > (VMware OpenWRT/15.05) or 1GB (Qemu examples).
> 
> By your own standards, the current x86 builds as shown above already
> meet the "relatively small" requirements. Great :)

True.  Though there are several pieces of low-hanging fruit for
improvement.


> > KVM/Qemu, Hyper-V and Xen are likely the bigger gains.  My impression is
> > the Virtio stuff is fairly encompassing and thus a drag on others.  While
> > Hyper-V is pretty complex and also a major drag on others.  Meanwhile Xen
> > is very small and thus gets dragged down by others.
> 
> Maybe there is a case for a separate all-in-one virt profile if it
> fills up with exotic things, but as others have said IMHO its nicer to
> stay with a flexible "run-everywhere" image, unless there is a very
> clear reason not to. I'm sure no-one would mind some more virt storage
> drivers in the current x86/64 builds, but something like Xen might
> only be possible with a separate kernel/target.

The problem with one-size-fits all, is it fits everyone, badly.  I think
even though the generic target will boot on most hypervisors, the
hypervisors could use specialized targets.  The specialized targets
would allow smaller images for the hypervisors and hypervisors are
becoming more common.


The problem with citing low-hanging fruit is I can't be 100% sure which
configuration option shrank things.  Turning off SCSI won't have any
effect until you remove everything which depends on it, then suddenly
there is a large drop.  There were 5 large chunks in the default OpenWRT
kernel though which are of note:

The delta from F2FS was 28KB.  While it does have its downsides, using
only limited amounts of memory is useful.  I expected EXT4 to be larger,
but the delta was disturbing.
For EXT4 the delta is slightly over 2MB.  Yes, the EXT4 filesystem
executable module is by itself quite a bit larger than many operating
systems of 25 years ago.  Perhaps EXT4's features are valuable enough to
keep, but everyone needs to keep in mind it is *BIG*.

Another large blob is USB support.  Also slightly over 2MB, but just
under the size of EXT4.  Certainly required for real hardware, likely
also required for hardware-emulating hypervisors (VMware).  Could also
be needed if a USB-connected 802.11 module was fed into a VM.  Yet not
required if you're connecting via some flavor of serial device (emulated
or actual 16550).

The largest single blob is the Direct Rendering Manager, weighing in at
over 2MB.  Without a doubt required if you're using a VGA console, yet a
graphical console is optional under most hypervisors.

Unfortunately the remaing 4MB of savings wasn't in one place.  Simply
involves removing lots of things a VM guest will never see.  Notably
inside a VM, ACPI/EFI will have much less available.  A VM will NOT be
allowed to load microcode (unless the admin is insane).  A VM doesn't
care about processor frequency, thermals, sensors or any features of
most hardware.


All in all, a 10MB reduction off a 28MB kernel.  That seems worthwhile.

Does strike me having both EXT4 and F2FS is overkill.  Removing F2FS
might not result in huge savings, but the small bits can consume
memory quite effectively too.  Whereas EXT4 is huge.  I looked for pieces
of EXT4 which could be removed, but there were none.  The size is quite
concerning.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg at m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445