Objective of OpenWRT/x86?

[Alberto Bursi]

On 26/04/23 22:17, Elliott Mitchell wrote:
> Well, was a specific objective ever chosen for the x86 version of
> OpenWRT?
> 
> I can state my goal/hope for OpenWRT/x86.  The *WRT Linux distributions
> were so named for originally targeting the LinkSys WRT54G.  This was a
> small AP, so one might expect builds to be for small APs.  By today's
> standards 128MB RAM and 16MB of storage is relatively small.
 >

Afaik the x86_64 images approach has always been "include what you can 
to make it run on everything without need to customize the image".
Storage is integrated in kernel because otherwise it won't see the 
rootfs and fail to boot, everything else is in modules (possibly) but 
modules for most ethernet controllers are always included in default build.

Then there is the virtualized drivers that were just added to the kernel 
and not properly modulized, maybe it was easier like that, don't know.

Because on x86_64 storage and RAM is measured in Gigabytes so saving a 
few hundred kilobytes by removing the "all hardware just works with a 
single image" feature is NOT a worthy tradeoff.

> 
> Since this network is in the moving towards the server phase, the AP drew
> my attention.  Turns out all of the hardware of an AP is available for
> servers, though in distinct form-factors.  If the server has a full
> IOMMU, the hardware can be directed to a VM and you have a proper AP in a
> VM.
> 
> Problem is instead of the recommended 128MB memory, 16MB of storage
> (https://openwrt.org/supported_devices/432_warning) the virtualization
> examples (https://openwrt.org/docs/guide-user/virtualization/start) are
> suggesting massively more memory.  256MB (small Xen example), 512MB
> (VMware OpenWRT/15.05) or 1GB (Qemu examples).
> 

Did you actually check that OpenWrt x86 needs that much memory on x86 or 
are you running on assumptions someone made when writing wiki articles?

Because I've been running OpenWRT on both x86 hardware and VMs and for a 
"normal" router build (i.e. default image with Luci) it really does not 
need more than 128MB RAM.

For a more "fat" image with OpenMPTCProuter (an OpenWrt downstream 
project that allows true wan bonding using MPTCP feature in a customized 
Linux kernel), where it's handling the bonding of 3 different WANs over 
VPN tunnels to a cloud server, I have given it 512MB RAM and according 
to the "free" command or htop it's using around 80 MB of RAM with 300 MB 
that are literally free, not even used for caching.
And this is a very "kitchen sink" image with tons of packages/services 
included and running (there is also ModemmManager for example), a custom 
kernel with more stuff enabled and all that. So it's just a ballpark 
estimate but still, 80MB isn't that much for what it is doing.

root at OpenMPTCProuter:~# free
               total        used        free      shared  buff/cache 
available
Mem:         495952       81028      313944         752      100980 
361636
Swap:             0           0           0


> 
> I don't yet have a full handle on the issues.  My first thought was the
> large sizes come from early stage development and not wanting to bother
> with memory limitations.  Another possibility is this comes from simply a
> thought pattern of "x86 memory is cheap, just throw memory at it".
> 
> Yet if one is wanting to use this for serious purposes, memory IS a
> concern.  GCC is pretty capable for x86, a quick check suggests GCC tends
> to produce binaries for x86 which are four-fifths the size of those for
> ARM.  Yet everything for OpenWRT/x86 is suggesting at least *double* the
> memory?!
> 

Again with this "suggesting". The people writing the docs may or may not 
have had the same assumptions as you have. Maybe like me they are 
running the VM on a cluster with 128GB of RAM per host so they have just 
chosen whatever number they feel is a "low amount of RAM" without going 
deeper, if they waste 512MB or 1GB it's no big deal.

It's always recommended to do some tests yourself to get some first-hand 
information so you can frame the issue without someone else's 
assumptions in the way.

> 
> One issue I've found is the kernel configurations seem ill-suited to x86.
> Almost any storage driver used by 10 or more people is built into the
> kernel.  As a result the *single* kernel is huge.
>

"huge" in what context? Yes it's more than for most embedded devices 
that have 16MB flash but let's be real on x86 it's hard to find storage 
that is smaller than 1GB and RAM is likewise plentiful so what are we 
talking about. Even ancient Geode devices like Alix boards have 256MB 
RAM which is still plenty for what they can actually do.

I don't know if the storage on the default image for x86 is still 
limited to 128MB by default because as said in the commit "this is the 
smaller size of CF card available", but if that is still the case this 
is imho something more worth investing time than trying to shave a MB or 
two from the kernel on a system that can have hundreds of GB of storage 
and RAM. There were PRs for auto-expanding the storage partition to be 
able to use all the storage space on first boot, to have dual boot 
system (so you always have a failover when you upgrade) and so on.

> 
> The conventional approach for x86 is to use an initial ramdisk and build
> everything as modules.  Issue here is OpenWRT doesn't currently have the
> scripting/tools for runtime probing of a storage subsystem.  I think this
> is a fine approach if the committers are up for all the change this will
> entail.
> 
> Alternatively x86 could be broken into more builds.  This would emulate
> how OpenWRT works for all other devices.  "generic" would likely be 2-3
> distinct builds.  "64" would be 4-6 distinct builds.  Issue here is how
> many distinct builds should be created?
> 
> If one was to go this direction, I suppose there might be "giant" or
> "desktop" build.  Each hypervisor could use a target, include "hardware"
> guaranteed to be present.  Then build all network drivers as modules (so
> any device can be passed-in).
> 
> 
> 
> Examples of things which don't seem to fit well are CONFIG_AGP and
> CONFIG_DRM.  I would expect those for a desktop Linux distribution due
> to GUI.  For OpenWRT which tends towards networking/embedded those seem
> out of place.  CONFIG_FB is similar, though some devices do have actual
> limited frame-buffers.
 >

I don't know exactly how much impact this has but on many x86 systems 
it's very nice to have the video ports work so you can connect a screen 
and interact with console. On some older boards or laptops it might be 
over AGP bus so dropping this would drop their screen support.

For reference, other network appliance projects like pfSense and 
OpnSense also support basic text console on video ports so that's 
something many "expect".

Now, it's probably all ancient stuff that not a lot of people are using. 
So yeah ok remove it from x86_64 but leave it on the x86 and the legacy 
images because that's where people with ancient hardware might need it.

> 
> Another item is the goal of trying to self-host.  Being able to self-host
> is a worthy goal, but that has very distinct needs from an embedded
> networking device.
> 
> 

Imho this is is very much out of scope. Other linux distros aren't going 
to disappear any time soon.

-Alberto