Objective of OpenWRT/x86?

[Elliott Mitchell]

On Fri, Apr 28, 2023 at 12:04:15PM -0600, Philip Prindeville wrote:
> 
> > On Apr 26, 2023, at 2:17 PM, Elliott Mitchell <ehem+openwrt at m5p.com> wrote:
> > 
> > Well, was a specific objective ever chosen for the x86 version of
> > OpenWRT?
> 
> Does it need one?  As the most ubiquitous hardware out there for many users, why would we need to box it in?
> 
> Someone might want to throw a generic image onto a PC they have lying around until they decide it's worth bringing up specialized hardware... going out an buying it, building an image, installing it, troubleshooting it, etc.

The usual solution is to make most things modules.  Then only the
portions needed for the particular situation get loaded.  Having a few
specifically targeted builds could also accomplish the goal.


> > I can state my goal/hope for OpenWRT/x86.  The *WRT Linux distributions
> > were so named for originally targeting the LinkSys WRT54G.  This was a
> > small AP, so one might expect builds to be for small APs.  By today's
> > standards 128MB RAM and 16MB of storage is relatively small.
> 
> Yeah, we had a discussion some years ago about "FAT" vs. "SKINNY" platforms that never went anywhere.  All of the AMD64 platforms were FAT.

s/FAT/OBESE/

> Having robust hardware that you can prototype on, including CPU or memory hungry development, allows us to prove the value of new capabilities that might be ubiquitous in the next generation of COTS and SoHo routers.

I've been repeatedly typing I've got no objection to having the jumbo
configurations.  My objection is to the lack of smaller configurations.

> After all, the BoM for an extra GB of DRAM is literally pennies in large production volumes.

If you're willing to accept the quality of memory in that price range.

> BTW, I run OpenWrt for its firewall capabilities... None of my boxes actually has wireless hardware at this point, and I use a few Ubiquiti AP-AC-Pro's or U6-LR's scattered around the house.

I can see that.  Fun part with VMs is being able to reboot the system,
but NAT connections remaining connected.

> The rule of thumb when I worked at MIT on the X Window System, was that what's the top-of-the-line workstation today will be "budget" hardware 3-4 years out, so to not worry about hardware constraints because by the time you release stable, production quality software, the hardware will have gone through one or two evolutions...
> 
> Today's one-off advanced prototyping platform is tomorrow's BestBuy budget/clearance item...

Quite true.  There is still a need to try to restrain memory consumption.

> > Since this network is in the moving towards the server phase, the AP drew
> > my attention.  Turns out all of the hardware of an AP is available for
> > servers, though in distinct form-factors.  If the server has a full
> > IOMMU, the hardware can be directed to a VM and you have a proper AP in a
> > VM.
> 
> "Moving towards the server phase"?  I'm not getting what you mean by this.

Any reasonably sized network will have a phase of things moving towards
the server room and a phase of things moving towards the desktop (or
phone).  You can try to damp out the oscillations, but both phases will
be there.  Right now things are moving towards the center for me.

> See above: the radios and antennae I can get as add-ons for a Xeon-D 1U pizza box or even an APU6 mPCIe slot are vastly inferior to what ODM purpose-built hardware like an U6-LR can do in terms of cost and performance.
> 
> Um... you can't "virtualize" WiFi in any VM I've ever seen.

You can though pass PCIe devices to a VM.  The hardware will physically
attach to the control host, but a VM will be able to do anything it wants
with it.

> > Problem is instead of the recommended 128MB memory, 16MB of storage
> > (https://openwrt.org/supported_devices/432_warning) the virtualization
> > examples (https://openwrt.org/docs/guide-user/virtualization/start) are
> > suggesting massively more memory.  256MB (small Xen example), 512MB
> > (VMware OpenWRT/15.05) or 1GB (Qemu examples).
> 
> Sorry, why is this a "problem"?
> 
> I spent $1100 on a Xeon-D box with 128GB of DRAM, 16 hyper threaded cores, and 2TB of NVMe.

If those numbers are to be believed (which is now suspect), it means a
*requirement* to devote that much to network operations.  Not being a
requirement means one could use the memory for other things.  Or I could
allow more than that to do extra complicated network things.

> > I don't yet have a full handle on the issues.  My first thought was the
> > large sizes come from early stage development and not wanting to bother
> > with memory limitations.  Another possibility is this comes from simply a
> > thought pattern of "x86 memory is cheap, just throw memory at it".
> 
> You're looking at it backwards.
> 
> Think of it as "what could I do if I have more RAM and CPU?"

Rather a lot of things completely unrelated to the things you listed.
Having lower estimates provides better reassurance I can do the things
*I* want to do.  (say, use more memory to spend less time on builds)


> > One issue I've found is the kernel configurations seem ill-suited to x86.
> > Almost any storage driver used by 10 or more people is built into the
> > kernel.  As a result the *single* kernel is huge.
> 
> If it's not used as a boot device, we could make it kmod-able... otherwise we'd need to add initramfs...  I don't think anyone wants to go down that road.  Too easy to brick devices.
> 
> I think we should leverage more subtargets and profiles, but that's a separate discussion.

This wraps back to my original issue.  x86 has some differences and they
haven't been adapted to.

x86 is easier to recover, so an initramfs is quite viable, perhaps x86
should be the exception and have one.  Alternatively, indeed more
targets.

Perhaps "x86" and "x86vm"?

> > If one was to go this direction, I suppose there might be "giant" or
> > "desktop" build.  Each hypervisor could use a target, include "hardware"
> > guaranteed to be present.  Then build all network drivers as modules (so
> > any device can be passed-in).
> 
> The number of interfaces supported by virtualization (at least in KVM) are quite limited (e1000/igbe, ixgbe, rtl8139, and virtio) so I don't see this as much of a problem.

The number of interface types supported by KVM is quite limited.  The
number of interface types supported by Xen is quite limited.  I suspect
the list for Hyper-V and VMware are similarly limited.

Yet, each of these sets is disjoint from the others.  Hyper-V's
interfaces add ~1MB to the kernel.  One of VMware's interfaces adds
~350KB to the kernel.  If that happens to be your hypervisor, you
urgently need that interface, but if that isn't your hypervisor that is
wasted memory for which ECC is inappropriate.


> > Examples of things which don't seem to fit well are CONFIG_AGP and
> > CONFIG_DRM.  I would expect those for a desktop Linux distribution due
> > to GUI.  For OpenWRT which tends towards networking/embedded those seem
> > out of place.  CONFIG_FB is similar, though some devices do have actual
> > limited frame-buffers.
> 
> There are many mini-ITX boards suitable for networking (based on ATOM or Via C7 chips) that have VGA hardware and are adequately supported by the FB drivers doing console emulation...  Especially since during prototyping, a crash or panic might have time to write helpful messages to video memory, but not be able to write data out a relatively slow serial port before everything goes sideways.

I'll take the serial port.  Much more readily captured and if things
go sufficiently South, you'll lose graphics output anyway.

> > Another item is the goal of trying to self-host.  Being able to self-host
> > is a worthy goal, but that has very distinct needs from an embedded
> > networking device.
> 
> "Self-host"?  Meaning what in this case?  You want to build your images on the same box that you run your firewall?

There is a desire to build OpenWRT on OpenWRT devices.  I'm not in this
boat, but the desire has been expressed.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg at m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445