Objective of OpenWRT/x86?

Sun Apr 30 21:40:40 PDT 2023

> On Apr 28, 2023, at 11:18 PM, Elliott Mitchell <ehem+openwrt at m5p.com> wrote:
> 
> On Fri, Apr 28, 2023 at 12:04:15PM -0600, Philip Prindeville wrote:
>> 
>>> [snip]
> 
>> See above: the radios and antennae I can get as add-ons for a Xeon-D 1U pizza box or even an APU6 mPCIe slot are vastly inferior to what ODM purpose-built hardware like an U6-LR can do in terms of cost and performance.
>> 
>> Um... you can't "virtualize" WiFi in any VM I've ever seen.
> 
> You can though pass PCIe devices to a VM.  The hardware will physically
> attach to the control host, but a VM will be able to do anything it wants
> with it.

So the guest has the potential to crash or hang the host?

> 
>>> Problem is instead of the recommended 128MB memory, 16MB of storage
>>> (https://openwrt.org/supported_devices/432_warning) the virtualization
>>> examples (https://openwrt.org/docs/guide-user/virtualization/start) are
>>> suggesting massively more memory.  256MB (small Xen example), 512MB
>>> (VMware OpenWRT/15.05) or 1GB (Qemu examples).
>> 
>> Sorry, why is this a "problem"?
>> 
>> I spent $1100 on a Xeon-D box with 128GB of DRAM, 16 hyper threaded cores, and 2TB of NVMe.
> 
> If those numbers are to be believed (which is now suspect), it means a
> *requirement* to devote that much to network operations.  Not being a
> requirement means one could use the memory for other things.  Or I could
> allow more than that to do extra complicated network things.

Which part is a lie?

[philipp at kvm1 ~]$ sudo dmidecode
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
39 structures occupying 2183 bytes.
Table at 0x000EB000.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: American Megatrends Inc.
Version: 1.1c
Release Date: 10/03/2016
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 16 MB
Characteristics:
PCI is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
ACPI is supported
USB legacy is supported
BIOS boot specification is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 5.6

Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Supermicro
Product Name: Super Server
Version: 0123456789
Serial Number: 0123456789
UUID: 00000000-0000-0000-0000-0cc47a971b8e
Wake-up Type: Power Switch
SKU Number: To be filled by O.E.M.
Family: To be filled by O.E.M.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: Supermicro
Product Name: X10SDV-8C-TLN4F
Version: 2.00
Serial Number: ZM16AS037277
Asset Tag: To be filled by O.E.M.
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: To be filled by O.E.M.
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0

Handle 0x0003, DMI type 3, 25 bytes
Chassis Information
Manufacturer: Supermicro
Type: Main Server Chassis
Lock: Not Present
Version: 0123456789
Serial Number: 0123456789
Asset Tag: To Be Filled By O.E.M.
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: None
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: 1
Contained Elements: 1
<OUT OF SPEC> (0)
SKU Number: To be filled by O.E.M.

...

Handle 0x0019, DMI type 16, 23 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: Multi-bit ECC
Maximum Capacity: 128 GB
Error Information Handle: Not Provided
Number Of Devices: 4

Handle 0x001A, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x010000003FF
Range Size: 64 GB
Physical Array Handle: 0x0019
Partition Width: 2

Handle 0x001B, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x0019
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: DIMM
Set: None
Locator: DIMMA1
Bank Locator: P0_Node0_Channel0_Dimm0
Type: DDR4
Type Detail: Synchronous
Speed: 2400 MT/s
Manufacturer: Undefined
Serial Number: F019B10C
Asset Tag: (Date:16/50)
Part Number: 9965640-006.A01G    Rank: 2
Configured Memory Speed: 2133 MT/s
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: 0.003 V

...

Handle 0x001D, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x0019
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: DIMM
Set: None
Locator: DIMMA2
Bank Locator: P0_Node0_Channel0_Dimm1
Type: DDR4
Type Detail: Synchronous

Handle 0x001E, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x0019
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: DIMM
Set: None
Locator: DIMMB1
Bank Locator: P0_Node0_Channel1_Dimm0
Type: DDR4
Type Detail: Synchronous
Speed: 2400 MT/s
Manufacturer: Undefined
Serial Number: ED19640C
Asset Tag: (Date:16/50)
Part Number: 9965640-006.A01G    Rank: 2
Configured Memory Speed: 2133 MT/s
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: 0.003 V

...

Handle 0x0025, DMI type 4, 42 bytes
Processor Information
Socket Designation: CPU1
Type: Central Processor
Family: Xeon
Manufacturer: Intel
ID: 63 06 05 00 FF FB EB BF
Signature: Type 0, Family 6, Model 86, Stepping 3
Flags:
FPU (Floating-point unit on-chip)
VME (Virtual mode extension)
DE (Debugging extension)
PSE (Page size extension)
TSC (Time stamp counter)
MSR (Model specific registers)
PAE (Physical address extension)
MCE (Machine check exception)
CX8 (CMPXCHG8 instruction supported)
APIC (On-chip APIC hardware supported)
SEP (Fast system call)
MTRR (Memory type range registers)
PGE (Page global enable)
MCA (Machine check architecture)
CMOV (Conditional move instruction supported)
PAT (Page attribute table)
PSE-36 (36-bit page size extension)
CLFSH (CLFLUSH instruction supported)
DS (Debug store)
ACPI (ACPI supported)
MMX (MMX technology supported)
FXSR (FXSAVE and FXSTOR instructions supported)
SSE (Streaming SIMD extensions)
SSE2 (Streaming SIMD extensions 2)
SS (Self-snoop)
HTT (Multi-threading)
TM (Thermal monitor supported)
PBE (Pending break enabled)
Version: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
Voltage: 1.8 V
External Clock: 100 MHz
Max Speed: 4000 MHz
Current Speed: 2100 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: 0x0022
L2 Cache Handle: 0x0023
L3 Cache Handle: 0x0024
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 8
Core Enabled: 8
Thread Count: 16
Characteristics:
64-bit capable
Multi-Core
Hardware Thread
Execute Protection
Enhanced Virtualization
Power/Performance Control

>>> I don't yet have a full handle on the issues.  My first thought was the
>>> large sizes come from early stage development and not wanting to bother
>>> with memory limitations.  Another possibility is this comes from simply a
>>> thought pattern of "x86 memory is cheap, just throw memory at it".
>> 
>> You're looking at it backwards.
>> 
>> Think of it as "what could I do if I have more RAM and CPU?"
> 
> Rather a lot of things completely unrelated to the things you listed.
> Having lower estimates provides better reassurance I can do the things
> *I* want to do.  (say, use more memory to spend less time on builds)

I was giving examples of things that benefited from more CPU and DRAM.  It wasn't an exhaustive list.

> 
>>> One issue I've found is the kernel configurations seem ill-suited to x86.
>>> Almost any storage driver used by 10 or more people is built into the
>>> kernel.  As a result the *single* kernel is huge.
>> 
>> If it's not used as a boot device, we could make it kmod-able... otherwise we'd need to add initramfs...  I don't think anyone wants to go down that road.  Too easy to brick devices.
>> 
>> I think we should leverage more subtargets and profiles, but that's a separate discussion.
> 
> This wraps back to my original issue.  x86 has some differences and they
> haven't been adapted to.
> 
> x86 is easier to recover, so an initramfs is quite viable, perhaps x86
> should be the exception and have one.  Alternatively, indeed more
> targets.
> 
> Perhaps "x86" and "x86vm"?

There were sound reasons for avoiding initramfs.

> 
>>> If one was to go this direction, I suppose there might be "giant" or
>>> "desktop" build.  Each hypervisor could use a target, include "hardware"
>>> guaranteed to be present.  Then build all network drivers as modules (so
>>> any device can be passed-in).
>> 
>> The number of interfaces supported by virtualization (at least in KVM) are quite limited (e1000/igbe, ixgbe, rtl8139, and virtio) so I don't see this as much of a problem.
> 
> The number of interface types supported by KVM is quite limited.  The
> number of interface types supported by Xen is quite limited.  I suspect
> the list for Hyper-V and VMware are similarly limited.
> 
> Yet, each of these sets is disjoint from the others.  Hyper-V's
> interfaces add ~1MB to the kernel.  One of VMware's interfaces adds
> ~350KB to the kernel.  If that happens to be your hypervisor, you
> urgently need that interface, but if that isn't your hypervisor that is
> wasted memory for which ECC is inappropriate.

Do we even know if *all* of these hypervisors are in use?

I wouldn't pay for VMware when I can get KVM for free... and it has better support.

>>> Examples of things which don't seem to fit well are CONFIG_AGP and
>>> CONFIG_DRM.  I would expect those for a desktop Linux distribution due
>>> to GUI.  For OpenWRT which tends towards networking/embedded those seem
>>> out of place.  CONFIG_FB is similar, though some devices do have actual
>>> limited frame-buffers.
>> 
>> There are many mini-ITX boards suitable for networking (based on ATOM or Via C7 chips) that have VGA hardware and are adequately supported by the FB drivers doing console emulation...  Especially since during prototyping, a crash or panic might have time to write helpful messages to video memory, but not be able to write data out a relatively slow serial port before everything goes sideways.
> 
> I'll take the serial port.  Much more readily captured and if things
> go sufficiently South, you'll lose graphics output anyway.

It takes a few hundred cycles to flush a buffer to VRAM before dying.  It takes 3-4 orders of magnitude longer to do the same at 115.2kb/s...  You might not have that long before the box is completely locked up.

> 
>>> Another item is the goal of trying to self-host.  Being able to self-host
>>> is a worthy goal, but that has very distinct needs from an embedded
>>> networking device.
>> 
>> "Self-host"?  Meaning what in this case?  You want to build your images on the same box that you run your firewall?
> 
> There is a desire to build OpenWRT on OpenWRT devices.  I'm not in this
> boat, but the desire has been expressed.

And... it feels like we're going down too many garden paths of what could be done.

What tangible needs to be done that hasn't, which will benefit a large number of people?

-Philip