OpenWrt Next Generation Ideas

[Daniel Golle]

On Fri, Mar 31, 2023 at 03:52:47PM +0300, Arınç ÜNAL wrote:
> On 31.03.2023 14:33, Daniel Golle wrote:
> > Hi!
> > 
> > On Fri, Mar 31, 2023 at 12:44:12PM +0300, Arınç ÜNAL wrote:
> > > Hi all,
> > > 
> > > These are the ideas I've been thinking about for the future of OpenWrt for a
> > > while. It looks complete enough to share it with all of you.
> > > 
> > > I'm willing to put a great deal of effort to get as much out-of-tree patches
> > > on mainline Linux as possible.
> > > 
> > > You can make a comment on Notion or discuss it here, I'm wondering if the
> > > ideas are feasible and how well it would benefit the people maintaining
> > > OpenWrt.
> > > 
> > > https://arinc9.notion.site/OpenWrt-next-gen-ideas-6db745e7584b4823950291c96f2326bb
> > 
> > I will comment here, I don't have an account on Notion and it seems
> > to be required to be able to comment there.
> > 
> > > defconfig for each device instead of config for each (sub)target.
> > 
> > Given that we support thousands of devices this will not only increase
> > the time needed to build a release or snapshot by several magnitudes,
> > but also make debugging **much** harder. As of now, all devices of a
> > subtarget are using the same kernel, hence e.g. symbol offsets in a
> > kernel stack dump match for all of them. To reproduce or investigate
> > a problem it's hence enough to have similar hardware, not necessarily
> > the exact same board. As we are already lacking testers and maintainers
> > for the relatively small amount of targets/subtargets, have a build for
> > each board would make things much worse...
> > 
> > Per-device builds would also be an invitation to downstream users to
> > introduce device-specific (kernel-)hacks. If you want that, better go
> > for OpenEmbedded.
> > 
> > We can modularize things more or even have more sub-targets if it's
> > really needed to save space.
> > 
> > The disadvantages outweight the advantages imho when it comes to having
> > a complete kernel build for each device.
> 
> Hmm, what about we enable the bare minimum of kernel options for a target,
> which is already how it is, then select the rest as kernel modules (like on
> the makefile of a target for each device) on the defconfig for each device?
> So, in the end, it wouldn't be any different than selecting a kernel module
> package from the OpenWrt SDK which, I believe, does not change the symbol
> offsets in the kernel stack.
> 
> My reason for pushing for the use defconfigs is that anyone can build the
> Linux kernel for their device, without needing OpenWrt. So the work for
> adding support for a device would benefit far more people.

This is pretty much what we are currently doing.
The exception are network drivers to allow for failsafe mode to work
and provide SSH access **before** any modules are loaded.

> 
> > 
> > > Some Modest Virtualization Observations
> > 
> > How is this related? Virtualization (with OpenWrt being the guest)
> > matters on x86 which is usually not that space-constraint.
> > And maybe armvirt. If space is a problem for older x86 boards, let's
> > disable guest support in x86/legacy.
> 
> It was a broad example without any explanation. What caught my attention
> there is the configuration of the kernel that causes problems, if I
> understand correctly.
> 
> > 
> > > Contribute defconfigs and all the devicetrees on OpenWrt to Linux.
> > 
> > For devicetrees this would of course be desirable, but also implies a
> > lot of work and discussions. If you are up to get it started (ie. setup
> > a tree to collect cleaned-up and ready to submit dts), I think it would
> > be worth the effort, at least for more recent targets/SoCs.
> 
> I've been meaning to do this for the mt7621 SoC devices for months. The main
> roadblock is that some drivers are out-of-tree, like the NAND flash so it
> makes no sense to have them defined on the devicetree. Getting the
> out-of-tree patches on mainline Linux is another step so it'll happen
> eventually.

Hm, I thought that Weijie had sent the mt7621-nand driver also upstream,
but I haven't been following the process...

> 
> I'll get this started with my Linux fork on GitHub.

Very nice, I will join in there.

> 
> > 
> > Regarding defconfigs I don't think we need an individual defconfig for
> > each board. The problem here is also that OpenWrt currently has a layered
> > approach (generic->target->subtarget) approach while Linux itself has
> > a flat approach, and using that would result in a lot of duplication,
> > which would in turn make keeping all those defconfigs up-to-date quite
> > a lot of work...
> 
> Not really, once you make a defconfig for a device, the only case for it to
> change in the future is if a kernel option was renamed, which is very rare.
> Anyway, even in that case, there are known ways to update them by bulk.
> 
> https://lore.kernel.org/kernel-janitors/20220929090645.1389-1-lukas.bulwahn@gmail.com/

I agree for "established" platforms for which drivers are already
upstream and not many changes are still expected to happen.
Yet, a defconfig **for each board** seems like overkill to me, esp.
because 90%+ will not be able to boot the resulting image generated
by a vanilla kernel build in any meaningful way.
Having defconfig for each sub-target would be agreeable, though I must
say that I do like our layerd approach to defconfig.
Maybe we should pick single representative **and resourceful** boards
and try upstreaming defconfig for those (in case it didn't already happen).
However, not using the generic defconfig anyway is only relevant for
rather resource-constraint boards (as otherwise you'd just use, let's say
multi_v7_defconfig or the like). Are you sure upstream is interested in
being flooded with hundreds of defconfigs, most of them for very similar
devices?

> 
> > 
> > > Either submit all out-of-tree patches on OpenWrt to Linux or get rid
> > > of them and find a better solution for what the unacceptable patch
> > > does.
> > 
> > This would of course be great, but especially for legacy devices it may
> > not be possible in many cases. Think of all the devices stuck on
> > swconfig, just to name one example... Think of all the completely
> > broken vendor bootloaders which require hacks (mangeling kernel cmdline
> > and such) and cannot easily be replaced...
> 
> Those can stay until eventually the support for them will be dropped on
> newer OpenWrt versions. I believe there are a lot of out-of-tree patches
> that are not for old devices and can be on mainline Linux.

In terms of hardware support you are probably right.
When it comes to vendor bootloaders which require mangeling kernel
bootargs the kernel folks have always argued that users should just
change the bootloader environment -- however, this requires a serial
console attached and makes reverting to stock firmware harder.

Another example is selection of the rootfs. Kernel folks argue that
we should use an initramfs for that, however, we try to avoid the
overhead of using an initramfs with it's own userland just for that.

To resolve this situation at least for future boards (ie. with
post-2020 versions of U-Boot) I've put quite some effort into adding
native support for mapping the filesystem sub-image(s) of a uImage.FIT
image as a block device in Linux:

https://github.com/dangowrt/linux/commit/8a70f52fddd518d1d7093a12874ca3a5de86fab1

I've already had quite a lot of debate about that, currently I'm waiting
for add/remove/size-change notifications for in-kernel block device users
being added to the kernel, as that would be required for a really clean
implementation.

Most other patches in target/linux/generic/pending-5.* should be sutiable
for upstream inclusion. Patches in target/linux/generic/hack-5.* are more
tragic cases...

Apart from the technical debate the issue here is of course also of
organizational nature. While not accepting new patches unless they are
already accepted upstream is a matter of policy, upstreaming all the
existing patches is a lot of work, and the question arises *who* is
going to volunteer do that...

> 
> > 
> > > Bugfix backporting should happen only after it's accepted to Linux.
> > > The patch must be identical to the commit on Linux.
> > 
> > The wording here might be a bit too strict to support our existing
> > mess, but I generally agree. So I'd say 'should' instead of 'must', but
> > otherwise agree.
> > 
> > > Feature backporting should be done only if it's thoroughly tested.
> > 
> > ... and testing often happens in the OpenWrt tree. So it's a bit of
> > a chicken-egg problem, as often developers don't even have all the
> > different hardware needed for testing. But generally I agree.
> > A way to ease testing *before* pushing to openwrt.git or posting to
> > upstream mailing lists would be to have snapshot builds also for
> > developers' staging trees.
> > 
> > > Kernel Solution
> > > Make a mode menu.
> > > Filesystem only.
> > 
> > So which kernel headers should be used to build e.g. libc and netlink
> > users?
> > In a way it is also currently possible to build generic images for
> > most architectures using the armvirt, malta and x86 targets. Of course,
> > also in this case a kernel is being built.
> > 
> > > Make a kernel selection menu where the user can choose to feed the
> > > kernel directory of their own or use the longterm one defined on the
> > > OpenWrt SDK. Add this as a suboption to the full image mode.
> > 
> > What about CONFIG_EXTERNAL_KERNEL_TREE and friends...?
> 
> I've never tried this but looks like it may need a bit of extra options.

No, I'm using this on a daily base to use linux-next and friends together
with OpenWrt userland. For some devices you may need to add some
downstream patches, but e.g. recent MediaTek stuff works out-of-the-box.

> No patches must be applied, kernel module packages must not be compiled,
> etc.

Kernel module packages work fine, just the version will be wrong in case
of an external kernel tree being used.