[OpenWrt-Devel] Running k3s on OpenWrt

[Yousong Zhou]

On Fri, 29 Nov 2019 at 00:32, Paul Spooren <mail at aparcar.org> wrote:
>
> Hi Yousong,
>
> > A few months ago, I started working on bringing up k3s on OpenWrt.  I
> > managed to have a small k8s cluster with nodes running centos and
> > openwrt.  Likely other linux distros will also do, but I didn't test
> > combination of different architectures yet.
> Cool! That looks like an interesting project to make OpenWrt even more
> attractive for "the cloud".
> > K3s is on its own a k8s distribution.  It provides a single packaged
> > binary for amd64, arm64, armhf.  If I understand it right, the binary
> > is supposed to be as userland-dependency-free as possible, with
> > dependencies only on the kernel.
> >
> > One thing worth noting is that the openwrt package is NOT official,
> > neither from OpenWrt nor k3s upstream project.  It packages binary
> > executables distributed by k3s upstream project which is generally
> > frowned upon if sources are available.
> It looks like Go code, have you tried to compile it via the build
> system? There are other packages like syncthing which already make use
> of it, maybe you can use it as an inspiration. Overall I'd like to see
> this package as part of packages.git and maybe can help out if you're
> stumbling.

By being a distribution, k3s really means it, literally.   It packs up
a collection of static linked binaries built from buildroot.  That
part is also open-sourced but my impression is that the automation
level is not very high.

Even if we managed to patch the k3s source to use just whatever is
available in the host system, or building those exact binaries from
source code in the OpenWrt build system, we will still need to
establish a level of confidence that the result artifacts as a whole
can pass k8s e2e test ourselves.  This is a lot of effort and time.

> > The other thing is that k3s (or k8s) requires quite a few kernel
> > features to run, some of them only available as builtin (compared to
> > kmod).  This means you will likely to build your own image to use it.
> > The required features as of k3s 1.0.0 are recorded as well and
> > available as a default off config symbol CONFIG_k3s_KERNEL [2]
>
> I know nothing about sunix, but the other three targets appear mostly
> space "indipendent". Maybe these could be enabled by default? I think
> pretty much the same kernel options are required for LXD, docker and
> related things.
>
> x86_64, aarch64, sunxi, armvirt
>
>      select KERNEL_CGROUPS
>      select KERNEL_KEYS
>      select KERNEL_MEMCG
>      select KERNEL_NAMESPACES
>      select KERNEL_SECCOMP
>      select KERNEL_SECCOMP_FILTER
>      select KERNEL_X86_VSYSCALL_EMULATION if x86_64
>
> >   [1] https://github.com/yousong/waller/tree/master/k3s
> >   [2] https://github.com/yousong/waller/blob/c3ee00451ce12e869277bbd73b08652882c52010/k3s/Makefile#L69-L79
>
> Best,
> Paul
>

Personally I prefer we keep them small and slim by default.  On this
regard, that and being highly-customizable are two things I think make
OpenWrt as a linux distribution stand out.

Regards,
               yousong

_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel