[OpenWrt-Devel] "UCI overlay" proposal

[David Lang]

changing the format doesn't solve the problem, it just causes churn in all the 
software.

The problem is the lack of standardization and the fact that the way things are 
configured changes over time. Not everything is configured in UCI (and this will 
always be the case because it's not worth trying to change every piece of 
software available)

When you try to design a "standardized config space" that will work in the 
future for equipment and software that nobody has imagined yet, you are either 
going to go crazy-complicated to try and cover everything and have people going 
cross-eyed trying to understand the spec, or you are going to lock things down 
to be so ridged that new things will be working around your config spec. The 
best that you can do is to setup a framework to keep the configs from stepping 
on each other and try to converge similar software to use the same terms in the 
config where it makes sense (remember, the newcomer may have a better way of 
looking at the problem, so trying to force it to define it's config the way the 
legacy things did it can cause big problems)

I'm all for tools to convert the format to whatever you want to use, and (as I 
posted earlier), to have tools to assemble/generate the configs, but requiring 
changes to every piece of software to just change the format requires more than 
"every langauage can understand JSON"

David Lang

On Fri, 6 Mar 2015, Zefir Kurtisi wrote:

> Hello Matthias,
>
> can't provide qualified feedback to your proposal, but at least want to post my
> experience and hope that it is valuable for the discussion.
>
> Our company develops highly specialized WLAN products for decades, for a long time
> based on OpenWRT. The niche market we are in has a strict requirement for
> long-term availability and compatibility. In the context of configuration, in
> addition to the default Web-UI configurability, we must support SNMP and stay
> compatible with MIBs we initially released like 10 years ago.
>
> Consequently, for a long time we are following an approach similar to what you are
> proposing. In the beginning, the config overlay was a simple mapper between
> backend (uci) and frontend (snmp). Later, when config space and complexity grew,
> we defined and used a 'meta-config-space' that was generated at runtime from uci
> to simplify the mapping to snmp.
>
> Alas, what looked like a reasonable and clever approach, over time turned into
> maintenance-hell. We were faced with two moving targets (uci config gradually
> changing, MIB extensions) that need continuous synchronization and render the
> approach error-prone and resource-demanding. From that experience, I tend to
> believe that an overlay- or meta-configuration is only practicable as interim
> solution during a transition towards a standardized config space.
>
> If I had a chance to go back and do things better, this would be my approach:
>
> 1) define standardized config space
> The hardest of all parts - config space must be generic and complete enough to
> cover each and every potential setup, while at the same time it has to be
> extensible, backwards-compatible, and well defined. Whoever is familiar with
> standardization processes knows this will be no picnic. On the good side, with
> ubus gradually becoming the core framework for OpenWRT, the format for the new
> config to be JSON is more or less given. As for the structure, in [1] an
> announcement for a generic network configuration standard in JSON was posted that
> might well serve as starting point.
> 2) implement interim overlay
> Once the config space in 1) is standardized, implement an temporary overlay that
> provides transparent access to the configuration via legacy uci and new JSON
> config spaces. From that point on, new applications / packages must not use uci
> any more.
> 3) gradually move to new config space
> Over a defined (and relatively short) transition period, existing packages has to
> be converted to use the JSON config. At the end of that period, uci should not be
> used any more.
>
>
> I have developed a few private tools for the overlay-config approach (like
> conversion from uci to JSON), which are not suitable for upstreaming (C++,
> proprietary dependencies). From there, my estimate for the implementation workload
> for the transition (after the config space was standardized) is around 3 man-months.
>
>
> Good Luck,
> Zefir
>
>
> [1] https://lists.openwrt.org/pipermail/openwrt-devel/2015-January/030422.html
>
>
> On 03/05/2015 01:36 PM, Matthias Schiffer wrote:
>> Hi,
>> during the development of our Freifunk firmware framework "Gluon" we've
>> come to the conclusion that the current UCI configuration format using
>> static files doesn't always fit our needs. Therefore we propose a new
>> feature we've called "UCI overlays".
>>
>> The basic idea is that we'd like to provide UCI configuration by other
>> means than the static files in /etc/config, for example from a database
>> or generated by scripts on the fly (the latter being our main usecase -
>> we'd like to generate configuration for netifd and fw3 based on
>> meta-configuration data). This should work transparently, without
>> needing changes in the config consumers (applications).
>>
>> The overlay-provided configuration packages should be merged with the
>> config read from /etc/config. We'd like to generate both config sections
>> which may be overruled by corresponding options in /etc/config, and
>> read-only sections which can't be changed by the user through UCI.
>>
>> We see two different ways to implement this:
>>
>> (1) Make the "overlay" an alternative backend which uses the "file"
>> backend to merge generated config with the one from files
>>
>> (2) Introduce overlays as a new concept in libuci
>>
>>
>> While the first option would need less changes in libuci, it doesn't
>> allow stacking different overlays, so we're in favour of option 2.
>>
>> Both ways would need a config file (/etc/uci.conf?) to configure the
>> overlays being used. Our plan is to implement overlays as dlopen-able
>> plugins loaded from somewhere like /lib/uci/overlay so it is possible
>> for other packages to provide overlay implementations.
>>
>> In addition, we'd like to add a new attribute "readonly" to the
>> uci_element structure so changing read-only configuration will fail as
>> soon as someone tries to uci_set it.
>>
>> Does this sound reasonable? We can develop the needed extensions
>> ourselves, but of course we'd like to get this feature upstream to avoid
>> carrying the patches downstream indefinitly, so we're eager to know what
>> you think of this proposal.
>>
>> Regards,
>> Matthias
>>
>>
>>
>> _______________________________________________
>> openwrt-devel mailing list
>> openwrt-devel at lists.openwrt.org
>> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>>
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel at lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel