Standard meta JSON files available at downloads.openwrt.org [Was: Re: Firmware Selector Setup]
Paul Spooren
mail at aparcar.org
Fri Sep 11 16:55:43 EDT 2020
Hi team,
On 20.08.20 05:59, Moritz Warning wrote:
> Ack.
>
> On 8/20/20 2:11 PM, Rich Brown wrote:
>> I haven't weighed in here, except to be a cheerleader. I really want to see this happen.
>>
>> This discussion feels dangerously close to premature optimization: we don't know how many people will use it; we don't know where the pain points would be; we don't even know that it really works (although early versions seemed fine.)
>>
>> Given that we control both ends of the product (web GUI and the backend JSON files), we can make adjustments "on the fly" (say, over the course of a week, if we're paying attention - and I expect we will be.
>>
> My view is based on the idea that I do not control the json generation in openwrt and I do not even know what format is ideal for the firmware selector. :)
In previous discussions with you we already agreed on keeping things
small, I'd therefore suggest to have an index containing profiles of all
targets and single files containing profile specific information. This
results in 2+ request for the firmware wizard but also allows routers do
download small files containing sysupgrade information.
So we could add `profiles.json` to to the version root (e.g. snapshots/
or releases/20.12.0/) containing a dict with profile names (e.g.
meraki_mr24) as keys and target (e.g. apm821xx/nand) + titles (e.g.
Cisco Meraki MX60/MX60W).
head mini.json
{
"profiles": {
"7links_px-4885-8m": {
"titles": [{
"model": "PX-4885",
"variant": "8M",
"vendor": "7Links"
}],
"target": "ramips/rt305x"
},
The resulting file would is currently 108Kb (gziped 17Kb). Now the
firmware wizard can use the information to either build a search based
interface[1] or a vendor based interface (lineage)[2] or a dropdown menu
appraoch[3].
On selecting a profile, the profile specific information should be
loaded, containing information about the created images, default
packages, etc. The files could be stored in $VERSION_ROOT/json or even
hidden via ".json" to avoid "bloating" the directories.
Each device profile file is only a few bytes in size, however there
exists 1018 (and counting) of them.
cat 7links_px-4885-8m.json
{
"device_packages": ["kmod-usb-dwc2", "kmod-usb2", "kmod-usb-ohci",
"kmod-usb-ledtrig-usbport", "kmod-leds-gpio"],
"image_prefix": "openwrt-ramips-rt305x-7links_px-4885-8m",
"images": [{
"name":
"openwrt-ramips-rt305x-7links_px-4885-8m-squashfs-sysupgrade.bin",
"sha256":
"387ce1ca7ec3a9c968a39a1927f6a9b85fb893d1f88e6ba8f1ba51381c68e176",
"type": "sysupgrade"
}],
"supported_devices": ["7links,px-4885-8m", "px-4885-8M"],
"target": "ramips/rt305x",
"titles": [{
"model": "PX-4885",
"variant": "8M",
"vendor": "7Links"
}]
}
This should cover the firmware wizard. From my understanding fast with
the small index, allowing multiple structures as the titles are split
into vendor/model/variant and still containing all available information
with the "on demand" loading.
Optional bonus:
We could use `dump-targets-info.pl` to create list or targets and it's
architectures.
head targets.json
{
"apm821xx/nand": {
"arch": "powerpc_464fp"
},
"apm821xx/sata": {
"arch": "powerpc_464fp"
},
"arc770/generic": {
"arch": "arc_arc700"
},
We could offer a supported.json file containing the `supported_devices`
metadata information mapping to the profile name. This would allow
running routers to check the content of /tmp/sysinfo/board_name and see
what firmware image could be installed. This file is currently 51Kb
(gziped 10Kb).
head supported.json
{
"7links,px-4885-8m": "7links_px-4885-8m",
"px-4885-8M": "7links_px-4885-8m",
"8dev,carambola2": "8dev_carambola2",
"carambola2": "8dev_carambola2",
"8dev,habanero-dvk": "8dev_habanero-dvk",
"8dev,jalapeno": "8dev_jalapeno",
"8dev,lima": "8dev_lima",
"lima": "8dev_lima",
"8dev,rambutan": "8dev_rambutan",
We could offer a version.json file that allows both firmware wizards to
show all available releases without manual modifications[4] and also
running devices to check for upgrades and e.g. show notification within
LuCI. This was previously discussed on this mailing list, sadly the
archive misses these mails.
However a possible implementation could be something like the following,
which is a basically a JSONified OpenWrt copy of Alpines releases table[5].
{
"metadata_version": 1,
"branches": [{
"branch": "openwrt-19.07",
"release_date": "2020-01-31",
"latest": "19.07.1",
"minor": [
"19.07.0"
],
"updates": "bugs",
"eol": "2021-01-31"
},
{
"branch": "openwrt-18.06",
"release_date": "2020-01-31",
"latest": "18.06.7",
"minor": [
"18.06.6",
"18.06.5",
"18.06.4",
"18.06.3",
"18.06.2",
"18.06.1",
"18.06.0"
],
"updates": "security",
"eol": "2020-08-01"
},
{
"branch": "lede-17.01",
"release_date": "2018-01-01",
"latest": "17.01.2",
"minor": [
"17.1.1",
"17.1.0"
],
"updates": "none",
"eol": "2019-06-01"
},
{
"branch": "master",
"path": "snapshots",
"updates": "dev"
}
}
}
I'm happy to build reliable Python scripts to create all those files but
would like some feedback first. Ideally we get the firmware wizard going
first and think about on-device update notification a bit later.
Sunshine,
Paul
[1]: https://mwarning.github.io/yet-another-firmware-selector/www/
[2]: https://download.lineageos.org/
[3]: https://firmware.darmstadt.freifunk.net/
[4]:
https://github.com/mwarning/yet-another-firmware-selector/blob/master/www/config.js#L9
[5]: https://wiki.alpinelinux.org/wiki/Alpine_Linux:Releases
>
>> My counsel would be to deploy the existing implementation on a test page. The folks on the list can beat the tar out of it first, then mention it on the Forum when we think it's ready.
>>
>> If we get feedback that it's too slow, or that it doesn't do quite the right thing, or something else, we will know exactly what needs to be adjusted. In the meantime, we will have added a wonderful new feature to OpenWrt.
>>
>> My two cents. Thanks.
>>
>> Rich
>>
>>> On Aug 20, 2020, at 5:18 AM, Moritz Warning <moritzwarning at web.de> wrote:
>>>
>>> On 8/20/20 10:15 AM, Petr Štetiar wrote:
>>>> Moritz Warning <moritzwarning at web.de> [2020-08-19 17:17:05]:
>>>>
>>>>>> * are we going to generate separate JSON for every application out there?
>>>>>> - I mean, wouldn't it make sense to generate something generic, something like:
>>>>>>
>>>>>> meta/json/targets-full-v1.json (complete amalgamation of profiles.json)
>>>>>> meta/json/targets-minimal-v1.json (just the bare minimum)
>>>>>> meta/json/targets-images-v1.json (just the images)
>>>>> Do you argue that OpenWrt should generate these files?
>>>> Yes, instead of having to generate and maintain several different JSON formats
>>>> for different applications, lets think about it and try to provide some subset
>>>> which might cover most of the possible usecases.
>>> I am more cautious here.
>>>
>>>>> I argue that OpenWrt should generate the raw data (albeit machine readable)
>>>>> as json files.
>>>> It does, but as you can see, it's barely usable as people still need to scrape
>>>> all targets directories in snapshot/releases for `profile.json` instead of
>>>> getting complete `targets-full-v1.json` directly in one HTTP call.
>>> A combined profiles.json would make that certainly easier.
>>>
>>> But even then, at least for a firmware selector, a list of available releases need to be scraped (well, or written by hand).
>>> Then the single combined profiles.json would need to be downloaded for each such release.
>>>
>>>> -- ynezz
>>>>
>
> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-adm
More information about the openwrt-adm
mailing list