Standard meta JSON files available at downloads.openwrt.org [Was: Re: Firmware Selector Setup]

Moritz Warning moritzwarning at web.de
Sat Sep 12 16:43:22 EDT 2020


Hi Paul,

thank you for your suggestions!

The current approach is a compromise between size and simplicity. The overview.json is currently ~310KB (~27KB gzip) for each release. [0]
If you want to include default packages and hashes for each device, then the file size is going to blow up (approx. 4x for the gzip file).

A pure index file and a file for each device sounds fine to me. But please let us not hold up the deployment of the firmware selector for this.

best,
Moritz :-)

[0] https://mwarning.github.io/yet-another-firmware-selector/www/

On 9/11/20 10:55 PM, Paul Spooren wrote:
> Hi team,
> 
> On 20.08.20 05:59, Moritz Warning wrote:
>> Ack.
>>
>> On 8/20/20 2:11 PM, Rich Brown wrote:
>>> I haven't weighed in here, except to be a cheerleader. I really want to see this happen.
>>>
>>> This discussion feels dangerously close to premature optimization: we don't know how many people will use it; we don't know where the pain points would be; we don't even know that it really works (although early versions seemed fine.)
>>>
>>> Given that we control both ends of the product (web GUI and the backend JSON files), we can make adjustments "on the fly" (say, over the course of a week, if we're paying attention - and I expect we will be.
>>>
>> My view is based on the idea that I do not control the json generation in openwrt and I do not even know what format is ideal for the firmware selector. :)
> 
> In previous discussions with you we already agreed on keeping things small, I'd therefore suggest to have an index containing profiles of all targets and single files containing profile specific information. This results in 2+ request for the firmware wizard but also allows routers do download small files containing sysupgrade information.
> 
> So we could add `profiles.json` to to the version root (e.g. snapshots/ or releases/20.12.0/) containing a dict with profile names (e.g. meraki_mr24)  as keys and target (e.g. apm821xx/nand) + titles (e.g. Cisco Meraki MX60/MX60W).
> 
> head mini.json
> {
>     "profiles": {
>         "7links_px-4885-8m": {
>             "titles": [{
>                 "model": "PX-4885",
>                 "variant": "8M",
>                 "vendor": "7Links"
>             }],
>             "target": "ramips/rt305x"
>         },
> 
> The resulting file would is currently 108Kb (gziped 17Kb). Now the firmware wizard can use the information to either build a search based interface[1] or a vendor based interface (lineage)[2] or a dropdown menu appraoch[3].
> 
> On selecting a profile, the profile specific information should be loaded, containing information about the created images, default packages, etc. The files could be stored in $VERSION_ROOT/json or even hidden via ".json" to avoid "bloating" the directories.
> 
> Each device profile file is only a few bytes in size, however there exists 1018 (and counting) of them.
> 
> cat 7links_px-4885-8m.json
> {
>     "device_packages": ["kmod-usb-dwc2", "kmod-usb2", "kmod-usb-ohci", "kmod-usb-ledtrig-usbport", "kmod-leds-gpio"],
>     "image_prefix": "openwrt-ramips-rt305x-7links_px-4885-8m",
>     "images": [{
>         "name": "openwrt-ramips-rt305x-7links_px-4885-8m-squashfs-sysupgrade.bin",
>         "sha256": "387ce1ca7ec3a9c968a39a1927f6a9b85fb893d1f88e6ba8f1ba51381c68e176",
>         "type": "sysupgrade"
>     }],
>     "supported_devices": ["7links,px-4885-8m", "px-4885-8M"],
>     "target": "ramips/rt305x",
>     "titles": [{
>         "model": "PX-4885",
>         "variant": "8M",
>         "vendor": "7Links"
>     }]
> }
> 
> This should cover the firmware wizard. From my understanding fast with the small index, allowing multiple structures as the titles are split into vendor/model/variant and still containing all available information with the "on demand" loading.
> 
> Optional bonus:
> 
> We could use `dump-targets-info.pl` to create list or targets and it's architectures.
> 
> head targets.json
> {
>     "apm821xx/nand": {
>         "arch": "powerpc_464fp"
>     },
>     "apm821xx/sata": {
>         "arch": "powerpc_464fp"
>     },
>     "arc770/generic": {
>         "arch": "arc_arc700"
>     },
> 
> We could offer a supported.json file containing the `supported_devices` metadata information mapping to the profile name. This would allow running routers to check the content of /tmp/sysinfo/board_name and see what firmware image could be installed. This file is currently 51Kb (gziped 10Kb).
> 
> head supported.json
> {
>     "7links,px-4885-8m": "7links_px-4885-8m",
>     "px-4885-8M": "7links_px-4885-8m",
>     "8dev,carambola2": "8dev_carambola2",
>     "carambola2": "8dev_carambola2",
>     "8dev,habanero-dvk": "8dev_habanero-dvk",
>     "8dev,jalapeno": "8dev_jalapeno",
>     "8dev,lima": "8dev_lima",
>     "lima": "8dev_lima",
>     "8dev,rambutan": "8dev_rambutan",
> 
> We could offer a version.json file that allows both firmware wizards to show all available releases without manual modifications[4] and also running devices to check for upgrades and e.g. show notification within LuCI. This was previously discussed on this mailing list, sadly the archive misses these mails.
> 
> However a possible implementation could be something like the following, which is a basically a JSONified OpenWrt copy of Alpines releases table[5].
> 
> 
> {
>         "metadata_version": 1,
>         "branches": [{
>                         "branch": "openwrt-19.07",
>                         "release_date": "2020-01-31",
>                         "latest": "19.07.1",
>                         "minor": [
>                                 "19.07.0"
>                         ],
>                         "updates": "bugs",
>                         "eol": "2021-01-31"
>                 },
>                 {
>                         "branch": "openwrt-18.06",
>                         "release_date": "2020-01-31",
>                         "latest": "18.06.7",
>                         "minor": [
>                                 "18.06.6",
>                                 "18.06.5",
>                                 "18.06.4",
>                                 "18.06.3",
>                                 "18.06.2",
>                                 "18.06.1",
>                                 "18.06.0"
>                         ],
>                         "updates": "security",
>                         "eol": "2020-08-01"
>                 },
>                 {
>                         "branch": "lede-17.01",
>                         "release_date": "2018-01-01",
>                         "latest": "17.01.2",
>                         "minor": [
>                                 "17.1.1",
>                                 "17.1.0"
>                         ],
>                         "updates": "none",
>                         "eol": "2019-06-01"
>                 },
>                 {
>                         "branch": "master",
>                         "path": "snapshots",
>                         "updates": "dev"
>                 }
>         }
> }
> 
> 
> I'm happy to build reliable Python scripts to create all those files but would like some feedback first. Ideally we get the firmware wizard going first and think about on-device update notification a bit later.
> 
> Sunshine,
> Paul
> 
> [1]: https://mwarning.github.io/yet-another-firmware-selector/www/
> [2]: https://download.lineageos.org/
> [3]: https://firmware.darmstadt.freifunk.net/
> [4]: https://github.com/mwarning/yet-another-firmware-selector/blob/master/www/config.js#L9
> [5]: https://wiki.alpinelinux.org/wiki/Alpine_Linux:Releases
> 
>>
>>> My counsel would be to deploy the existing implementation on a test page. The folks on the list can beat the tar out of it first, then mention it on the Forum when we think it's ready.
>>>
>>> If we get feedback that it's too slow, or that it doesn't do quite the right thing, or something else, we will know exactly what needs to be adjusted. In the meantime, we will have added a wonderful new feature to OpenWrt.
>>>
>>> My two cents. Thanks.
>>>
>>> Rich
>>>
>>>> On Aug 20, 2020, at 5:18 AM, Moritz Warning <moritzwarning at web.de> wrote:
>>>>
>>>> On 8/20/20 10:15 AM, Petr Štetiar wrote:
>>>>> Moritz Warning <moritzwarning at web.de> [2020-08-19 17:17:05]:
>>>>>
>>>>>>>    * are we going to generate separate JSON for every application out there?
>>>>>>>      - I mean, wouldn't it make sense to generate something generic, something like:
>>>>>>>
>>>>>>>        meta/json/targets-full-v1.json (complete amalgamation of profiles.json)
>>>>>>>        meta/json/targets-minimal-v1.json (just the bare minimum)
>>>>>>>        meta/json/targets-images-v1.json (just the images)
>>>>>> Do you argue that OpenWrt should generate these files?
>>>>> Yes, instead of having to generate and maintain several different JSON formats
>>>>> for different applications, lets think about it and try to provide some subset
>>>>> which might cover most of the possible usecases.
>>>> I am more cautious here.
>>>>
>>>>>> I argue that OpenWrt should generate the raw data (albeit machine readable)
>>>>>> as json files.
>>>>> It does, but as you can see, it's barely usable as people still need to scrape
>>>>> all targets directories in snapshot/releases for `profile.json` instead of
>>>>> getting complete `targets-full-v1.json` directly in one HTTP call.
>>>> A combined profiles.json would make that certainly easier.
>>>>
>>>> But even then, at least for a firmware selector, a list of available releases need to be scraped (well, or written by hand).
>>>> Then the single combined profiles.json would need to be downloaded for each such release.
>>>>
>>>>> -- ynezz
>>>>>
>>
>> _______________________________________________
>> openwrt-adm mailing list
>> openwrt-adm at lists.openwrt.org
>> https://lists.openwrt.org/mailman/listinfo/openwrt-adm



More information about the openwrt-adm mailing list