[OpenWrt-Devel] RFC: versions.json
Rich Brown
richb.hanover at gmail.com
Mon Mar 2 22:14:47 EST 2020
Thanks for all these good ideas. A summary of my thoughts:
- Use of JSON. Not really risky - there'll be JSON parsers for the foreseeable future. And sticking in a version field makes it possible to evolve. My primary concern is that we use a SINGLE straightforward format and create tooling that writes in that format. (I wouldn't want to have multiple programs that attempt to make sense of the directory structure of downloads.openwrt.org...)
- Backporting to 18.x. I am agnostic here.
- There are several tricky issues regarding upgrade policies. I am fairly certain the JSON file should encode the possible version numbers. I am agnostic on the policies - I almost always install Stable releases, so I don't know the pitfalls. Does the JSON file also need to incorporate upgrade policies as well? For example,
- Whether to offer to upgrade to xx.yy.z+1, or xx.yy+1.0?
- Whether to offer to upgrade to xx.yy.z+n, skipping interim (possibly crashing) releases?
- etc...
- The JSON file should definitely contain a URL to "about this upgrade" and perhaps also a changelog, etc.
- If a flawless upgrade experience required a more capable device, I am personally willing to give up on 4/32 devices (and maybe even 8/64 devices - although I'm sure I'll hear about their popularity)
- I like the idea of the progressive rollout, where a small number of devices get randomly selected to test new stable versions at the outset. This limits the damage and heartache from a bum update. After we get confidence in a particular release, we can open the floodgates.
- I like the idea of a callback that says "I got home safely, Mom". If there's a significant divergence between the number of auto-upgrade downloads and the successful upgrade messages, we can investigate/halt the upgrade process.
- re: performing the upgrade check on login. That check should never hinder/hang the login. That said, a person's attention to the router is still a good trigger for queueing a process to make the check. (Weekly/daily checks are fine, but it would be annoying if it checked seconds after I log in, and I never heard about it 'til the next time I looked...)
- And finally: what changes to the initial "pseudo-wiki-page" (below) would be necessary from what we've decided so far? Thanks.
Rich
<start wiki page>
====== OpenWrt Automatic Upgrades ======
OpenWrt can check for newer versions of its firmware and offer to upgrade automatically. These automatic upgrades restore the router to substantially the same state, except running newer firmware. In particular, the upgrade process preserves both the settings (implementing the "Keeps settings..." checkbox) and the hand-installed packages that aren't part of the standard image.
When this features is enabled, every LuCI web login or SSH login triggers a version check. If upgrades are available, the firmware displays a notice specifying the various options, with a link to a page that gives information about that particular upgrade, and the steps for applying that upgrade.
When "Upgrade" is initiated, the router downloads the new image along with all necessary packages. After all files have been retrieved successfully, a final human confirmation begins the upgrade process.
The web GUI indicates that the upgrade process has started, and periodically attempts to make connections back to the login page. The SSH interface gives a "Initiating upgrade to <version>" message before severing the SSH connection.
<end of wiki page>
> On Mar 2, 2020, at 5:43 AM, Paul Spooren <mail at aparcar.org> wrote:
>
> Hi,
>
>>>> A first step could be to establish a *versions.json* file at the root of
>>>> downloads.openwrt.org! The file would allow to check if a device still runs
>>>> the latest release. JSON seems common enough and is well supported by LuCIs
>>>> JavaScript implementation and also via jshn.sh on a CLI/script level.
>>> I'm wondering whether this JSON is really needed, wouldn't just some kind of
>>> unified symlink/directory structure would work as well? I mean, why to care
>>> about another JSON file content if the same could be achieved otherwise.
>> It's true that the actual representation (symlink/directory structure, JSON file, a massive text file listing every release, etc) doesn't matter. In some way, they're all "convertible" to the other forms.
>>
>> But we are creating an API here. People want to write software against that API, knowing they never will have to tweak their software because a new version or important branch, etc arises.
>>
>> What matters is that OpenWrt pick ONE FORMAT to be the canonical representation of this information, and that our build system automatically create data IN THAT ONE FORMAT as a matter of course, so that everyone can rely on it for their needs.
> It's might be a risky bet, but I could imagine JSON is around for the next years to come. It's also easier extendable than other formats using dicts. Something like `metadata_version: 1` could also be used to allow long-term compatibility.
>>> Do we need to care about archive releases?
>>> Older versions of the firmware (pre-20.xx) won't know how to interpret the API data, so we don't need to cater to those older images. However, in the future when Stable is 23.07.2, and you encounter a 20.07.1 machine, you'd like to it to show you the upgrade options based in its image.
>
> I think average device memory growth faster than our release file so we can risk storing these information, too. If we want to follow this "minor release by minor release" upgrade approach we should keep all versions, even archived ones.This way a device know "where" it is in the long chain of newer firmware releases.
>
> Ideally this would simply be a LuCI extension which could even be backport to 18.x...
>
>>> Do the choices below cover the waterfront for a complete set of useful names? (Let's assume that 19.07.2 has been released, and that there are two RCx builds: 19.07.3-rc1 and 20.07.0-rc2...)
> [...]
>> ...
>> release/endoflife -> 17.01.0
>> release/endoflife -> 15.05.
>> release/endoflife -> 14.07
>> ...
>> release/endoflife -> 0.9 (all the way back to White Russian...)
>
> I like the idea of having something like `latest-stable` pointing to the latest branch plus latest point release. Besides this feature, symlinks seem for me too limited to cover the use case I want to focus: The running device is not able to check if something newer is available.
>
> As Adrian mentioned earlier it could make sense to separate on the branches. This way a device can suggest a "safe" upgrade within the branch first and a "branch" upgrade only once EOL is near.
>
> Using the "branch" as key is problematic as the branch order is unclear. A dict doesn't have to be ordered (aka key order not stable).
>
> I checked Alpine and awkwardly they came up with the same question a version JSON file about a month ago[0]. However they have a great release overview[1] which inspired me for a JSON draft:
>
> {
> "metadata_version": 1,
> "branches": [{
> "branch": "openwrt-19.07",
> "release_date": "2020-01-31",
> "latest": "19.07.1",
> "minor": [
> "19.07.0"
> ],
> "updates": "bugs",
> "eol": "2021-01-31"
> },
> {
> "branch": "openwrt-18.06",
> "release_date": "2020-01-31",
> "latest": "18.06.7",
> "minor": [
> "18.06.6",
> "18.06.5",
> "18.06.4",
> "18.06.3",
> "18.06.2",
> "18.06.1",
> "18.06.0"
> ],
> "updates": "security",
> "eol": "2020-08-01"
> },
> {
> "branch": "lede-17.01",
> "release_date": "2018-01-01",
> "latest": "17.01.2",
> "minor": [
> "17.1.1",
> "17.1.0"
> ],
> "updates": "none",
> "eol": "2019-06-01"
> },
> {
> "branch": "master",
> "path": "snapshots",
> "updates": "dev"
> }
> }
> }
>
> So the update check performed on device would read "versions.json" and select the currently running branch. It can then go through the "minor" versions and select the latest or "one newer" - see below. If end of life (eol) is reached (or close) it offer an upgrade to the newer branch.
>
>>>> Update check script should look for the closest version found in the same
>>>> channel. So a *stable* 19.12.3 device updates to 19.12.5
>>> Wouldn't it be safer to upgrade first to 19.12.4? :-)
>> This is an interesting and important question, but it's orthogonal to the question of an API that represents the variety of builds that are available.
> I'd say it makes sense to always upgrade to the latest as we don't use any migration scripts anyway (do we?) and say a release is broken and quickly a followup is published, should all devices make this (possibly bricking) extra loop?
>>>> This could also introduce channels like "stable" (latest point release),
>>>> "testing" (rcN) and "unstable" (snapshots). As a dict is used the *versions*
>>>> array could be extended without losing compatibility.
>>> Déjà vu[1]? :-)
>>>
>>> 1. http://lists.infradead.org/pipermail/openwrt-devel/2019-August/018646.html
>
> Yea I'll stop my subtle Debian/OpenWrt merging ambitions now.
>
> Paul
>
> [0]: https://gitlab.alpinelinux.org/alpine/infra/infra/issues/10672
> [1]: https://wiki.alpinelinux.org/wiki/Alpine_Linux:Releases
>
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel
More information about the openwrt-devel
mailing list