[OpenWrt-Devel] [PATCH v2] build: refactor JSON info files to `profiles.json`
Paul Spooren
mail at aparcar.org
Sun Mar 1 19:03:13 EST 2020
On 01.03.20 02:29, Moritz Warning wrote:
> Hi Paul,
>
> as I see it, there are now two JSON options:
>
> - creates the single JSON files in the $(KDIR)/tmp folder
> - create a per target overview JSON that contains nearly all data of the single JSON files
This is the way I implemented it now :)
> Questions:
>
> - Why not only have one JSON menu option that create the one per target JSON file?
See above.
> - The Python script puts variable VERSION_CODE (e.g. r12145-4716c843d6) into a JSON field called version_commit.
> It is probably better to use version_code, before it cannot be changed anymore.
Thanks, included in v3
>
> On 3/1/20 3:48 AM, Paul Spooren wrote:
>> JSON info files contain machine readable information of built profiles
>> and resulting images. These files where added via 881ed09ee6e2. They are
>> useful for firmware wizards and script checking for reproducibility.
>>
>> Currently all JSON files are stored next to the built images, resulting
>> in up to 168 individual files for the ath79/generic target.
>>
>> This PR refactors the JSON creation to store individual files in
>> $(KDIR)/tmp and create an single overview file called `profiles.json` in
>> the target dir.
>>
>> As before, this creation is enabled by default only if `BUILDBOT` is set.
>>
>> The previous implementation used the functions `json.dumps()` which seem
>> to have caused broken files. Now the `pathlib` library is used to deal
>> with files and the `json` library only reads/writes into variables.
>>
>> Tested via buildroot & ImageBuilder on ath79/generic.
>>
>> Signed-off-by: Paul Spooren <mail at aparcar.org>
>> ---
>> v2:
>> * One instead of three CONFIG options
>> * Only created `profiles.json` without copying individual JSON files
>> * Add merging functionality to ImageBuilder
>> * Use underscores in Makefile function name
>> * Fix wrong `rm -f` path (missing /tmp)
>> * Use `pathlib` instead of `json.dump`
>> * Use `os.getenv` from Python stdlib
>> * Remove "generic" subtarget fallback as it is implement in image.mk
>>
>> Makefile | 6 ++++
>> config/Config-build.in | 10 ++++---
>> include/image.mk | 6 ++--
>> scripts/json_add_image_info.py | 45 ++++++++++++++---------------
>> scripts/json_overview_image_info.py | 33 +++++++++++++++++++++
>> target/imagebuilder/files/Makefile | 7 +++++
>> 6 files changed, 77 insertions(+), 30 deletions(-)
>> create mode 100755 scripts/json_overview_image_info.py
>>
>> diff --git a/Makefile b/Makefile
>> index 181c33b180..b9f92babf6 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -96,6 +96,11 @@ buildversion: FORCE
>> feedsversion: FORCE
>> $(SCRIPT_DIR)/feeds list -fs > $(BIN_DIR)/feeds.buildinfo
>>
>> +json_overview_image_info: FORCE
>> + INPUT_DIR=$(BUILD_DIR)/linux-$(BOARD)$(if $(SUBTARGET),_$(SUBTARGET))/tmp \
>> + TARGET_DIR=$(BIN_DIR) \
>> + $(SCRIPT_DIR)/json_overview_image_info.py
>> +
>> diffconfig: FORCE
>> mkdir -p $(BIN_DIR)
>> $(SCRIPT_DIR)/diffconfig.sh > $(BIN_DIR)/config.buildinfo
>> @@ -108,6 +113,7 @@ prepare: .config $(tools/stamp-compile) $(toolchain/stamp-compile)
>>
>> world: prepare $(target/stamp-compile) $(package/stamp-compile) $(package/stamp-install) $(target/stamp-install) FORCE
>> $(_SINGLE)$(SUBMAKE) -r package/index
>> + $(if $(CONFIG_JSON_OVERVIEW_IMAGE_INFO),$(_SINGLE)$(SUBMAKE) -r json_overview_image_info)
>> $(_SINGLE)$(SUBMAKE) -r checksum
>>
>> .PHONY: clean dirclean prereq prepare world package/symlinks package/symlinks-install package/symlinks-clean
>> diff --git a/config/Config-build.in b/config/Config-build.in
>> index 6a6fb2882c..57428399ab 100644
>> --- a/config/Config-build.in
>> +++ b/config/Config-build.in
>> @@ -7,12 +7,14 @@
>>
>> menu "Global build settings"
>>
>> - config JSON_ADD_IMAGE_INFO
>> - bool "Create JSON info files per build image"
>> + config JSON_OVERVIEW_IMAGE_INFO
>> + bool "Create JSON info file overview per target"
>> default BUILDBOT
>> + select JSON_CREATE_IMAGE_INFO
>> help
>> - The JSON info files contain information about the device and
>> - build images, stored next to the firmware images.
>> + Create a JSON info file called profiles.json in the target
>> + directory containing machine readable list of built profiles
>> + and resulting images.
>>
>> config ALL_NONSHARED
>> bool "Select all target specific packages by default"
>> diff --git a/include/image.mk b/include/image.mk
>> index fd04d4020b..933d844e8e 100644
>> --- a/include/image.mk
>> +++ b/include/image.mk
>> @@ -568,9 +568,9 @@ define Device/Build/image
>>
>> $(BIN_DIR)/$(call IMAGE_NAME,$(1),$(2)): $(KDIR)/tmp/$(call IMAGE_NAME,$(1),$(2))
>> cp $$^ $$@
>> - $(if $(CONFIG_JSON_ADD_IMAGE_INFO), \
>> + $(if $(CONFIG_JSON_OVERVIEW_IMAGE_INFO), \
>> DEVICE_ID="$(DEVICE_NAME)" \
>> - BIN_DIR="$(BIN_DIR)" \
>> + BIN_DIR="$(KDIR)/tmp" \
>> IMAGE_NAME="$(IMAGE_NAME)" \
>> IMAGE_TYPE=$(word 1,$(subst ., ,$(2))) \
>> IMAGE_PREFIX="$(IMAGE_PREFIX)" \
>> @@ -612,7 +612,7 @@ define Device/Build/artifact
>> endef
>>
>> define Device/Build
>> - $(shell rm -f $(BIN_DIR)/$(IMG_PREFIX)-$(1).json)
>> + $(shell rm -f $(KDIR)/tmp/$(IMG_PREFIX)-$(1).json)
>>
>> $(if $(CONFIG_TARGET_ROOTFS_INITRAMFS),$(call Device/Build/initramfs,$(1)))
>> $(call Device/Build/kernel,$(1))
>> diff --git a/scripts/json_add_image_info.py b/scripts/json_add_image_info.py
>> index 44b4031f85..c7f28a2183 100755
>> --- a/scripts/json_add_image_info.py
>> +++ b/scripts/json_add_image_info.py
>> @@ -1,18 +1,15 @@
>> #!/usr/bin/env python3
>>
>> import json
>> -import os
>> +from os import getenv
>> import hashlib
>> +from pathlib import Path
>>
>> +bin_dir = Path(getenv("BIN_DIR"))
>>
>> -def e(variable, default=None):
>> - return os.environ.get(variable, default)
>> +json_path = (bin_dir / getenv("IMAGE_PREFIX")).with_suffix(".json")
>>
>> -
>> -json_path = "{}{}{}.json".format(e("BIN_DIR"), os.sep, e("IMAGE_PREFIX"))
>> -
>> -with open(os.path.join(e("BIN_DIR"), e("IMAGE_NAME")), "rb") as image_file:
>> - image_hash = hashlib.sha256(image_file.read()).hexdigest()
>> +image_hash = hashlib.sha256((bin_dir / getenv("IMAGE_NAME")).read_bytes()).hexdigest()
>>
>>
>> def get_titles():
>> @@ -20,36 +17,38 @@ def get_titles():
>> for prefix in ["", "ALT0_", "ALT1_", "ALT2_"]:
>> title = {}
>> for var in ["vendor", "model", "variant"]:
>> - if e("DEVICE_{}{}".format(prefix, var.upper())):
>> - title[var] = e("DEVICE_{}{}".format(prefix, var.upper()))
>> + if getenv("DEVICE_{}{}".format(prefix, var.upper())):
>> + title[var] = getenv("DEVICE_{}{}".format(prefix, var.upper()))
>>
>> if title:
>> titles.append(title)
>>
>> if not titles:
>> - titles.append({"title": e("DEVICE_TITLE")})
>> + titles.append({"title": getenv("DEVICE_TITLE")})
>>
>> return titles
>>
>>
>> -if not os.path.exists(json_path):
>> +if not json_path.is_file():
>> device_info = {
>> - "id": e("DEVICE_ID"),
>> - "image_prefix": e("IMAGE_PREFIX"),
>> + "id": getenv("DEVICE_ID"),
>> + "image_prefix": getenv("IMAGE_PREFIX"),
>> "images": [],
>> "metadata_version": 1,
>> - "supported_devices": e("SUPPORTED_DEVICES").split(),
>> - "target": "{}/{}".format(e("TARGET"), e("SUBTARGET", "generic")),
>> + "supported_devices": getenv("SUPPORTED_DEVICES").split(),
>> + "target": "{}/{}".format(getenv("TARGET"), getenv("SUBTARGET")),
>> "titles": get_titles(),
>> - "version_commit": e("VERSION_CODE"),
>> - "version_number": e("VERSION_NUMBER"),
>> + "version_commit": getenv("VERSION_CODE"),
>> + "version_number": getenv("VERSION_NUMBER"),
>> }
>> else:
>> - with open(json_path, "r") as json_file:
>> - device_info = json.load(json_file)
>> + device_info = json.loads(json_path.read_text())
>>
>> -image_info = {"type": e("IMAGE_TYPE"), "name": e("IMAGE_NAME"), "sha256": image_hash}
>> +image_info = {
>> + "type": getenv("IMAGE_TYPE"),
>> + "name": getenv("IMAGE_NAME"),
>> + "sha256": image_hash,
>> +}
>> device_info["images"].append(image_info)
>>
>> -with open(json_path, "w") as json_file:
>> - json.dump(device_info, json_file, sort_keys=True, indent=" ")
>> +json_path.write_text(json.dumps(device_info))
>> diff --git a/scripts/json_overview_image_info.py b/scripts/json_overview_image_info.py
>> new file mode 100755
>> index 0000000000..bba13dd80c
>> --- /dev/null
>> +++ b/scripts/json_overview_image_info.py
>> @@ -0,0 +1,33 @@
>> +#!/usr/bin/env python3
>> +
>> +import json
>> +from pathlib import Path
>> +from os import getenv
>> +
>> +target_dir = Path(getenv("TARGET_DIR"))
>> +input_dir = Path(getenv("INPUT_DIR", target_dir))
>> +
>> +output_json = {}
>> +
>> +assert target_dir, "Target directory required"
>> +
>> +for json_file in input_dir.glob("*.json"):
>> + profile_info = json.loads(json_file.read_text())
>> + if not output_json:
>> + output_json = {
>> + "metadata_version": 1,
>> + "target": profile_info["target"],
>> + "version_commit": profile_info["version_commit"],
>> + "version_number": profile_info["version_number"],
>> + "profiles": {},
>> + }
>> +
>> + output_json["profiles"][profile_info["id"]] = {
>> + "supported_devices": profile_info["supported_devices"],
>> + "images": profile_info["images"],
>> + "titles": profile_info["titles"],
>> + }
>> +
>> +Path(target_dir / "profiles.json").write_text(
>> + json.dumps(output_json, sort_keys=True, indent=" ")
>> +)
>> diff --git a/target/imagebuilder/files/Makefile b/target/imagebuilder/files/Makefile
>> index 15b3d5c35c..ed9e298636 100644
>> --- a/target/imagebuilder/files/Makefile
>> +++ b/target/imagebuilder/files/Makefile
>> @@ -118,6 +118,7 @@ _call_image: staging_dir/host/.prereq-build
>> $(MAKE) package_install
>> $(MAKE) -s prepare_rootfs
>> $(MAKE) -s build_image
>> + $(if $(CONFIG_JSON_OVERVIEW_IMAGE_INFO),$(_SINGLE)$(SUBMAKE) -r json_overview_image_info)
>> $(MAKE) -s checksum
>>
>> _call_manifest: FORCE
>> @@ -163,12 +164,18 @@ prepare_rootfs: FORCE
>> $(CP) $(TARGET_DIR) $(TARGET_DIR_ORIG)
>> $(call prepare_rootfs,$(TARGET_DIR),$(USER_FILES),$(DISABLED_SERVICES))
>>
>> +
>> build_image: FORCE
>> @echo
>> @echo Building images...
>> $(NO_TRACE_MAKE) -C target/linux/$(BOARD)/image install TARGET_BUILD=1 IB=1 EXTRA_IMAGE_NAME="$(EXTRA_IMAGE_NAME)" \
>> $(if $(USER_PROFILE),PROFILE="$(USER_PROFILE)")
>>
>> +json_overview_image_info: FORCE
>> + INPUT_DIR=$(BUILD_DIR)/linux-$(BOARD)$(if $(SUBTARGET),_$(SUBTARGET))/tmp \
>> + TARGET_DIR=$(BIN_DIR) \
>> + $(SCRIPT_DIR)/json_overview_image_info.py
>> +
>> checksum: FORCE
>> @echo
>> @echo Calculating checksums...
>>
>
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel
More information about the openwrt-devel
mailing list