[PATCH] scripts: add size_compare.sh
Adrian Schmutzler
mail at adrianschmutzler.de
Thu Sep 3 09:36:20 EDT 2020
> -----Original Message-----
> From: openwrt-devel [mailto:openwrt-devel-bounces at lists.openwrt.org]
> On Behalf Of Paul Spooren
> Sent: Freitag, 31. Juli 2020 05:15
> To: openwrt-devel at lists.openwrt.org
> Cc: freifunk at adrianschmutzler.de; Paul Spooren <mail at aparcar.org>
> Subject: [PATCH] scripts: add size_compare.sh
>
> As package size changes are a continuous topic on the mailing list this scripts
> helps developers to compare their local package modifications against latest
> upstream.
>
> The script downloads the latest package indexes based on env variables or
> the `.config` file. The script compares the actual installed size
> (data.tar.gz) or the IPK package size.
Just gave this a quick try, and generally it looks good.
However, I got 82 packages listed for " Compare packages of ath79/generic/mips_24kc", but only 66 were in the final comparison.
Will check the code in detail later. A few comment below.
>
> An example output is found below:
>
> ```
> user at dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh Compare
> packages of ath79/tiny/mips_24kc:
> dropbear busybox iw ubus
>
> Checking installed size
>
> Fetching latest package indexes...
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 80634 100 80634 0 0 33499 0 0:00:02 0:00:02 --:--:-- 33485
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 54082 100 54082 0 0 24252 0 0:00:02 0:00:02 --:--:-- 24252
>
> Comparing package sizes...
> Change Local Package
> 611B 208715B busybox
> 39B 5612B ubus
> -42B 34940B iw
> -14916B 89265B dropbear
> ```
>
> I plan to integrate this script into the CI so we have a summary how sizes
> change over different architectures.
>
> Signed-off-by: Paul Spooren <mail at aparcar.org>
> ---
> scripts/size_compare.sh | 117
> ++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 117 insertions(+)
> create mode 100755 scripts/size_compare.sh
>
> diff --git a/scripts/size_compare.sh b/scripts/size_compare.sh new file mode
> 100755 index 0000000000..b310a085a4
> --- /dev/null
> +++ b/scripts/size_compare.sh
> @@ -0,0 +1,117 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later # # Copyright (C) 2020 Paul
> +Spooren <mail at aparcar.org> # ### ### size_compare - compare size of
> +OpenWrt packages against upstream ### ### The script compares locally
> +compiled package with the package indexes ### available upstream. This
> +way the storage impact of optimizations or ### feature modifiactions is
> +easy to see.
> +###
> +### If no environmental variables are set the scritp reads the current
> +### .config file. The evaluated env variables are the following:
> +###
> +### TARGET SUBTARGET ARCH PACKAGES BIN_DIR BASE_URL
> CHECK_INSTALLED
> +###
> +### Usage:
> +### ./scripts/size_compare.sh
> +###
> +### Options:
> +### -p --package-size Check IPK package size and not installed size
> +### -h --help This message
> +
> +CONFIG_TARGET=$(sed -n 's/^CONFIG_TARGET_BOARD="\(.*\)"$/\1/p'
> .config)
> +CONFIG_SUBTARGET=$(sed -n
> 's/^CONFIG_TARGET_SUBTARGET="\(.*\)"$/\1/p'
> +.config) CONFIG_ARCH=$(sed -n
> +'s/^CONFIG_TARGET_ARCH_PACKAGES="\(.*\)"$/\1/p' .config)
> +CONFIG_PACKAGES=$(sed -n 's/^CONFIG_PACKAGE_\(.*\)=y$/\1/p'
> .config |
> +tr '\n' ' ') CONFIG_BIN_DIR=$(sed -n
> +'s/^CONFIG_BINARY_DIR="\(.*\)"$/\1/p' .config)
> +
> +TARGET=${TARGET:-$CONFIG_TARGET}
> +SUBTARGET=${SUBTARGET:-$CONFIG_SUBTARGET}
> +ARCH=${ARCH:-$CONFIG_ARCH}
> +PACKAGES=${PACKAGES:-$CONFIG_PACKAGES}
> +BIN_DIR=${CONFIG_BIN_DIR:-./bin}
> +BASE_URL="${BASE_URL:-https://downloads.openwrt.org/snapshots}"
> +CHECK_INSTALLED="${CHECK_INSTALLED:-y}"
> +
> +TARGET_URL="$BASE_URL/targets/$TARGET/$SUBTARGET/packages/Pack
> ages.gz"
> +PACKAGES_URL="$BASE_URL/packages/$ARCH/base/Packages.gz"
> +
> +help() {
> + sed -rn 's/^### ?//;T;p' "$0"
> +}
> +
> +package_size () {
> + FOUND_PACKAGE=
> + if [ -z "$CHECK_INSTALLED" ]; then
> + SEARCH_PATTERN="Size"
> + else
> + SEARCH_PATTERN="Installed-Size"
> + fi
> + while IFS= read -r line; do
> + if [ "$line" = "Package: $2" ]; then
> + FOUND_PACKAGE=y
> + fi
> + if [ -n "$FOUND_PACKAGE" ]; then
> + case $line in
> + "$SEARCH_PATTERN"*)
> + echo "$line" | cut -d ' ' -f 2
> + break
> + ;;
> + esac
> + fi
> + done < "$1"
> +}
> +
> +compare_sizes () {
> + for PACKAGE in $PACKAGES; do
> + if [ "$PACKAGE" = "libc" ]; then
> + continue
> + fi
> + PACKAGE_FILE=$(find "$BIN_DIR/packages/$ARCH/"
> "$BIN_DIR/targets/$TARGET/$SUBTARGET/" -name "${PACKAGE}_*.ipk")
When I first run this, I was in x86 target where only a kernel refresh has happened before (so, no build).
This then produces a lot of warning due to missing packages folder:
find: ‘./bin/targets/x86/generic/’: No such file or directory
find: ‘./bin/packages/i386_pentium4/’: No such file or directory
So, one should either suppress the message here for the find command, or alternatively check for the folders' existence beforehand.
> + if [ -z "$PACKAGE_FILE" ]; then
> + continue
> + fi
> + if [ -z "$CHECK_INSTALLED" ]; then
> + SIZE_LOCAL=$(stat -c '%s' "$PACKAGE_FILE")
> + else
> + SIZE_LOCAL=$(tar tzvf "$PACKAGE_FILE" | grep
> data.tar.gz | awk '{ print $3 }')
> + fi
> + SIZE_UPSTREAM=$(package_size "$TMP_INDEX"
> "$PACKAGE")
> + SIZE_DIFF="$((SIZE_LOCAL-SIZE_UPSTREAM))"
> + echo "${SIZE_DIFF}B ${SIZE_LOCAL}B $PACKAGE"
Personally, I think the "B" looks ugly. As we only have bytes, we could consider to drop it.
One could also consider to add a "+" for the first column when change is positive. Both changes here are lowest priority.
> + done
> +}
> +
> +if [ "$1" = "-h" ]; then
> + help
> + exit 0
> +fi
> +
> +if [ "$1" = "-p" ]; then
> + CHECK_INSTALLED=
> +fi
> +
> +echo "Compare packages of $TARGET/$SUBTARGET/$ARCH":
> +echo "$PACKAGES"
> +echo
> +
> +if [ -z "$CHECK_INSTALLED" ]; then
> + echo "Checking IPK package size"
> +else
> + echo "Checking installed size"
> +fi
> +echo
> +
> +echo "Fetching latest package indexes..."
> +TMP_INDEX=$(mktemp /tmp/size_compare_package_index.XXXXXX)
> +curl "$TARGET_URL" | gzip -d > "$TMP_INDEX" || exit 1 curl
> +"$PACKAGES_URL" | gzip -d >> "$TMP_INDEX" || exit 1 echo
> +
> +echo "Comparing package sizes..."
> +echo "Change Local Package"
Since we have enough space, I'd go for "Change Local Remote Package" here.
Best
Adrian
> +compare_sizes | sort -n -r
> +
> +rm "$TMP_INDEX"
> --
> 2.25.1
>
>
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: openpgp-digital-signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.openwrt.org/pipermail/openwrt-devel/attachments/20200903/a18e0775/attachment.sig>
More information about the openwrt-devel
mailing list