[PATCH] scripts: add size_compare.sh

Paul Spooren mail at aparcar.org
Mon Nov 16 14:32:07 EST 2020


On Thu Sep 3, 2020 at 3:36 AM HST, Adrian Schmutzler wrote:
> > -----Original Message-----
> > From: openwrt-devel [mailto:openwrt-devel-bounces at lists.openwrt.org]
> > On Behalf Of Paul Spooren
> > Sent: Freitag, 31. Juli 2020 05:15
> > To: openwrt-devel at lists.openwrt.org
> > Cc: freifunk at adrianschmutzler.de; Paul Spooren <mail at aparcar.org>
> > Subject: [PATCH] scripts: add size_compare.sh
> > 
> > As package size changes are a continuous topic on the mailing list this scripts
> > helps developers to compare their local package modifications against latest
> > upstream.
> > 
> > The script downloads the latest package indexes based on env variables or
> > the `.config` file. The script compares the actual installed size
> > (data.tar.gz) or the IPK package size.
>
> Just gave this a quick try, and generally it looks good.
>
> However, I got 82 packages listed for " Compare packages of
> ath79/generic/mips_24kc", but only 66 were in the final comparison.

The current script also interprets some options as packages, which is
couldn't find a clean solution for:

CONFIG_PACKAGE_MAC80211_DEBUGFS=y
# CONFIG_PACKAGE_MAC80211_TRACING is not set
CONFIG_PACKAGE_MAC80211_MESH=y

Also, this script only checks for packages of openwrt.git, so if you
have packages from other repositories installed, they'd be missing.

>
> Will check the code in detail later. A few comment below.
>
> > 
> > An example output is found below:
> > 
> > ```
> > user at dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh Compare
> > packages of ath79/tiny/mips_24kc:
> > dropbear busybox iw ubus
> > 
> > Checking installed size
> > 
> > Fetching latest package indexes...
> >   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
> >                                  Dload  Upload   Total   Spent    Left  Speed
> > 100 80634  100 80634    0     0  33499      0  0:00:02  0:00:02 --:--:-- 33485
> >   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
> >                                  Dload  Upload   Total   Spent    Left  Speed
> > 100 54082  100 54082    0     0  24252      0  0:00:02  0:00:02 --:--:-- 24252
> > 
> > Comparing package sizes...
> > Change 	Local 	Package
> > 611B	208715B	busybox
> > 39B	5612B	ubus
> > -42B	34940B	iw
> > -14916B	89265B	dropbear
> > ```
> > 
> > I plan to integrate this script into the CI so we have a summary how sizes
> > change over different architectures.
> > 
> > Signed-off-by: Paul Spooren <mail at aparcar.org>
> > ---
> >  scripts/size_compare.sh | 117
> > ++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 117 insertions(+)
> >  create mode 100755 scripts/size_compare.sh
> > 
> > diff --git a/scripts/size_compare.sh b/scripts/size_compare.sh new file mode
> > 100755 index 0000000000..b310a085a4
> > --- /dev/null
> > +++ b/scripts/size_compare.sh
> > @@ -0,0 +1,117 @@
> > +# SPDX-License-Identifier: GPL-2.0-or-later # # Copyright (C) 2020 Paul
> > +Spooren <mail at aparcar.org> # ### ### size_compare - compare size of
> > +OpenWrt packages against upstream ### ### The script compares locally
> > +compiled package with the package indexes ### available upstream. This
> > +way the storage impact of optimizations or ### feature modifiactions is
> > +easy to see.
> > +###
> > +### If no environmental variables are set the scritp reads the current
> > +### .config file. The evaluated env variables are the following:
> > +###
> > +###   TARGET SUBTARGET ARCH PACKAGES BIN_DIR BASE_URL
> > CHECK_INSTALLED
> > +###
> > +### Usage:
> > +###   ./scripts/size_compare.sh
> > +###
> > +### Options:
> > +###   -p --package-size 	Check IPK package size and not installed size
> > +###   -h --help 		This message
> > +
> > +CONFIG_TARGET=$(sed -n 's/^CONFIG_TARGET_BOARD="\(.*\)"$/\1/p'
> > .config)
> > +CONFIG_SUBTARGET=$(sed -n
> > 's/^CONFIG_TARGET_SUBTARGET="\(.*\)"$/\1/p'
> > +.config) CONFIG_ARCH=$(sed -n
> > +'s/^CONFIG_TARGET_ARCH_PACKAGES="\(.*\)"$/\1/p' .config)
> > +CONFIG_PACKAGES=$(sed -n 's/^CONFIG_PACKAGE_\(.*\)=y$/\1/p'
> > .config |
> > +tr '\n' ' ') CONFIG_BIN_DIR=$(sed -n
> > +'s/^CONFIG_BINARY_DIR="\(.*\)"$/\1/p' .config)
> > +
> > +TARGET=${TARGET:-$CONFIG_TARGET}
> > +SUBTARGET=${SUBTARGET:-$CONFIG_SUBTARGET}
> > +ARCH=${ARCH:-$CONFIG_ARCH}
> > +PACKAGES=${PACKAGES:-$CONFIG_PACKAGES}
> > +BIN_DIR=${CONFIG_BIN_DIR:-./bin}
> > +BASE_URL="${BASE_URL:-https://downloads.openwrt.org/snapshots}"
> > +CHECK_INSTALLED="${CHECK_INSTALLED:-y}"
> > +
> > +TARGET_URL="$BASE_URL/targets/$TARGET/$SUBTARGET/packages/Pack
> > ages.gz"
> > +PACKAGES_URL="$BASE_URL/packages/$ARCH/base/Packages.gz"
> > +
> > +help() {
> > +    sed -rn 's/^### ?//;T;p' "$0"
> > +}
> > +
> > +package_size () {
> > +	FOUND_PACKAGE=
> > +	if [ -z "$CHECK_INSTALLED" ]; then
> > +		SEARCH_PATTERN="Size"
> > +	else
> > +		SEARCH_PATTERN="Installed-Size"
> > +	fi
> > +	while IFS= read -r line; do
> > +		if [ "$line" = "Package: $2" ]; then
> > +			FOUND_PACKAGE=y
> > +		fi
> > +		if [ -n "$FOUND_PACKAGE" ]; then
> > +			case $line in
> > +				"$SEARCH_PATTERN"*)
> > +					echo "$line" | cut -d ' ' -f 2
> > +					break
> > +					;;
> > +			esac
> > +		fi
> > +	done < "$1"
> > +}
> > +
> > +compare_sizes () {
> > +	for PACKAGE in $PACKAGES; do
> > +		if [ "$PACKAGE" = "libc" ]; then
> > +			continue
> > +		fi
> > +		PACKAGE_FILE=$(find "$BIN_DIR/packages/$ARCH/"
> > "$BIN_DIR/targets/$TARGET/$SUBTARGET/" -name "${PACKAGE}_*.ipk")
>
> When I first run this, I was in x86 target where only a kernel refresh
> has happened before (so, no build).
> This then produces a lot of warning due to missing packages folder:
>
> find: ‘./bin/targets/x86/generic/’: No such file or directory
> find: ‘./bin/packages/i386_pentium4/’: No such file or directory
>
> So, one should either suppress the message here for the find command, or
> alternatively check for the folders' existence beforehand.

I surpressed the error message.

>
> > +		if [ -z "$PACKAGE_FILE" ]; then
> > +			continue
> > +		fi
> > +		if [ -z "$CHECK_INSTALLED" ]; then
> > +			SIZE_LOCAL=$(stat -c '%s' "$PACKAGE_FILE")
> > +		else
> > +			SIZE_LOCAL=$(tar tzvf "$PACKAGE_FILE" | grep
> > data.tar.gz | awk '{ print $3 }')
> > +		fi
> > +		SIZE_UPSTREAM=$(package_size "$TMP_INDEX"
> > "$PACKAGE")
> > +		SIZE_DIFF="$((SIZE_LOCAL-SIZE_UPSTREAM))"
> > +		echo "${SIZE_DIFF}B	${SIZE_LOCAL}B	$PACKAGE"
>
> Personally, I think the "B" looks ugly. As we only have bytes, we could
> consider to drop it.

Dropped.

> One could also consider to add a "+" for the first column when change is
> positive. Both changes here are lowest priority.

Added.

I created a updated version on GitHub as I'm not the biggest fan of
patchworks: https://github.com/openwrt/openwrt/pull/3599

> > +	done
> > +}
> > +
> > +if [ "$1" = "-h" ]; then
> > +    help
> > +    exit 0
> > +fi
> > +
> > +if [ "$1" = "-p" ]; then
> > +    CHECK_INSTALLED=
> > +fi
> > +
> > +echo "Compare packages of $TARGET/$SUBTARGET/$ARCH":
> > +echo "$PACKAGES"
> > +echo
> > +
> > +if [ -z "$CHECK_INSTALLED" ]; then
> > +	echo "Checking IPK package size"
> > +else
> > +	echo "Checking installed size"
> > +fi
> > +echo
> > +
> > +echo "Fetching latest package indexes..."
> > +TMP_INDEX=$(mktemp /tmp/size_compare_package_index.XXXXXX)
> > +curl "$TARGET_URL" | gzip -d > "$TMP_INDEX" || exit 1 curl
> > +"$PACKAGES_URL" | gzip -d >> "$TMP_INDEX" || exit 1 echo
> > +
> > +echo "Comparing package sizes..."
> > +echo "Change 	Local 	Package"
>
> Since we have enough space, I'd go for "Change Local Remote Package"
> here.
>
> Best
>
> Adrian
>
> > +compare_sizes | sort -n -r
> > +
> > +rm "$TMP_INDEX"
> > --
> > 2.25.1
> > 
> > 
> > _______________________________________________
> > openwrt-devel mailing list
> > openwrt-devel at lists.openwrt.org
> > https://lists.openwrt.org/mailman/listinfo/openwrt-devel




More information about the openwrt-devel mailing list