[FS#2734] Opkg update fails although router has enough memory

Sat Dec 26 11:58:35 EST 2020

THIS IS AN AUTOMATED MESSAGE, DO NOT REPLY.

The following task has a new comment added:

FS#2734 - Opkg update fails although router has enough memory
User who did this - Nick (kousu)

----------
I'm seeing this too on on Archer C50 v4 (https://openwrt.org/toh/hwdata/tp-link/tp-link_archer_c50_v4), but only on list-upgradable., I can run update as many times as I want, it seems.

This router was manufactured just last year. Though it's not a 16/128 device like [[https://openwrt.org/toh/views/toh_available_16128|recommended]], neither is it a [[https://openwrt.org/supported_devices/openwrt_on_432_devices|banned 4/32]]. It's in the middle: a 8/64, and that's supposed to be powerful enough.

I'm running LuCI openwrt-19.07 branch (git-20.272.48698-9883bb3) / OpenWrt 19.07.4 r11208-ce6496d796.

I tried running Baptiste's experiment:

Summary:

**opkg update**: 6.7MB
**opkg list**: 14MB
**opkg list-upgradable**: 44 - 50 MB (and then a crash)
**opkg list-installed**: 3.8MB
**opkg upgrade**: 49MB (and then a crash); presumably this is calling list-upgradable internally?

root at router:~# time -v opkg update >/dev/null
	Command being timed: "opkg update"
	User time (seconds): 3.83
	System time (seconds): 0.86
	Percent of CPU this job got: 42%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 10.98s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 6736
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 4355
	Voluntary context switches: 1032
	Involuntary context switches: 1636
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
root at router:~# time -v opkg update >/dev/null
	Command being timed: "opkg update"
	User time (seconds): 4.01
	System time (seconds): 0.62
	Percent of CPU this job got: 43%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 10.73s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 6736
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 4365
	Voluntary context switches: 1026
	Involuntary context switches: 1539
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
root at router:~# time -v opkg list >/dev/null
	Command being timed: "opkg list"
	User time (seconds): 2.15
	System time (seconds): 0.15
	Percent of CPU this job got: 94%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 2.43s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 13936
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 1947
	Voluntary context switches: 584
	Involuntary context switches: 937
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
root at router:~# time -v opkg list-upgradable # not >/dev/null because this errors and print the error to *stdout*
Collected errors:
 * pkg_hash_add_from_file: Failed to open /var/opkg-lists/openwrt_telephony: Out of memory.
	Command being timed: "opkg list-upgradable"
	User time (seconds): 3.32
	System time (seconds): 0.35
	Percent of CPU this job got: 96%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 3.80s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 50176
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 6698
	Voluntary context switches: 732
	Involuntary context switches: 1351
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
root at router:~# time -v opkg list-upgradable 
Collected errors:
 * pkg_hash_add_from_file: Failed to open /var/opkg-lists/openwrt_telephony: Out of memory.
Command exited with non-zero status 255
	Command being timed: "opkg list-upgradable"
	User time (seconds): 2.80
	System time (seconds): 0.28
	Percent of CPU this job got: 81%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 3.78s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 44288
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 5985
	Voluntary context switches: 664
	Involuntary context switches: 1201
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 255
root at router:~# time -v (opkg list-upgradable | head)
-ash: syntax error: unexpected "("
root at router:~# time -v opkg list-installed >/dev/null
	Command being timed: "opkg list-installed"
	User time (seconds): 0.05
	System time (seconds): 0.00
	Percent of CPU this job got: 94%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 0.05s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3872
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 81
	Voluntary context switches: 1
	Involuntary context switches: 8
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
root at router:~# time -v opkg upgrade 
Collected errors:
 * pkg_hash_add_from_file: Failed to open /var/opkg-lists/openwrt_routing: Out of memory.
Command exited with non-zero status 255
	Command being timed: "opkg upgrade"
	User time (seconds): 2.61
	System time (seconds): 0.24
	Percent of CPU this job got: 93%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 3.05s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 49792
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 4460
	Voluntary context switches: 640
	Involuntary context switches: 1133
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 255

Turning on sysctl -w vm.overcommit_memory=1; allows the process to c

A side issue: why is opkg upgrade running list-upgradable if it's just going to stop when it notices I haven't told it what packages to upgrade?

root at router:~# sysctl -w vm.overcommit_memory=1
vm.overcommit_memory = 1
root at router:~# time -v opkg list-upgradable
luci-app-statistics - git-20.348.38488-caae7ad-1 - git-20.358.74878-e019fd2-1
luci-app-opkg - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
libuci20130104 - 2019-09-01-415f9e48-3 - 2019-09-01-415f9e48-4
luci-lib-ip - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-mod-system - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-theme-bootstrap - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-mod-status - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-lib-iptparser - git-20.348.38488-caae7ad-1 - git-20.358.74878-e019fd2-1
luci-app-firewall - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
tcpdump - 4.9.3-1 - 4.9.3-2
uci - 2019-09-01-415f9e48-3 - 2019-09-01-415f9e48-4
luci-compat - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
rpcd-mod-luci - 20191114 - 20201107
luci-proto-ppp - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-mod-admin-full - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-base - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-app-simple-adblock - git-20.348.38488-caae7ad-50 - git-20.358.74878-e019fd2-50
luci-proto-ipv6 - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
uhttpd - 2020-03-13-975dce23-1 - 2020-10-01-3abcc891-1
luci-lib-nixio - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-lib-jsonc - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-mod-network - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
wireless-regdb - 2019.06.03-1 - 2020.11.20-1
luci-app-openvpn - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
	Command being timed: "opkg list-upgradable"
	User time (seconds): 3.44
	System time (seconds): 0.29
	Percent of CPU this job got: 96%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 3.87s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 54096
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 6994
	Voluntary context switches: 729
	Involuntary context switches: 1482
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
root at router:~# time -v opkg upgrade 
opkg: the ``upgrade'' command requires at least one argument
usage: opkg [options...] sub-command [arguments...]
where sub-command is one of:

Package Manipulation:
	update			Update list of available packages
	upgrade 		Upgrade packages
	install 		Install package(s)
	configure 	Configure unpacked package(s)
	remove 	Remove package(s)
	flag  	Flag package(s)
	 =hold|noprune|user|ok|installed|unpacked (one per invocation)

Informational Commands:
	list			List available packages
	list-installed		List installed packages
	list-upgradable		List installed and upgradable packages
	list-changed-conffiles	List user modified configuration files
	files 		List files belonging to 
	search 	List package providing 
	find 		List packages whose name or description matches 
	info [pkg|regexp]	Display all info for 
	status [pkg|regexp]	Display all status for 
	download 		Download  to current directory
	compare-versions   
	                    compare versions using  >= = >
	print-architecture	List installable package architectures
	depends [-A] [pkgname|pat]+
	whatdepends [-A] [pkgname|pat]+
	whatdependsrec [-A] [pkgname|pat]+
	whatrecommends[-A] [pkgname|pat]+
	whatsuggests[-A] [pkgname|pat]+
	whatprovides [-A] [pkgname|pat]+
	whatconflicts [-A] [pkgname|pat]+
	whatreplaces [-A] [pkgname|pat]+

Options:
	-A			Query all packages not just those installed
	-V[]		Set verbosity level to .
	--verbosity[=]	Verbosity levels:
					0 errors only
					1 normal messages (default)
					2 informative messages
					3 debug
					4 debug level 2
	-f 		Use  as the opkg configuration file
	--conf 
	--cache 	Use a package cache
	-d 		Use  as the the root directory for
	--dest 	package installation, removal, upgrading.
				 should be a defined dest name from
				the configuration file, (but can also be a
				directory name in a pinch).
	-o 		Use  as the root directory for
	--offline-root 	offline installation of packages.
	--add-arch :	Register architecture with given priority
	--add-dest :	Register destination with given path

Force Options:
	--force-depends		Install/remove despite failed dependencies
	--force-maintainer	Overwrite preexisting config files
	--force-reinstall	Reinstall package(s)
	--force-overwrite	Overwrite files from other package(s)
	--force-downgrade	Allow opkg to downgrade packages
	--force-space		Disable free space checks
	--force-postinstall	Run postinstall scripts even in offline mode
	--force-remove	Remove package even if prerm script fails
	--force-checksum	Don't fail on checksum mismatches
	--no-check-certificate Don't validate SSL certificates
	--noaction		No action -- test only
	--download-only	No action -- download only
	--nodeps		Do not follow dependencies
	--nocase		Perform case insensitive pattern matching
	--size			Print package size when listing available packages
	--force-removal-of-dependent-packages
				Remove package and all dependencies
	--autoremove		Remove packages that were installed
				automatically to satisfy dependencies
	-t			Specify tmp-dir.
	--tmp-dir		Specify tmp-dir.
	-l			Specify lists-dir.
	--lists-dir		Specify lists-dir.

 regexp could be something like 'pkgname*' '*file*' or similar
 e.g. opkg info 'libstd*' or opkg search '*libop*' or opkg remove 'libncur*'
Command exited with non-zero status 1
	Command being timed: "opkg upgrade"
	User time (seconds): 2.91
	System time (seconds): 0.25
	Percent of CPU this job got: 96%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 3.29s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 49808
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 5428
	Voluntary context switches: 709
	Involuntary context switches: 1378
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 1

Trying the "rm" solution doesn't help me because it comes back as soon as re-run opkg update.

root at router:~# sysctl -w vm.overcommit_memory=0
vm.overcommit_memory = 0
root at router:~# rm /var/opkg-lists/openwrt_*
root at router:~# time -v opkg list-upgradable   # this suceeds, but vacuously: it just prints an empty list
	Command being timed: "opkg list-upgradable"
	User time (seconds): 0.09
	System time (seconds): 0.06
	Percent of CPU this job got: 94%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 0.17s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3984
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 1
	Minor (reclaiming a frame) page faults: 102
	Voluntary context switches: 1
	Involuntary context switches: 18
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
root at router:~# time -v opkg update
Downloading http://downloads.openwrt.org/releases/19.07.4/targets/ramips/mt76x8/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_core
Downloading http://downloads.openwrt.org/releases/19.07.4/targets/ramips/mt76x8/packages/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/base/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_base
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/base/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/luci/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_luci
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/luci/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_packages
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/packages/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/routing/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_routing
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/routing/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/telephony/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_telephony
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/telephony/Packages.sig
Signature check passed.
	Command being timed: "opkg update"
	User time (seconds): 4.02
	System time (seconds): 1.59
	Percent of CPU this job got: 45%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 12.27s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 7072
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 56
	Minor (reclaiming a frame) page faults: 4349
	Voluntary context switches: 1096
	Involuntary context switches: 2190
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
root at router:~# time -v opkg list-upgradable
Collected errors:
 * pkg_hash_add_from_file: Failed to open /var/opkg-lists/openwrt_telephony: Out of memory.
Command exited with non-zero status 255
	Command being timed: "opkg list-upgradable"
	User time (seconds): 2.82
	System time (seconds): 0.28
	Percent of CPU this job got: 94%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 3.28s
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 44320
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 5980
	Voluntary context switches: 670
	Involuntary context switches: 1315
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 255

I was finally able to get my system updated with

sysctl -w vm.overcommit_memory=1
time -v opkg upgrade $(opkg list-upgradable | cut -f 1 -d ' ')

-------------------------------

I share your anxiety, Karlito. Dropping support for older devices leaves a lot of perfectly good hardware sitting wasted, and leaves us. The same thing happens with smartphones all the time; when the most common messenger apps your friends use decide your phone is too old then you're forced to buy a new one, or be cut off if you can't afford it. I imagine this turning out to be critical for hackers trying to network their communities in developing countries.

But I think we should remember that this is a community-run volunteer project that we're all benefitting from. Bugs are everywhere, and if they seem more prominent in free software that's more because there's not nearly as much funding for it; proprietary routers are also full of bugs, but they have the kind of money to patch over the most common ones, or pay for technical support teams to teach people how to work around them, that a project like OpenWRT just can't afford. Plus they can control exactly what is in their hardware which makes their job 100 times easier than the job of the OpenWRT volunteers. By dropping support for older models they are trying to reduce the amount of bugs they need to solve. But that same process makes cheaper models more prone to bugs because they're not getting tested as thoroughly anymore.

I don't have a good solution. We need to get open source funded properly I guess!

I do hope we can extend the life of cheaper routers. We've identified list-upgradable as the problem here, so maybe we can pitch in by tracing the code and seeing if there's a way to make it process the data in more of a stream instead of a blob.
----------

More information can be found at the following URL:
https://bugs.openwrt.org/index.php?do=details&task_id=2734#comment9214

You are receiving this message because you have requested it from the Flyspray bugtracking system.  If you did not expect this message or don't want to receive mails in future, you can change your notification settings at the URL shown above.