Build issues gcc10

[Adrian Schmutzler]

Hi again,

> -----Original Message-----
> From: openwrt-devel [mailto:openwrt-devel-bounces at lists.openwrt.org]
> On Behalf Of Rui Salvaterra
> Sent: Dienstag, 14. September 2021 00:29
> To: Adrian Schmutzler <mail at adrianschmutzler.de>
> Cc: openwrt-devel at lists.openwrt.org
> Subject: Re: Build issues gcc10
> 
> Hi, Adrian,
> 
> On Mon, 13 Sept 2021 at 23:10, Adrian Schmutzler
> <mail at adrianschmutzler.de> wrote:
> >
> > Just had one try before I went to bed, but it appears the problem is that
> cc1plus generated by the toolchain has no execute bit set:

I've resolved the thing. It turned out to be a RAM virtualization issue. Read the full story below (pasted here FYI and for documentation):

I've been using Hyper-V for virtualization of the relevant systems.
Hyper-V provides a "Dynamic Memory" setting where one can set a minimum/maximum RAM value, so memory is provided to the machines "by demand" with some buffer.
In addition, there is a "Startup RAM" setting that is used for boot etc. where the integration drivers for Hyper-V are not yet available in the guest system.

For all relevant systems, Startup RAM had been set to 2048 MiB, and maximum memory was set to 8G or higher.

During further investigation of my build logs, I found that - apart from the error reported here previously - I had another error which occurred earlier in the process.
I received

collect2: fatal error: ld terminated with signal 9 [Killed]

during the following command:

g++ -no-pie   -g -O2 -DIN_GCC  -DCROSS_DIRECTORY_STRUCTURE   -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings   -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc  -o cc1plus \
      cp/cp-lang.o c-family/stub-objc.o cp/call.o cp/class.o cp/constexpr.o cp/constraint.o cp/coroutines.o cp/cp-gimplify.o cp/cp-objcp-common.o cp/cp-ubsan.o cp/cvt.o cp/cxx-pretty-print.o cp/decl.o cp/decl2.o cp/dump.o cp/error.o cp/except.o cp/expr.o cp/friend.o cp/init.o cp/lambda.o cp/lex.o cp/logic.o cp/mangle.o cp/method.o cp/name-lookup.o cp/optimize.o cp/parser.o cp/pt.o cp/ptree.o cp/rtti.o cp/search.o cp/semantics.o cp/tree.o cp/typeck.o cp/typeck2.o cp/vtable-class-hierarchy.o attribs.o incpath.o c-family/c-common.o c-family/c-cppbuiltin.o c-family/c-dump.o c-family/c-format.o c-family/c-gimplify.o c-family/c-indentation.o c-family/c-lex.o c-family/c-omp.o c-family/c-opts.o c-family/c-pch.o c-family/c-ppoutput.o c-family/c-pragma.o c-family/c-pretty-print.o c-family/c-semantics.o c-family/c-ada-spec.o c-family/c-ubsan.o c-family/known-headers.o c-family/c-attribs.o c-family/c-warn.o c-family/c-spellcheck.o arm-c.o glibc-c.o cc1plus-checksum.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a libcommon.a ../libcpp/libcpp.a   ../libbacktrace/.libs/libbacktrace.a ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a   -L/data/openwrt/staging_dir/host/lib -L/data/openwrt/staging_dir/host/lib -L/data/openwrt/staging_dir/host/lib -lmpc -lmpfr -lgmp -rdynamic -ldl  -L./../zlib -lz

So, effectively, the creation of cc1plus, which is done in this command, has been failing due to insufficient memory.
Curiously, the file cc1plus is still created after that step, but missing the proper permissions:

ls -al ./build_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-10.3.0_musl_eabi/gcc-10.3.0-final/gcc/cc1plus
-rw-r--r-- 1 adsc adsc 244655320 Sep 15 23:00 ./build_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-10.3.0_musl_eabi/gcc-10.3.0-final/gcc/cc1plus

Due to that reason, in subsequent builds Make did see the file and did not rerun the command above.
Thus, build then only failed later when the gcc checks were investigating the file and found that it was not set up properly.
 (And hence, looking at it after the n-th try, I didn't find the root cause in the first place).

Calling the command manually (in the correct folder) worked (with the same memory setup), and created a file with proper permissions:

ls -al cc1plus
-rwxr-xr-x 1 adsc adsc 244655320 Sep 15 23:16 cc1plus

Note that the size is exactly the same as for the non-executable file after the failure before. (I did not compare contents).

After fixing this (i.e. manually running the command), build continued to run.

Internet told me that the "collect2 ..." error means insufficient memory.
So, since I never reached the maximum level, I tried to increase the "Startup RAM" and found that this seems to be the solution (4G is already fine for -j12).

It appears that although Hyper-V does signal that Dynamic Memory is adjusting values, this is not or not fully working inside the guest.

Still, it's quite remarkable that the system reproducibly failed for gcc 10 and 11, while it worked for gcc 8 and 9, and all of that at the very same individual build step.
I don't really have an answer here, but that probably depends on the way and degree of how Dynamic Memory is broken or limited.

So, why did it work on Ubuntu, and failed on Debian?
This was quite a hard one. Both were set up with the same settings, and since hyper-v integration is in the kernel nowadays, support should be similar on both distros (and I can't blame the driver).
Eventually, it turned out that the difference was the size of the swap partition which I never touched, and which defaults to 2G on Ubuntu and 1G on Debian ;-). (So, startup memory + swap simply was considerably bigger on Ubuntu).

Anyway, now everything appears to be working with the adjusted settings.

Best

Adrian