Packages buildbot is erratic, both master and 23.05 packages fail often

[Petr Štetiar]

Thibaut <hacks at slashdirt.org> [2023-06-02 11:09:48]:

Hi,

> the build is actually hung. dmesg might have more info.

So having following in buildbot log:

 2023-06-01 23:53:12+0000 [-] command timed out: 3600 seconds without output running [b'make', b'-j7', b'IGNORE_ERRORS=n m y', b'BUILD_LOG=1', b'CONFIG_AUTOREMOVE=y', b'CONFIG_SIGNED_PACKAGES='], attempting to kill
 2023-06-01 23:53:12+0000 [-] trying to kill process group 1528179

I've looked at the system logs around that time and found following:

 Jun 01 22:23:19 audit[3844576]: AVC apparmor="DENIED" operation="mkdir" info="Failed name lookup - name too long"
                 error=-36 profile="docker-default"
		 name="/shared-workdir/build/sdk/build_dir/hostpkg/gettext-0.21.1/gettext-tools/confdir3/confdir3/confdir3/confdir3...[snip very long repeating pattern]...
		 confdir3/confdir3/confdir3/confdir3/confdir3" pid=3844576 comm="conftest" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000
 Jun 01 22:23:45 kernel: conftest[3855174]: segfault at 0 ip 00007fe9581067e7 sp 00007ffd94ca2118 error 4 in libc-2.31.so[7fe958085000+159000]                  
 Jun 01 22:23:45 kernel: Code: 00 00 00 48 39 d1 0f 82 47 67 06 00 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 10 0f 82 fa 00 00 00 48 83 fa 20 0f 87 3f 01 00 00 <0f> 10 06 0f 10 4c 16 f0 0f 11 07 0f 11 4c 17 f0 c3 0f 1f 84 00 00

Since the host is shared with other 3 build workers I can't be sure, that it
originated from that timeouted build.

> I’m happy to share my config if interested, it’s not a very complex setup.

That would be appreciated, thanks!

> I’m confused with that sentence: the du step shows 36G used, but df says all
> 60G are full; which suggests something *outside* of the build directory is
> eating space?

I just got following on openwrt-23.05 and osuosl-dock-02 build worker:

 16G	dl
 36G	shared-workdir
     14G	shared-workdir/build/sdk/staging_dir
     13G	shared-workdir/build/sdk/build_dir
    8.6G	shared-workdir/build/sdk/tmp/go-build

And looking at the other snapshot packages buildworkers, situation is similar
on some of them:

 17G	osuosl-dock-09/dl
 17G	osuosl-dock-10/dl
 17G	osuosl-dock-11/dl
 17G	osuosl-dock-12/dl

 38G	osuosl-dock-09/shared-workdir/build
 35G	osuosl-dock-10/shared-workdir/build
 3.3G	osuosl-dock-11/shared-workdir/build
 7.1G	osuosl-dock-12/shared-workdir/build

So probably some cruft accumulated in the shared-workdir for some reason, will
dig deeper once I find some time.

Cheers,

Petr