Packages buildbot is erratic, both master and 23.05 packages fail often

[Thibaut]

Hi,

> Le 2 juin 2023 à 21:07, Petr Štetiar <ynezz at true.cz> a écrit :
> 
> Thibaut <hacks at slashdirt.org> [2023-06-02 11:09:48]:
> 
> Hi,
> 
>> the build is actually hung. dmesg might have more info.
> 
> So having following in buildbot log:
> 
> 2023-06-01 23:53:12+0000 [-] command timed out: 3600 seconds without output running [b'make', b'-j7', b'IGNORE_ERRORS=n m y', b'BUILD_LOG=1', b'CONFIG_AUTOREMOVE=y', b'CONFIG_SIGNED_PACKAGES='], attempting to kill
> 2023-06-01 23:53:12+0000 [-] trying to kill process group 1528179
> 
> I've looked at the system logs around that time and found following:
> 
> Jun 01 22:23:19 audit[3844576]: AVC apparmor="DENIED" operation="mkdir" info="Failed name lookup - name too long"
>                 error=-36 profile="docker-default"
> 		 name="/shared-workdir/build/sdk/build_dir/hostpkg/gettext-0.21.1/gettext-tools/confdir3/confdir3/confdir3/confdir3...[snip very long repeating pattern]...
> 		 confdir3/confdir3/confdir3/confdir3/confdir3" pid=3844576 comm="conftest" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000
> Jun 01 22:23:45 kernel: conftest[3855174]: segfault at 0 ip 00007fe9581067e7 sp 00007ffd94ca2118 error 4 in libc-2.31.so[7fe958085000+159000]                  
> Jun 01 22:23:45 kernel: Code: 00 00 00 48 39 d1 0f 82 47 67 06 00 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 10 0f 82 fa 00 00 00 48 83 fa 20 0f 87 3f 01 00 00 <0f> 10 06 0f 10 4c 16 f0 0f 11 07 0f 11 4c 17 f0 c3 0f 1f 84 00 00
> 
> Since the host is shared with other 3 build workers I can't be sure, that it
> originated from that timeouted build.

Whether or not this was the cause of the timeout, it still looks suspicious.
Then again, it may also be a side effect of running out of space (possibly a looped symlink?)

The timeout logs aren’t available any more but on second thought I wonder if they too could simply be a side effect of OOS.

>> I’m happy to share my config if interested, it’s not a very complex setup.
> 
> That would be appreciated, thanks!

Will followup in separate channel.

Cheers,
T