[OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys and wyrding timeouts largest firmwares

[Hannu Nyman]

Buildbot needs some maintenance, as some of the buildslaves are behaving 
badly :-(

1)
gb15 has a wrong host key and silently fails all uploads. It just completed 4 
builds successfully, but did not upload them. That has been going on for some 
time now.

Build status green, but upload has failed:
http://buildbot.openwrt.org:8010/builders/brcm47xx/builds/123
http://buildbot.openwrt.org:8010/builders/brcm47xx/builds/123/steps/shell_11/logs/stdio
http://buildbot.openwrt.org:8010/builders/x86.kvm_guest/builds/113/steps/shell_11/logs/stdio

   Upload Snapshot to Openwrt
   Host key verification failed.
   rsync: connection unexpectedly closed (0 bytes received so far) [sender]
   rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]
   Host key verification failed.

Somebody should fix the ssh host key (or whatever the rdate uses at the 
update step).

It is rather sad that the buildbot script silently fails the binary uploads, 
because that makes the error hard to notice. The build shows green on the 
status screen, so unless you wonder about the missing binaries of a 
successful build and really check the upload step's log file, you won't 
notice the error. It would be great if the buildbot upload script could be 
modified to show the upload error more clearly.

2)
wyrding timeouts the imagebuilder/SDK tar file creation step for the largest 
firmwares. I have noticed that as ar71xx has suffered from that several times 
in the last few weeks. The 20 minutes given for waiting for any output from 
the process is sometimes too short for the tar file creation step and the 
buildbot breaks the build. Apparently "tar" does not produce any intermediate 
output.  (When that timeout happens, the file cleanup step may also take 
hours, so there may be something wrong with the buildslave's disk or I/O in 
general.)

Example:
http://buildbot.openwrt.org:8010/builders/ar71xx/builds/103
http://buildbot.openwrt.org:8010/builders/ar71xx/builds/103/steps/compile_8/logs/stdio
http://buildbot.openwrt.org:8010/builders/ar71xx/builds/97/steps/compile_8/logs/stdio
  The cleanup took 1 h 56 min build build 97:
       shell_15 './cleanup.sh wyrding ...' ( 1 hrs, 56 mins, 24 secs )
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel