Heads up: Fakeroot package catastrophe has removed most of master snapshot package downloads

[Daniel Golle]

On Tue, Sep 01, 2020 at 06:13:53PM +0300, Hannu Nyman wrote:
> Heads up, the usage of the new fakeroot package has failed in a major way in
> buildbot, and the phase2 packages buildbot has failed to build any package
> for 1-2 days.

Yes, it has been a busy night once I had realized what was going on...

> 
> That has caused ALL master snapshot downloadable packages to be removed from
> the download server for most package archs. The download dirs currently
> contain just the empty indices.
> 
> That will likely cause a flood of bug reports, forum discussions etc., as no
> casual user can download add-on packages for master until buildbot has
> rebuilt the packages and uploaded them to the server, package arch by arch.
> That will likely take several days until the last arch is done.  I noticed
> this after there were forum complaints away missing LuCI downloads with
> opkg.
> 
> 
> As of now, I am not sure if the problem has been fixed. The most recent
> phase2 builds still show empty download directories. But that is possibly
> due to package builds made with faulty SDKs from the phase1 images
> buildbot.  (And nbd just committed a MacOS fix a few minutes ago, so there
> are apparently effects also for private build envs...)
> 
> 
> The havoc has been most likely caused by commit "build: ipkg-build use
> fakeroot with PKG_FILE_MODES"
> 
> https://git.openwrt.org/?p=openwrt/openwrt.git;a=commitdiff;h=353ce2e521f57318c502c862457dc859d8d181a6;hp=2bd55d0a2b22e5923a433edd9f90be8ea2997bde
> 
> 
> Fix attempts were made by
> 
> https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=5587f19c362a805636da91d9d550928b4a2998fc
> 
> and
> 
> https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=729a75c3b2db0aa25745e8db425711cc761fa1f6
> 
> but those have effect in the phase2 packages build only after the build
> happens with the fixed/updated SDK from phase1, which fix is still waiting
> to be built for many targets.

Exactly. First builds with fixed SDK now seem to be ok:
https://buildbot.openwrt.org/master/packages/builders/arm_cortex-a7/builds/250
https://downloads.openwrt.org/snapshots/packages/arm_cortex-a7/base/

> 
> 
> Hopefully things have been fixed, but the rebuild of the packages will
> really take several days, maybe up to a week.

Looks like most phase1 builds have already run through with the fixes
included. Still waiting are archs38/generic, armvirt/64, lantiq/xway,
ramips/mt76x8, sunxi/cortexa7, x86/64 and zynq/generic. I expect them
to run through during the next hours (currently 5 parallel builds, each
run takes roughly 1.5h).

Most importantly, ath79/generic was one of the first to be built again
and mips24kc packages should return with the next phase2 run.
For all phase2 builds to run through it will take a day...