download: regression with USE_SOURCE_DIR and non-tarball packages?

[Lukas Zeller]

Hello Petr,

a while ago (after switching projects from 19.07.3 to 19.07.6) I experienced unexpected rebuilds of packages. I asked about this in the forum [1] with no echo, so last week I dug deeper and think I found your commit from last november "4e19cbc - download: handle possibly invalid local tarballs" [2] causing what I see.

I'm a build system newbie, so please apologize if I misunderstand things here. But it seems to me that this patch only works for packages that pull their upstream source from a tarball via download.pl, because only there a hash check prevents redownloads (causing rebuilds) in the last possible moment by exiting the script before the download would happen.

For packages with PKG_SOURCE_PROTO=git the new line in package.mk `$(DL_DIR)/$(FILE): FORCE` forces re-cloning the upstream repo and thus rebuilding the entire package every time. Also, the USE_SOURCE_DIR mechanism does not work any more.

Both git based packages and USE_SOURCE_DIR are central to my work (which is building automation devices based on OpenWrt, so most work is happening *in* my own packages, which are in git or, during hot development phases, via USE_SOURCE_DIR even directly linked dirs). I realize this is not a common situation for OpenWrt platform developers, but only for developers basing other stuff on OpenWrt.

As said, I don't understand the details of the build system enough to actually propose a proper patch right into the heart of it. But below is my current workaround to re-enable USE_SOURCE_DIR and git-based packages, hopefully not affecting anything else I can't see. So far, works for me, but definitely needs a thorough look by someone who actually understands the whole thing...

Lukas


[1] https://forum.openwrt.org/t/does-19-07-6-trigger-package-downloads-rebuilds-more-frequently-than-19-07-3/92471
[2] http://lists.openwrt.org/pipermail/openwrt-devel/2020-November/032235.html


--- a/include/package.mk
+++ b/include/package.mk
@@ -177,6 +177,12 @@ define Build/Exports/Default
 endef
 Build/Exports=$(Build/Exports/Default)

+ifneq ($(wildcard $(PKG_BUILD_DIR)/.source_dir),)
+  FORCE_DL:=
+else
+  FORCE_DL=$(if $(findstring git,$(PKG_SOURCE_PROTO)),,$(DL_DIR)/$(FILE): FORCE)
+endif
+
 define Build/CoreTargets
   STAMP_PREPARED:=$$(STAMP_PREPARED)
   STAMP_CONFIGURED:=$$(STAMP_CONFIGURED)
@@ -185,7 +191,7 @@ define Build/CoreTargets
   $(call Build/Autoclean)
   $(call DefaultTargets)

-  $(DL_DIR)/$(FILE): FORCE
+  $(FORCE_DL)

   download:
 	$(foreach hook,$(Hooks/Download),

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.openwrt.org/pipermail/openwrt-devel/attachments/20210614/03a82c07/attachment.sig>