[PATCH] scripts: create kernel configuration upgrade script

[Stijn Segers]

Hi Elliott,

Op zondag 3 maart 2024 om 15:24:50 -08:00:00 schreef Elliott Mitchell 
<ehem+openwrt at m5p.com>:
> Date: Tue, 6 Feb 2024 17:16:41 -0800
> 
> Create a script for automating kernel version changes.  This
> generates a pair of commits which cause history to remain attached to
> all versioned configuration files.
> 
> Crucially this makes `git blame` work without needing
> --find-copies-harder, which is too slow for routine use.  This also
> updates *everything*, which greatly simplifies rebasing patches
> which effect multiple devices.
> 
> Credit to Christian Marangi who knew of the technique:
> <https://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041672.html>
> 
> Signed-off-by: Elliott Mitchell <ehem+openwrt at m5p.com>


Is there a way to bump a specific target to a new kernel with your 
script? It doesn't look like it, but I might be mistaken. Would it be a 
lot of work to integrate that? With the shell equivalent being merged, 
I can understand any reluctance to adding this functionality, but it's 
worth asking.

Thanks

Stijn


> ---
> v3:
> Dust off knowledge of PerlOO.  Confine the fast-importer interface
> to an object.
> 
> Better layer the lowest I/O layer.  If fast-import grows a \0 command
> separation mode, we should be mostly ready (issue will be the commit
> message).
> 
> Switch to SPDX.  Try to match what the other scripts have.
> 
> I was kind of hoping for more review activity, the near-silence is
> almost deafening.  Using a script to handle this job seems best.  I
> feel what this script produces is rather easier for most developers
> to handle.
> 
> v2:
> Major tweaking.  No longer try to do `git merge --ff-only <hash>`,
> but instead advise user to do so.
> 
> Add usage message.
> 
> Add statement about strategy.
> 
> Adjust commit messages.  Add advice to run `git bisect skip` if
> someone ends up on that commit.
> ---
>  scripts/kernel_upgrade.pl | 280 
> ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 280 insertions(+)
>  create mode 100755 scripts/kernel_upgrade.pl
> 
> diff --git a/scripts/kernel_upgrade.pl b/scripts/kernel_upgrade.pl
> new file mode 100755
> index 0000000000..c2e5fb6078
> --- /dev/null
> +++ b/scripts/kernel_upgrade.pl
> @@ -0,0 +1,280 @@
> +#!/usr/bin/env perl
> +#########################################################################
> +# SPDX-License-Identifier: GPL-3.0-or-later				#
> +#									#
> +# Copyright (C) 2024 Elliott Mitchell <ehem+openwrt at m5p.com>		#
> +#########################################################################
> +
> +use warnings;
> +use strict;
> +
> +#
> +# I'm not certain the technique originated here, but this comes from:
> +# <https://devblogs.microsoft.com/oldnewthing/20190919-00/?p=102904>
> +#
> +# Problem is copying a file in git causes the new file to be created
> +# without any history.  Files can move around without losing their
> +# history, but that only leaves the history on the new location.
> +#
> +# As such this can be solved with two commits.  The first commit 
> moves
> +# files from their old name to their new name.  The second merges the
> +# original commit with the rename commit.  The merge commit then has
> +# files in both locations with the full history.
> +#
> +#
> +# Note, git handles discarded data by garbage collection.  When doing
> +# development on this script, beware this script is an excellent
> +# garbage generator.  Frequent use of `git gc` and `git prune` may be
> +# needed.
> +#
> +
> +
> +sub gethead()
> +{
> +	open(my $fd, '-|', 'git', 'rev-parse', 'HEAD');
> +	$_=<$fd>;
> +	chop;
> +	return $_;
> +}
> +
> +sub getlist($$)
> +{
> +	my ($target, $from)=@_;
> +	my $ret=[];
> +
> +	local $/="\0";
> +	open(my $fd, '-| :raw :bytes', 'git', 'ls-tree', '-trz', 
> '--full-name',
> +'--name-only', 'HEAD', '--', $target)||die("failed to read git 
> tree");
> +
> +	while(<$fd>) {
> +		chop($_);
> +		push(@$ret, substr($_, 0, -length($from)))
> +if(substr($_, -length($from)) eq $from);
> +	}
> +
> +	@$ret=sort({length($b)-length($a)} @$ret);
> +
> +	return $ret;
> +}
> +
> +{ # start of interface to git fast-import
> +package GitImporter;
> +
> +# git fast-import's protocol uses *linefeeds*
> +local $/="\n";
> +
> +sub new()
> +{
> +	my $class=shift;
> +	my $self={};
> +	my @child;
> +	(pipe($child[0], $self->{out})&&pipe($self->{in}, $child[1])) ||
> +die("pipe() failed");
> +	binmode($self->{out});
> +	binmode($self->{in});
> +
> +	$self->{pid}=fork();
> +	if($self->{pid}==0) {
> +		close($self->{out});
> +		close($self->{in});
> +
> +		open(STDIN, '<&', $child[0]);
> +		close($child[0]);
> +
> +		open(STDOUT, '>&', $child[1]);
> +		close($child[1]);
> +
> +		exec('git', 'fast-import', '--done');
> +		die('exec() of git failed');
> +	} elsif(!$self->{pid}) {
> +		die('fork() failed');
> +	}
> +	close($child[0]);
> +	close($child[1]);
> +	$self->{out}->autoflush(1);
> +
> +	return bless($self, $class);
> +}
> +
> +sub send($)
> +{
> +	my $self=shift;
> +	my ($data)=@_;
> +	return print({$self->{out}} $data);
> +}
> +
> +sub putcmd($)
> +{
> +	my $self=shift;
> +	my ($data)=@_;
> +	return $self->send("$data\n");
> +}
> +
> +sub recv()
> +{
> +	my $self=shift;
> +	return $_=readline($self->{in});
> +}
> +
> +sub getres()
> +{
> +	my $self=shift;
> +	$_=$self->recv();
> +	chomp;
> +	return $_;
> +}
> +
> +sub ls($$)
> +{
> +	my $self=shift;
> +	my ($commit, $name)=@_;
> +
> +	$commit.=' ' if($commit);
> +	$self->putcmd("ls $commit$name");
> +	$self->getres();
> +
> +	die('git ls failed') 
> unless(/^([0-8]+)\s+[a-z]+\s+([0-9a-z]+)\s+.+$/);
> +
> +	return [$1, $2];
> +}
> +
> +sub commit($$$)
> +{
> +	my $self=shift;
> +	my ($dest, $message, $mark)=@_;
> +
> +	use feature 'state';
> +	use Digest::SHA qw(sha1_hex);
> +
> +	state $author=undef;
> +	unless($author) {
> +		$author=['', ''];
> +		open(my $user, '-|', 'git', 'config', '--get', 'user.name');
> +		while(<$user>) {
> +			chomp;
> +			$author->[0].=$_;
> +		}
> +		$author->[0]=[split(/,/, [getpwuid($<)]->[6])]->[0]
> +unless($author->[0]);
> +
> +		open(my $email, '-|', 'git', 'config', '--get', 'user.email');
> +		while(<$email>) {
> +			chomp;
> +			$author->[1].=$_;
> +		}
> +		$author->[1]='anonymous at example.com' unless($author->[1]);
> +
> +		$author=$author->[0].' <'.$author->[1].'>';
> +	}
> +
> +	$_=sha1_hex(time());
> +	$self->putcmd("commit $_");
> +	$self->putcmd("mark $mark");
> +	$self->putcmd("committer $author ".time()." +0000");
> +
> +	$_=length($message);
> +	$self->putcmd("data $_");
> +	$self->send($message);
> +	$self->putcmd("from $dest");
> +}
> +
> +sub DESTROY()
> +{
> +	my $self=shift;
> +
> +	$self->putcmd('done');
> +	close($self->{out});
> +	delete $self->{out};
> +
> +	0 while(waitpid($self->{pid}, 0)!=$self->{pid});
> +	delete $self->{pid};
> +	close($self->{in});
> +	delete $self->{in};
> +
> +	print(STDERR <<~"__GIT_STATUS__") if($?);
> +	WARNING: git returned error exit status: $?
> +
> +	This likely means `git gc` needs to be run, but the return codes of
> +	`git fast-import` are undocumented.
> +
> +	__GIT_STATUS__
> +}
> +} # end of interface to git fast-import
> +
> +
> +die(<<"__USAGE__") if(@ARGV!=2);
> +Usage: $0 <old-version> <new-version>
> +
> +Copies all kernel configuration files and patches from the old 
> version
> +to the new version.  Git history is preserved on the copies by using 
> a
> +move/merge strategy.  Must be run while somewhere inside the git
> +repository directory, but it does not matter where.
> +__USAGE__
> +
> +my ($from, $to)=@ARGV;
> +
> +
> +my $target='target/linux';
> +
> +my $start=gethead();
> +
> +my $list=getlist($target, $from);
> +
> +die("no files matching \"$from\" found") unless(@$list);
> +
> +
> +my $git=GitImporter->new();
> +
> +$git->commit($start, <<"__TMP__", ':1');
> +kernel: add configs and patches for $to
> +
> +This is a special tool-generated commit.
> +
> +Copy configuration and patches from $from to $to.
> +
> +If you see this commit during a `git bisect` session, the recommended
> +course of action is to run `git bisect skip`.
> +__TMP__
> +
> +foreach my $name (@$list) {
> +	my $new=$git->ls($start, "$name$from");
> +	$git->putcmd("M $new->[0] $new->[1] $name$to");
> +	$git->putcmd("D $name$from");
> +}
> +$git->putcmd('');
> +
> +
> +$git->commit(':1', <<"__TMP__", ':2');
> +kernel: finish update from $from to $to
> +
> +This is a special tool-generated commit.
> +
> +Merge the add commit into HEAD to create all files with full history.
> +__TMP__
> +
> +$git->putcmd("merge $start");
> +
> +foreach my $name (@$list) {
> +	my $new=$git->ls($start, "$name$from");
> +	$git->putcmd("M $new->[0] $new->[1] $name$from");
> +}
> +$git->putcmd('');
> +
> +
> +$git->putcmd('get-mark :2');
> +my $result=$git->getres();
> +
> +undef($git);
> +
> +print(<<"__END__");
> +Result is commit $result.
> +
> +Depending on the setup of your development environment, you now 
> likely
> +want to run one of two commands:
> +
> +	`git merge --ff-only $result`
> +or:
> +	`git branch linux-$to $result`
> +__END__
> +
> +exit(0);
> --
> 2.39.2
> 
> 
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel