Proposal: manage all the source code in one repo

[luchsh]

Background

The CompoundVM JDK was built from source code of two repositories: modified jdk17u (aka this-repo), unmodified jdk8u. and the jdk8u source code was downloaded using wget at build time, see cvm.mk:150.

This approach has two drawbacks:

• the external dependency may be unstable, affected by network, permission, etc.;
• it would be hard to maintain changes in the downloaded package;

Proposal

This proposal tries to maintain all the source code in one repository (this one).

Design

multiple jdk*u upstreams

• add different upstreams using git remote, below commands use jdk8u as example.

# add the jdk8u remote
cd compoundVM/
git remote add jdk8u https://github.com/openjdk/jdk8u.git
git fetch jdk8u

# create a local branch to track jdk8u/master
git branch jdk8u_master -t jdk8u/master

# set up a new worktree in sub-dir cvm/jdk8u
git worktree add cvm/jdk8u jdk8u_master

Branch jdk8u/master has many shared history nodes with CompoundVM/master, thus no need to duplicate the whole git history tree.

• since CVM project has got extra patches on upstream branches, we must create a CVM-jdk*u for each upstream

git checkout -b cvm8/jdk8u jdk8u/master
# add our patches

the new cvm master

• create a new cvm master branch with no OpenJDK commit logs

git checkout jdk17u-target8
git checkout --orphan cvm8/master
rm -rf ADDITIONAL_LICENSE_INFO ASSEMBLY_EXCEPTION configure CONTRIBUTING.md doc make Makefile SECURITY.md src bin .jcheck
mv cvm/* ./
rmdir cvm
git add *
git commit -m 'initial load the cvm8/master branch'

accomodate multiple worktrees

• Modify cvm.mk (now renamed to Makefile) to checkout the upstream branches into different worktrees at build time.

-jdk8u/jdk/src:
-       wget -nc https://github.com/openjdk/jdk8u/archive/refs/tags/jdk8u452-ga.tar.gz
-       [[ -d $(JDK8_SRCROOT) ]] || (mkdir -p $(JDK8_SRCROOT) && tar -xzf jdk8u452-ga.tar.gz -C $(JDK8_SRCROOT) --strip-components=1)
+$(JDK8_SRCROOT)/jdk/src:
+       git worktree add $(JDK8_SRCROOT) cvm8/jdk8u
+
+$(JDK17_SRCROOT)/src:
+       git worktree add $(JDK17_SRCROOT) cvm8/jdk17u

Overhead

Repository size with/without multiple remotes

About 250 MB size increase in dir .git/ after adding one jdk8u remote in current repo.

repo content	size of .git/
CompoundVM now (c777c7d)	1226 MB
plus jdk8u remote	1476 MB

Demo

I've created a demo in my forked repo, steps to play with this demo:

git clone https://github.com/luchsh/CompoundVM.git cvm_test
cd cvm_test
make cvm8

[luchsh]

After an offline discussion, a progressive transition proposal is here.

key changes:

• use git worktree to manage jdk8u code in a separate branch and checked out in directory cvm/jdk8u;
• no overlay mechanisms any more;
• keep existing cvm code in the jdk17u repo as top level repository.

directories and repositories

dirs	repos
./	jdk17u-target8
./cvm/jdk8u/	jdk8u

Demo

• url: CompoundVM layout v2
• branch: dev/v2/jdk17u-target8