Add Git submodules remote tracking

[neutronth]

Add Git submodules remote tracking

• Always update the registered submodules using the remote-tracking branch
  instead of the superproject's recorded SHA-1.
  Only apply to the submodules' name that listed in the
  --submodules-remote-tracking parameter, otherwise default behavior will be applied.

  • Use case:
    Update the latest registered configurations submodule from remote repository
  • git-sync parameter:
    ./git-sync ... --submodules-remote-tracking="deployment-configs,app-configs"

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[thockin]

/lifecycle frozen
/remove-lifecycle stale
/remove-lifecycle rotten

[thockin]

@nirutgupta

I finally got around to looking at this, and I think it is worth discussing the design, since I do not use this feature of git myself.

Should it be fundamentally async to the main sync loop? The current PR seems to be, but this is a problem because such updates are not atomic, meaning a consumer could see the repo in an inconsistent state. I think, for sanity, we need to do it all in-phase with the main sync loop, and do it in a new worktree.

This is an issue because there's a sort of contract that the basename of the thing the link points to is the hash that is checked out. If the remote tracking needs to update, it could be the same "main" hash. We could maybe do some indiana-jones-swapping-the-idol trickery to retain this property, creating a new worktree and playing with links, like:

thockin-glaptop4 root c017f e2e-branch /$ git worktree add --detach /tmp/git-sync-e2e.2603115845/root/c017f5ae9aab079a210fec80004b16369b9a364d.old c017f5ae9aab079a210fec80004b16369b9a364d
Preparing worktree (detached HEAD c017f5a)
HEAD is now at c017f5a add submodule3

thockin-glaptop4 root c017f e2e-branch /$ ls -l
total 4
drwxr-xr-x 5 thockin primarygroup 4096 Nov 26 11:00 c017f5ae9aab079a210fec80004b16369b9a364d
drwxr-xr-x 5 thockin primarygroup 4096 Nov 26 11:28 c017f5ae9aab079a210fec80004b16369b9a364d.old
lrwxrwxrwx 1 thockin primarygroup   41 Nov 26 11:20 link -> c017f5ae9aab079a210fec80004b16369b9a364d/

thockin-glaptop4 root c017f e2e-branch /$ ls link
file  sub1  sub2  sub3

thockin-glaptop4 root c017f e2e-branch /$ ln -s c017f5ae9aab079a210fec80004b16369b9a364d.old link.old

thockin-glaptop4 root c017f e2e-branch /$ mkdir _old

thockin-glaptop4 root c017f e2e-branch /$ ln -s ../link.old _old/c017f5ae9aab079a210fec80004b16369b9a364d

thockin-glaptop4 root c017f e2e-branch /$ ln -s _old/c017f5ae9aab079a210fec80004b16369b9a364d link.new

thockin-glaptop4 root c017f e2e-branch /$ mv -T link.new link

thockin-glaptop4 root c017f e2e-branch /$ ls -l link
lrwxrwxrwx 1 thockin primarygroup 45 Nov 26 11:22 link -> _old/c017f5ae9aab079a210fec80004b16369b9a364d

thockin-glaptop4 root c017f e2e-branch /$ ls link
file  sub1  sub2  sub3

We still need to figure out for each remote tracking branch whether it needs to be updated before we do any of this work. So roughly:

check main repo for updates
check each tracking sub for updates (recursively?)
if any of them need update, do a full sync

Almost certainly needs some internal refactoring to accomodate.

Then we need to consider impact on hooks - do we re-run webhook and exechook notifiers? I think we have to, but they only get the hash of the main repo as an argument. We may need to change that, too.

There may be more to consider, but this is a start :)

[jesperbagge]

Hi @thockin!

check each tracking sub for updates (recursively?)
if any of them need update, do a full sync

Isn't this exactly what git submodule update --remote --rebase does?

[thockin]

Git-sync tries hard to not do non-atomic things which might show an observer data which is neither completely one SHA nor the other. This is what worktree offers.

Doing remote tracking while keeping this property would require treating each submodule as a worktree, or else triggering a new top-level worktree when any submodule is updated. Or we lose the atomic property.

I am sure we COULD do it, but it would be a fair amount of complexity to carry for a very small set of use-cases. I also worry that it's not repeatable - checking out the same rev of the main repo on two different days gets you two different results. Maybe less important since it is opt-in.

I am not saying no to this, but it needs a person to drive. I haven't got the time in the foreseeable future, nor the use-cases or expertise to even explore it fully .

[jesperbagge]

I completely understand your concerns about keeping Git-sync atomic and I fully respect your points.

I'm coming from the Apache Airflow DAG management point-of-view, trying to figure out how to have an umbrella repo with submodules pointing to several DAG repositories, outlined here. From my stance I have no concerns about atomicity. I just want the last commit of a specific branch, say main in all submodules respectively to always be synced into my umbrella repo automatically.

It would be nice to opt-in to this sort of behavior by setting the --remote flag via $GITSYNC_SUBMODULES or something similar.

[jhgoebbert]

@jesperbagge This is exactly the same use-case on our side with Apache Airflow DAG management.
Did you find a nice solution? Can you share it?

[jesperbagge]

Hi @jhgoebbert Yes, I did find a solution to my Airflow DAG management issues, in the end. On a high level overview I disabled Airflows own git-sync settings in the Helm chart and started two additional containers in the same namespace, instead. Each container is pulling a git-sync image that syncs a repo into a persistent volume that both the git-sync container and the airflow containers have mounted. I have all my DAGs in one volume and the entire dbt Core repo in the other volume.

[jhgoebbert]

Thank you for sharing. 👍 I will then probably try to introduce a custom git-sync image and add the logic there.