Skip to content

Refactor CI to run in fresh build box#1000

Open
fdupress wants to merge 1 commit intomainfrom
ci-in-fresh-build-box
Open

Refactor CI to run in fresh build box#1000
fdupress wants to merge 1 commit intomainfrom
ci-in-fresh-build-box

Conversation

@fdupress
Copy link
Copy Markdown
Member

@fdupress fdupress commented May 7, 2026

This refactors the CI jobs so compilation and checks (including external CI) run in a fresh build box. This will help ensure that changes to the box configuration (new dependencies, prover updates, prover additions, ...) are reflected quickly in the CI environment and can be leveraged in external proofs right away.

The workflow is now:

  1. Build base and build boxes, then save them as an artefact;
  2. Recover the build box, then run the compilation check;
  3. Recover the build box, then compile EasyCrypt and run a matrix check; (this recompiles EC 3 times in parallel, and likely needs changed)
  4. Recover the build box, then compile EasyCrypt and run the external matrix checks (this recompiles EC as many times as we have external checks, and likely needs changed)
  5. Build the test box, push the base and build boxes (all pushes) and the test box (release branches and tags)

The nix compilation check is unchanged, notification fires after all succeed. Documentation is built as normal.

@fdupress fdupress self-assigned this May 7, 2026
@fdupress fdupress added the chore Ungrateful tasks that need done but that nobody wants to do label May 7, 2026
@fdupress
Copy link
Copy Markdown
Member Author

fdupress commented May 8, 2026

OK. So this takes around the same time as the multiple workflows, but is much better in terms of actual checking.

Should I take the opportunity to try and avoid repeating work? I think we could:

  1. Push the base and build boxes in phase 1 under a temporary tag (with short retention);
  2. Build and push the test box in phase 2 (after compilation check with CI profile, rebuild with prod profile in the docker) under a temporary tag (with short retention);
  3. Use the test box in phase 3 to avoid the multiple parallel recompilations of EC;
  4. In phase 4, simply pull, retag, and push the appropriate boxes with the appropriate retention policy (indefinite for releases, attempting to keep only the latest for main).

The above would still have a minor race condition where we build the test box after the base and build boxes, so there could still be some small discrepancies in those environments if debian pushes some package updates between the two builds. I could also just build all 3 boxes in step 1 (4 if we keep the formosa images) before we check that compilation succeeds in the stricter CI profile.

@fdupress fdupress force-pushed the ci-in-fresh-build-box branch 3 times, most recently from daa33ee to 161ef7c Compare May 8, 2026 10:47
@fdupress fdupress force-pushed the ci-in-fresh-build-box branch 4 times, most recently from 3deac1e to 53efb8b Compare May 8, 2026 13:35
@fdupress
Copy link
Copy Markdown
Member Author

fdupress commented May 8, 2026

OK. So this takes around the same time as the multiple workflows, but is much better in terms of actual checking.

Should I take the opportunity to try and avoid repeating work? I think we could:

  1. Push the base and build boxes in phase 1 under a temporary tag (with short retention);
  2. Build and push the test box in phase 2 (after compilation check with CI profile, rebuild with prod profile in the docker) under a temporary tag (with short retention);
  3. Use the test box in phase 3 to avoid the multiple parallel recompilations of EC;
  4. In phase 4, simply pull, retag, and push the appropriate boxes with the appropriate retention policy (indefinite for releases, attempting to keep only the latest for main).

The above would still have a minor race condition where we build the test box after the base and build boxes, so there could still be some small discrepancies in those environments if debian pushes some package updates between the two builds. I could also just build all 3 boxes in step 1 (4 if we keep the formosa images) before we check that compilation succeeds in the stricter CI profile.

OK. So this takes around the same time as the multiple workflows, but is much better in terms of actual checking.

Should I take the opportunity to try and avoid repeating work? I think we could:

  1. Push the base and build boxes in phase 1 under a temporary tag (with short retention);
  2. Build and push the test box in phase 2 (after compilation check with CI profile, rebuild with prod profile in the docker) under a temporary tag (with short retention);
  3. Use the test box in phase 3 to avoid the multiple parallel recompilations of EC;
  4. In phase 4, simply pull, retag, and push the appropriate boxes with the appropriate retention policy (indefinite for releases, attempting to keep only the latest for main).

The above would still have a minor race condition where we build the test box after the base and build boxes, so there could still be some small discrepancies in those environments if debian pushes some package updates between the two builds. I could also just build all 3 boxes in step 1 (4 if we keep the formosa images) before we check that compilation succeeds in the stricter CI profile.

This is too much: the test Docker builds easycrypt from a git remote (via opam), which will not properly exist except in PR-related runs. Therefore, this should now be ready to review.

@fdupress fdupress marked this pull request as ready for review May 8, 2026 13:37
@fdupress fdupress requested a review from strub May 8, 2026 14:42
This refactors the CI jobs so compilation and checks (including external
CI) run in a fresh build box. This will help ensure that changes to the
box configuration (new dependencies, prover updates, prover additions,
...) are reflected quickly in the CI environment and can be leveraged in
external proofs right away.

The workflow is now:
1. Build base and build boxes, then save them as an artefact;
2. Recover the build box, then run the compilation check;
3. Recover the build box, then compile EasyCrypt and run a matrix check;
   (this recompiles EC 3 times in parallel, and likely needs changed)
4. Recover the build box, then compile EasyCrypt and run the external
   matrix checks (this recompiles EC as many times as we have external
   checks, and likely needs changed)
5. Build the test box, push the base and build boxes (all pushes) and
   the test box (release branches and tags)

The nix compilation check is unchanged, notification fires after all
succeed. Documentation is built as normal.
@fdupress fdupress force-pushed the ci-in-fresh-build-box branch from 53efb8b to 65fd21f Compare May 8, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Ungrateful tasks that need done but that nobody wants to do

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant