feat(zkevm): support external t8n stateless input/output byte generation

[jsign]

🗒️ Description

zkEVM added three important fields to fixtures: executionWitness, statelessInputBytes and statelessOutputBytes. The last two are the main crucial ones since are the raw bytes received by the stateless validator (i.e. guest programs) to validate a block (and thus generate a proof).

Currently, we can fill test fixtures but not benchmark fixtures -- mainly because filling benchmarks with the Python spec is not possible given that these are quite heavy blocks and is not doable. This isn't something particular to zkEVM efforts, but a general one (e.g. for Glamsterdam efforts, benchmarks aren't filled with Python either).

This means that benchmark fixtures should be filled with an external t8n implementation, usually based on some real EL that is performant enough. Geth is currently one of the main clients used in EEST for this purpose.

This PR:

• Allows to fill any fixture (in particular, benchmark fixtures) using an external t8n tool with a minimal requirement that executionWitness is returned as t8n output (not necessarily stateless{Input|Output}Bytes)
• The stateless{Input|Ouput}Bytes are constructed in the testing framework.

Ideally, the t8n should also return the stateless{Input| Output}Bytes, and it would make this PR a simpler (i.e. basically almost no diff at all), but currently Geth doesn't have good SSZ support to do this. But, it has good support for executionWitness generation, thus still doing the SSZ-work on this side. Eventually, we could push stateless{Input|Output}Bytes generation to Geth-t8n, and remove this on EEST side, which would be nice.

I also added a GH workflow that does a quick smoke test, checking that a particular benchmark fixture target filled with Python and with Geth leads to exactly the same output. See here for a run.

With this change, we could create zkEVM benchmark releases. These would target spec-compliant guest programs, plus also on the latest devnet target.

🔗 Related Issues or PRs

N/A.

✅ Checklist

• All: Ran fast static checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  just static

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Considered updating the online docs in the ./docs/ directory.
• All: Set appropriate labels for the changes (only maintainers can apply labels).
• Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
• Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.
• Ported Tests: All converted JSON/YML tests from ethereum/tests or tests/static have been assigned @ported_from marker.

Cute Animal Picture

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.26%. Comparing base (8d7e68f) to head (1ebc77f).

Additional details and impacted files

@@               Coverage Diff               @@
##           projects/zkevm    #2889   +/-   ##
===============================================
  Coverage           87.26%   87.26%           
===============================================
  Files                 600      600           
  Lines               36968    36968           
  Branches             3550     3550           
===============================================
  Hits                32259    32259           
  Misses               3975     3975           
  Partials              734      734           

Flag	Coverage Δ
unittests	87.26% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jsign]

I'm using these particular commit references since they are also used in other GH workflows.

[jsign]

This is just a file that fills ~150 fixtures -- feels like a light enough signal to compare Geth and Python are filling the same thing.

I think this should actually be a workflow useful for forks/amsterdam -- but we could think about upstreaming eventually.

[jsign]

I'm using 1M gas value to make it easy for Python filler. I think with this we get the signal we need reg fixture matching. Also, higher gas limit won't really change much executionWitness/stateless{Input|Output}Bytes mismatchs if they exist.

[jsign]

Of course, I had to do some work in Geth to:

• Be aware of the existing new t8n-input BlockHeaders, since it is required for headers execution witness construction.
• Add ExecutionWitness and BAL fields as part of t8n-output

Nothing new here reg t8n API definition -- is just making Geth be aware of these new optional fields and use/return them correctly.

It also has an important fix for correct execution witness generation, but that's something more related to Geth than EEST. I'm planning to eventually create the PR in geth, so we can try using Geth for this without my fork.

[jsign]

If hte t8n output does have stateless_input_bytes and stateless_output_bytes, then we are probably filling with Python t8n so "nothing new is done".

But if that isn't the case, then we are probably using t8n output from Geth which only returns executionWitness. What build_amsterdam_stateless_artifacts_from_t8n(...) does is constructing StatelessInput and StatelessValidationResult from parts, and doing the SSZ serialization to get the stateless{Input|Output}Bytes.

If at some point Geth supports returning stateless{Input|Output}Bytes in the t8n output, we can basically remove all the diffs in this file.

[jsign]

For now we are only going to use Geth for filling benchmarks which are always valid blocks.

If Geth adds the successful_validation boolean t8n-output, we could remove this todo. But to do that meaningfully, it requires some extra changes in Geth. We should try to do it, but it isn't worth being blocked on filling benchmark tests because of this.