test(generate): integration tests + gap report for generate sbom (depends on #291)#293
Closed
riteshnoronha wants to merge 46 commits intomainfrom
Closed
test(generate): integration tests + gap report for generate sbom (depends on #291)#293riteshnoronha wants to merge 46 commits intomainfrom
generate sbom (depends on #291)#293riteshnoronha wants to merge 46 commits intomainfrom
Conversation
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Adds 26 testscript scenarios under e2e/testdata/generate_sbom/ covering every feature documented in docs/spec/generate-sbom.md, plus a developer-facing findings doc (GAPS.md) and a scenario-to-gap map (REPORT.md). Scenarios pin current behaviour via negated assertions where the implementation diverges from the spec, so the suite passes today and will start failing when a gap is fixed. GAPS.md lists all 14 divergences with reproductions, actual vs expected output, and source-level fix hints. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Reopening as test-only PR; implementation is owned by #291. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds integration-test coverage for
sbomasm generate sbom(PR #291) alongwith a developer-facing gap report. This PR is stacked on top of #291;
it exists so the developer of #291 can see a concrete, reproducible list of
spec divergences and a harness to verify fixes.
What's new (the only files you need to review)
All additions live under
e2e/:e2e/generate_sbom_test.go— testscript entry point (26 scenarios).e2e/testdata/generate_sbom/01..26_*.txt— one scenario per spec feature.e2e/testdata/generate_sbom/GAPS.md— start here if you are the Feat/support SBOM generation #291developer. 14 divergences, each with spec quote,
file:line, reproducibleinput, actual output, expected output, and a fix hint.
e2e/testdata/generate_sbom/REPORT.md— scenario → gap map, runinstructions, NTIA scores.
Because this branch is cut from #291's HEAD (
abfcad0), the diff againstmainalso includes everything that #291 adds. Filter the file view toe2e/**to see only the test additions.Spec gaps flagged
14 items documented in
GAPS.md. One-line summary:.artifact-metadata.yamloutput:block is parsed but ignored.skip from hard error).
warnings.
name@version; spec requirespkg:generic/{name}@{version}.SOURCE_DATE_EPOCHis not honored (timestamp + serial number).tags.
scope: excludedcomponents are not dropped.platform,device-driver,machine-learning-model,data) are silently downgraded.{id},{id,text},{id,file}) fail JSON parse.hash_filecolumn is ignored.scope.NTIA minimum-element scores (sbomqs)
Against a well-formed manifest:
The CDX/SPDX delta is driven by gaps 13 and 14.
How the scenarios work
Every scenario passes today. Spec divergences are encoded as negated
assertions (e.g.
! cmp run1.json run2.jsonfor the determinism gap), sothe suite documents the current behaviour. When a gap is fixed, the
negation flips and the scenario fails until it is rewritten for the
spec-correct form. This turns each gap into a trip-wire instead of
perma-red noise.
Test plan
go test ./e2e/ -run TestSbomasmGenerateSBOM -count=1 -vpasses (26/26).sbomqsandjqare onPATHfor the scenarios that use them(install:
go install github.com/interlynk-io/sbomqs@latest).PR also flips the negation to the positive assertion.
🤖 Generated with Claude Code