test(generate): e2e integration tests + gap report for generate sbom (depends on #291)#294
Closed
riteshnoronha wants to merge 2 commits intomainfrom
Closed
test(generate): e2e integration tests + gap report for generate sbom (depends on #291)#294riteshnoronha wants to merge 2 commits intomainfrom
generate sbom (depends on #291)#294riteshnoronha wants to merge 2 commits intomainfrom
Conversation
Adds 26 testscript scenarios under e2e/testdata/generate_sbom/ covering every feature documented in docs/spec/generate-sbom.md, plus a developer-facing findings doc (GAPS.md) and a scenario-to-gap map (REPORT.md). Depends on the `generate sbom` implementation from #291. Test-only change. Scenarios pin current behaviour via negated assertions where the implementation diverges from the spec, so the suite passes against #291's branch and starts failing when a gap is fixed. GAPS.md lists all 14 divergences with reproductions, actual vs expected output, and source-level fix hints. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Keep the test scenarios only; the findings docs are shared out-of-band. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test-only PR. Adds e2e integration coverage for the
sbomasm generate sbomfeature and a developer-facing gap report. Depends on #291 for the
implementation; merge only after #291 lands.
Every file added lives under
e2e/. No source-code changes.Files
e2e/generate_sbom_test.go- testscript entry point.e2e/testdata/generate_sbom/01..26_*.txt- 26 scenarios, one per featurecluster in
docs/spec/generate-sbom.md.e2e/testdata/generate_sbom/GAPS.md- developer-facing findings. 14 specdivergences with spec quote,
file:line, reproducible input, actualoutput, expected output, fix hint.
e2e/testdata/generate_sbom/REPORT.md- scenario-to-gap map and runinstructions.
Spec gaps flagged in GAPS.md
.artifact-metadata.yamloutput:block is parsed but ignored.silent-skip from hard-error).
warnings.
name@version; spec requirespkg:generic/{name}@{version}.SOURCE_DATE_EPOCHis not honored.tags.
scope: excludedcomponents are not dropped.platform,device-driver,machine-learning-model,data) are silently downgraded.{id},{id,text},{id,file}) fail JSON parse.hash_filecolumn is ignored.scope.NTIA scores (sbomqs)
Against a well-formed manifest generated from #291:
The CDX/SPDX delta is driven by gaps 13 and 14.
How the scenarios work
Every scenario passes when run against #291's branch. Spec divergences are
pinned via negated assertions (e.g.
! cmp run1.json run2.jsonfor thedeterminism gap), so the suite documents current behaviour. When a gap is
fixed, the negation flips and the scenario starts failing until it is
rewritten for the spec-correct form. This turns each gap into a trip-wire.
Test plan
main.go test ./e2e/ -run TestSbomasmGenerateSBOM -count=1 -vpasses(26/26).
sbomqsandjqare onPATHfor scenarios 24 and 26 (install:go install github.com/interlynk-io/sbomqs@latest).the owning scenario from
! ...to....CI expectation
Until #291 merges, the
generate sbomsubcommand does not exist onmain,so
TestSbomasmGenerateSBOMwill fail every scenario with "unknowncommand." That is expected; CI should go green as soon as this is rebased
onto a main that contains #291.
🤖 Generated with Claude Code