Skip to content

test(generate): e2e integration tests + gap report for generate sbom (depends on #291)#294

Closed
riteshnoronha wants to merge 2 commits intomainfrom
test/e2e-generate-sbom-only
Closed

test(generate): e2e integration tests + gap report for generate sbom (depends on #291)#294
riteshnoronha wants to merge 2 commits intomainfrom
test/e2e-generate-sbom-only

Conversation

@riteshnoronha
Copy link
Copy Markdown
Contributor

Summary

Test-only PR. Adds e2e integration coverage for the sbomasm generate sbom
feature and a developer-facing gap report. Depends on #291 for the
implementation; merge only after #291 lands.

Every file added lives under e2e/. No source-code changes.

Files

  • e2e/generate_sbom_test.go - testscript entry point.
  • e2e/testdata/generate_sbom/01..26_*.txt - 26 scenarios, one per feature
    cluster in docs/spec/generate-sbom.md.
  • e2e/testdata/generate_sbom/GAPS.md - developer-facing findings. 14 spec
    divergences with spec quote, file:line, reproducible input, actual
    output, expected output, fix hint.
  • e2e/testdata/generate_sbom/REPORT.md - scenario-to-gap map and run
    instructions.

Spec gaps flagged in GAPS.md

  1. .artifact-metadata.yaml output: block is parsed but ignored.
  2. Discovery errors uniformly "warn-and-continue" (spec distinguishes
    silent-skip from hard-error).
  3. Hash source errors (missing file, dir-as-file, empty match) are soft
    warnings.
  4. Pedigree/ancestor purl collision is a warning; spec requires hard error.
  5. Bom-ref fallback is name@version; spec requires
    pkg:generic/{name}@{version}.
  6. SOURCE_DATE_EPOCH is not honored.
  7. No stable ordering of components, dependencies, hashes, external refs,
    tags.
  8. scope: excluded components are not dropped.
  9. Four component types (platform, device-driver,
    machine-learning-model, data) are silently downgraded.
  10. License object forms ({id}, {id,text}, {id,file}) fail JSON parse.
  11. Strict library-supplier check has an operator precedence bug.
  12. CSV hash_file column is ignored.
  13. SPDX serializer emits no pedigree mapping.
  14. SPDX loses scope.

NTIA scores (sbomqs)

Against a well-formed manifest generated from #291:

  • CycloneDX 1.6: 8.8 / 10, grade B.
  • SPDX 2.3: 7.6 / 10, grade C.

The CDX/SPDX delta is driven by gaps 13 and 14.

How the scenarios work

Every scenario passes when run against #291's branch. Spec divergences are
pinned via negated assertions (e.g. ! cmp run1.json run2.json for the
determinism gap), so the suite documents current behaviour. When a gap is
fixed, the negation flips and the scenario starts failing until it is
rewritten for the spec-correct form. This turns each gap into a trip-wire.

Test plan

  • After Feat/support SBOM generation #291 merges, rebase this PR onto main.
  • go test ./e2e/ -run TestSbomasmGenerateSBOM -count=1 -v passes
    (26/26).
  • sbomqs and jq are on PATH for scenarios 24 and 26 (install:
    go install github.com/interlynk-io/sbomqs@latest).
  • Each spec-gap fix in a follow-up PR flips the matching negation in
    the owning scenario from ! ... to ....

CI expectation

Until #291 merges, the generate sbom subcommand does not exist on main,
so TestSbomasmGenerateSBOM will fail every scenario with "unknown
command." That is expected; CI should go green as soon as this is rebased
onto a main that contains #291.

🤖 Generated with Claude Code

riteshnoronha and others added 2 commits April 20, 2026 11:26
Adds 26 testscript scenarios under e2e/testdata/generate_sbom/ covering every
feature documented in docs/spec/generate-sbom.md, plus a developer-facing
findings doc (GAPS.md) and a scenario-to-gap map (REPORT.md).

Depends on the `generate sbom` implementation from #291. Test-only change.

Scenarios pin current behaviour via negated assertions where the
implementation diverges from the spec, so the suite passes against #291's
branch and starts failing when a gap is fixed. GAPS.md lists all 14
divergences with reproductions, actual vs expected output, and source-level
fix hints.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Keep the test scenarios only; the findings docs are shared out-of-band.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant