Skip to content

test(generate): integration tests + gap report for generate sbom (depends on #291)#293

Closed
riteshnoronha wants to merge 46 commits intomainfrom
test/generate-sbom-integration
Closed

test(generate): integration tests + gap report for generate sbom (depends on #291)#293
riteshnoronha wants to merge 46 commits intomainfrom
test/generate-sbom-integration

Conversation

@riteshnoronha
Copy link
Copy Markdown
Contributor

Summary

Adds integration-test coverage for sbomasm generate sbom (PR #291) along
with a developer-facing gap report. This PR is stacked on top of #291;
it exists so the developer of #291 can see a concrete, reproducible list of
spec divergences and a harness to verify fixes.

What's new (the only files you need to review)

All additions live under e2e/:

  • e2e/generate_sbom_test.go — testscript entry point (26 scenarios).
  • e2e/testdata/generate_sbom/01..26_*.txt — one scenario per spec feature.
  • e2e/testdata/generate_sbom/GAPS.mdstart here if you are the Feat/support SBOM generation #291
    developer. 14 divergences, each with spec quote, file:line, reproducible
    input, actual output, expected output, and a fix hint.
  • e2e/testdata/generate_sbom/REPORT.md — scenario → gap map, run
    instructions, NTIA scores.

Because this branch is cut from #291's HEAD (abfcad0), the diff against
main also includes everything that #291 adds. Filter the file view to
e2e/** to see only the test additions.

Spec gaps flagged

14 items documented in GAPS.md. One-line summary:

  1. .artifact-metadata.yaml output: block is parsed but ignored.
  2. Discovery errors uniformly "warn-and-continue" (spec distinguishes silent
    skip from hard error).
  3. Hash source errors (missing file, dir-as-file, empty match) are soft
    warnings.
  4. Pedigree/ancestor purl collision is a warning; spec requires hard error.
  5. Bom-ref fallback is name@version; spec requires
    pkg:generic/{name}@{version}.
  6. SOURCE_DATE_EPOCH is not honored (timestamp + serial number).
  7. No stable ordering of components, dependencies, hashes, external refs,
    tags.
  8. scope: excluded components are not dropped.
  9. Four component types (platform, device-driver,
    machine-learning-model, data) are silently downgraded.
  10. License object forms ({id}, {id,text}, {id,file}) fail JSON parse.
  11. Strict library-supplier check has an operator precedence bug.
  12. CSV hash_file column is ignored.
  13. SPDX serializer emits no pedigree mapping.
  14. SPDX loses scope.

NTIA minimum-element scores (sbomqs)

Against a well-formed manifest:

  • CycloneDX 1.6: 8.8 / 10 (grade B)
  • SPDX 2.3: 7.6 / 10 (grade C)

The CDX/SPDX delta is driven by gaps 13 and 14.

How the scenarios work

Every scenario passes today. Spec divergences are encoded as negated
assertions (e.g. ! cmp run1.json run2.json for the determinism gap), so
the suite documents the current behaviour. When a gap is fixed, the
negation flips and the scenario fails until it is rewritten for the
spec-correct form. This turns each gap into a trip-wire instead of
perma-red noise.

Test plan

  • go test ./e2e/ -run TestSbomasmGenerateSBOM -count=1 -v passes (26/26).
  • sbomqs and jq are on PATH for the scenarios that use them
    (install: go install github.com/interlynk-io/sbomqs@latest).
  • After any spec-gap fix, the owning scenario starts failing; the fix
    PR also flips the negation to the positive assertion.

🤖 Generated with Claude Code

Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
viveksahu26 and others added 16 commits April 15, 2026 20:30
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Signed-off-by: Vivek Kumar Sahu <vivekkumarsahu650@gmail.com>
Adds 26 testscript scenarios under e2e/testdata/generate_sbom/ covering every
feature documented in docs/spec/generate-sbom.md, plus a developer-facing
findings doc (GAPS.md) and a scenario-to-gap map (REPORT.md).

Scenarios pin current behaviour via negated assertions where the
implementation diverges from the spec, so the suite passes today and will
start failing when a gap is fixed. GAPS.md lists all 14 divergences with
reproductions, actual vs expected output, and source-level fix hints.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@riteshnoronha
Copy link
Copy Markdown
Contributor Author

Reopening as test-only PR; implementation is owned by #291.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants