Sentry release smoke-test + scripted API tripwire

[gmaclennan]

Background

Recent Sentry-related regressions (159a997, b576a15, b975101) shared a failure mode that unit tests don't catch well: the JS / Node / native layer compiled and ran fine, but spans either didn't reach Sentry or arrived with wrong structure. The only signal was opening the Sentry UI after a build.

Three small unit tests now cover the cheapest cases (see backend/lib/sentry-frame.test.mjs, backend/lib/sentry.test.mjs, android/src/test/java/com/comapeo/core/SentryFgsBridgeTest.kt). They catch the logic regressions — what a processor does when it runs, whether the Node layer produces any envelope, whether envelopeToFrame stamps length correctly.

What they don't catch:

• "Production init forgot to install processor X" (wiring, not logic)
• "JS layer was the wrong shape so the SDK never sent" (cross-SDK behavior)
• "Native received the frame but the wire send was dropped" (offline/queue/transport)
• Any cross-process trace stitching regression

These need to be caught either by manual smoke-test or by a real-Sentry tripwire. This issue tracks both.

---

Part 1 — Release smoke-test checklist

Run before every release. Build the example app with diagnostics on, point at a dedicated test DSN, then verify each item in Sentry's UI within ~10 minutes.

Setup

• Example app built with the Expo Sentry plugin configured for the test project DSN
• Diagnostics toggle on; captureApplicationData on
• Known test trace label (e.g. comapeo-test-YYYY-MM-DD) set as a global tag so the run is findable

Trigger flows

• Cold-launch the app — boot transaction comapeo.boot should appear with boot.fgs-launch, boot.extract-assets, boot.node-spawn, boot.rootkey-load, boot.loader-init, boot.manager-init as child spans
• Perform one RPC call (e.g. open a project) — should produce a transaction with rpc.client span parented onto a rpc.server span (cross-process trace stitching)
• Force a fatal error (test-only menu or error-native injection) — should appear as an exception event

Verify in Sentry UI

• All three traces appear in Performance / Issues
• contexts.device.family == "Android" on FGS-emitted spans (not "Google")
• On iOS: no duplicate native-init signal in breadcrumbs
• Scope tags present: proc (main / fgs), layer (rn / node / native), comapeo.rn (version label)
• No PII strings in event JSON: rootkey base64, project IDs, lat/lng
• release matches the build's release name; environment matches expected

---

Part 2 — Scripted Sentry API tripwire (release-blocking)

A one-shot script that drives the example app through a known trace, polls Sentry's API, and asserts the trace arrived with the expected span structure. Replaces eyeballing for the "did anything actually arrive on the wire" question.

Shape

• Inputs: test DSN, Sentry auth token (read-only event:read), org/project slug
• Build & launch the example app in CI or on a connected device
• App triggers a known transaction with a hard-coded trace_id (env-var injected at boot)
• Script polls GET /api/0/projects/{org}/{project}/events/?query=trace:<id> for up to 60s
• Assert: at least one transaction event matches; op == comapeo.boot; expected child spans present; contexts.device.family == "Android" on the FGS-process span; no PII

Run cadence

• Pre-release CI workflow (manual trigger or branch-gated)
• ~5 minutes; doesn't need to run per-PR

Why not always-on per-PR

• Sentry ingestion latency makes it flaky as a blocking check
• Quota cost is non-trivial across many PRs
• Pre-release cadence catches the regression class without burning quota

Open questions

• Use a dedicated comapeo-rn-test Sentry project (separate from the real one) so test traffic doesn't pollute dashboards / rate limits
• Decide whether the script runs against a simulator/emulator (cheap, less realistic) or BrowserStack (real device, slower)
• Auth token storage: GitHub secrets vs. EAS environment secrets

---

Acceptance

• Smoke checklist lives in docs/ (e.g. docs/sentry-release-smoke.md) and is linked from the release runbook
• Tripwire script in scripts/sentry-tripwire.mjs (or similar) with a README block on usage
• CI workflow (manual-dispatch) that runs the tripwire against a build
• Smoke checklist run + signed off on at least one release before considering this issue done