feat(bulldozer): add verifyDataIntegrity to all Table types#1333
Merged
N2D4 merged 5 commits intoApr 15, 2026
Merged
Conversation
Add a `verifyDataIntegrity()` method to the Bulldozer `Table` interface that returns a SQL query producing error rows when materialized data diverges from what the input tables imply. Empty result = healthy. Each table operator implements verification appropriate to its semantics: - Full re-derivation (flat-map, sort, group-by, left-join, compact, limit) - Delegation to internal table (filter, map → nested flat-map) - Structural group-correspondence checks (reduce, l-fold, time-fold) - No-op for leaf/virtual tables (stored, concat) All queries are gated on isInitialized so uninitialized tables are silently skipped. A `verifyAllTablesIntegrity(tables)` helper UNION ALLs individual queries with a tableid column for easy debugging. The test file now runs verification after every test via an afterEach hook and a trackTable() wrapper around all 68 declare*Table() calls. Made-with: Cursor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughA new Bulldozer database engine is introduced, providing a hierarchical JSONB storage system with composable table operators (group-by, map, filter, sort, join, fold, compact, reduce). Dual-write integration syncs Prisma payment records to Bulldozer stored tables. Payment schema built atop Bulldozer derives transaction events, compacted entries, owned products, and item quantities through multi-phase transformations. Extensive test suites validate operators, performance, and end-to-end payment pipeline correctness. Changes
Sequence Diagram(s)sequenceDiagram
participant App as Application<br/>(Prisma)
participant DW as Dual-Write<br/>Converter
participant Engine as Bulldozer<br/>Storage Engine
participant Operator as Table<br/>Operator
participant Store as BulldozerStorage<br/>Engine JSONB
Note over App,Store: Payment Creation Flow
App->>App: Create/Update<br/>Subscription/OTP
App->>DW: Call bulldozerWrite*<br/>(prismaRow)
DW->>DW: Convert to<br/>stored row shape
DW->>Engine: Execute setRow()<br/>with converted data
Engine->>Store: UPSERT into<br/>keyPath hierarchy
Store-->>Engine: Row persisted
Engine-->>DW: Complete
DW-->>App: Dual-write done
Note over App,Store: Operator Composition Flow
Operator->>Operator: Register row-change<br/>trigger on input
App->>Store: Update source<br/>rows via setRow()
Store-->>Operator: Trigger fires
Operator->>Operator: Normalize changes,<br/>recompute affected<br/>groups
Operator->>Store: Upsert output<br/>rows & metadata
Store-->>Operator: Changes applied
Operator->>Operator: Emit downstream<br/>triggers
Note over App,Store: Payment Pipeline (Phase 1→3)
App->>Store: Seed source tables<br/>(subscriptions, OTPs)
Store-->>Operator: Data available
Operator->>Operator: Phase 1: Events<br/>(subscriptionStart,<br/>itemGrantRepeat, etc.)
Operator->>Operator: Phase 1→2: Transactions<br/>(flatten, concat, group)
Operator->>Operator: Phase 2: CompactedEntries<br/>(merge item-qty-change<br/>across expiry boundaries)
Operator->>Operator: Phase 3: OwnedProducts<br/>(LFold: accumulate<br/>grants/revocations)
Operator->>Operator: Phase 3: ItemQuantities<br/>(LFold: ledger with<br/>grants/debt/removals)
Operator->>Store: Persist all phase<br/>outputs
Store-->>App: Final state readable<br/>via customer-data queries
sequenceDiagram
participant TimeFold as TimeFold Table<br/>(e.g., subscription<br/>repeat schedule)
participant Queue as BulldozerTimeFold<br/>Queue
participant Worker as bulldozer_timefold<br/>_process_queue()<br/>Worker
participant Engine as Storage<br/>Engine
Note over TimeFold,Engine: TimeFold Initialization & Scheduling
TimeFold->>Engine: Register row-change<br/>trigger
TimeFold->>Engine: Initialize state<br/>for each input row<br/>(oldRowData=null)
Engine->>TimeFold: Trigger fires
TimeFold->>TimeFold: Run reducer at<br/>timestamp=null<br/>(compute nextTimestamp)
TimeFold->>Queue: Enqueue nextTimestamp<br/>if > cutoff
Queue-->>Engine: Queue row created
Note over TimeFold,Engine: TimeFold Scheduled Execution
Worker->>Queue: SELECT next due<br/>row (scheduledAt ≤ now)<br/>FOR UPDATE SKIP LOCKED
Queue-->>Worker: Queue row found
Worker->>Engine: Fetch current state<br/>from Storage
Engine-->>Worker: Current stateAfter
Worker->>Worker: Run reducer at<br/>timestamp (reprocess)
Worker->>Engine: Upsert state,<br/>emitted rows
Engine-->>Worker: Persisted
alt nextTimestamp > cutoff
Worker->>Queue: Re-enqueue with<br/>updated stateAfter,<br/>scheduledAt=nextTimestamp
end
Worker->>Queue: DELETE processed<br/>queue row
alt More due rows?
Worker->>Queue: Loop (GOTO SELECT)
else No more due rows
Worker->>Engine: Update lastProcessedAt<br/>in metadata
end
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120+ minutes This introduces a foundational new Bulldozer database system with:
Requires deep understanding of Bulldozer architecture, operator semantics, state folding algorithms, expiry/ledger logic, and integration points. Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches🧪 Generate unit tests (beta)
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands Bulldozer’s table interface to support integrity verification queries and adds a new payments “Bulldozer schema” pipeline (plus dual-write + ingestion wiring) that materializes customer payment state from Prisma data.
Changes:
- Add
Table.verifyDataIntegrity()and averifyAllTablesIntegrity(tables)helper to aggregate per-table integrity checks. - Introduce a payments Bulldozer schema (stored tables → events → transactions → compacted entries → owned-products / item-quantities) plus real-Postgres tests for the pipeline.
- Add payments dual-write + a migrations-time init/ingress script; add local dev pg_cron support and dev UX updates (launchpad + backend dev script).
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| pnpm-lock.yaml | Adds elkjs dependency to the lockfile. |
| docker/dev-postgres-with-extensions/Dockerfile | Installs/configures pg_cron in the dev Postgres image. |
| claude/CLAUDE-KNOWLEDGE.md | Adds Bulldozer/pg_cron and payments pipeline knowledge entries. |
| apps/dev-launchpad/public/index.html | Adds “Bulldozer Studio” to the launchpad services list. |
| apps/backend/src/route-handlers/prisma-handler.tsx | Makes Prisma create invocation safer via runtime Reflect checks. |
| apps/backend/src/prisma-client.tsx | Converts a static import to a dynamic import to avoid eager dependency loading. |
| apps/backend/src/lib/tokens.tsx | Removes signed_up_at from the access token payload. |
| apps/backend/src/lib/stripe.tsx | Adds payments dual-write calls after Stripe upserts. |
| apps/backend/src/lib/payments/schema/types.ts | Introduces payments pipeline TypeScript row/type definitions. |
| apps/backend/src/lib/payments/schema/phase-3/split-algo.ts | Extracts Phase 3 split algorithm SQL builder for reuse/testing. |
| apps/backend/src/lib/payments/schema/phase-3/owned-products.ts | Adds Phase 3 OwnedProducts fold table definition. |
| apps/backend/src/lib/payments/schema/phase-3/ledger-algo.ts | Extracts Phase 3 ledger reducer SQL builder for item quantities. |
| apps/backend/src/lib/payments/schema/phase-3/item-quantities.ts | Adds Phase 3 ItemQuantities fold table definition. |
| apps/backend/src/lib/payments/schema/phase-3/item-changes-with-expiries.ts | Adds Phase 3 join/split pipeline to enrich changes with expiries. |
| apps/backend/src/lib/payments/schema/phase-2/compacted-transaction-entries.ts | Adds Phase 2 entry flattening + compaction pipeline. |
| apps/backend/src/lib/payments/schema/phase-1/transactions.ts | Adds Phase 1 transaction assembly from events/manual inputs. |
| apps/backend/src/lib/payments/schema/phase-1/stored-tables.ts | Defines payments “seed” stored tables. |
| apps/backend/src/lib/payments/schema/phase-1/otp-timefold-algo.ts | Adds OTP TimeFold reducer SQL builder. |
| apps/backend/src/lib/payments/schema/phase-1/events.ts | Adds Phase 1 event derivation (TimeFold + maps + joins). |
| apps/backend/src/lib/payments/schema/index.ts | Composes the payments schema and exposes init-order table lists/categories. |
| apps/backend/src/lib/payments/schema/tests/test-helpers.ts | Adds real-Postgres test DB helper for payments pipeline tests. |
| apps/backend/src/lib/payments/schema/tests/phase-2.test.ts | Adds Phase 2 correctness tests against real Postgres. |
| apps/backend/src/lib/payments/schema/tests/integration-2-3.test.ts | Adds Phase 2→3 integration tests against real Postgres. |
| apps/backend/src/lib/payments/schema/tests/dual-write.test.ts | Adds dual-write conversion + setRow behavior tests. |
| apps/backend/src/lib/payments/customer-data.ts | Adds customer-facing reads from the Phase 3 output tables. |
| apps/backend/src/lib/payments/bulldozer-dual-write.ts | Adds Prisma→Bulldozer conversions and dual-write executors. |
| apps/backend/src/lib/payments.tsx | Adds dual-write calls after payment mutations in core payments logic. |
| apps/backend/src/lib/bulldozer/db/utilities.ts | Adds/updates Bulldozer SQL helper types/builders and path utilities. |
| apps/backend/src/lib/bulldozer/db/tables/stored-table.ts | Implements verifyDataIntegrity() for stored tables (no-op empty result). |
| apps/backend/src/lib/bulldozer/db/tables/map-table.ts | Delegates verifyDataIntegrity() via nested FlatMap implementation. |
| apps/backend/src/lib/bulldozer/db/tables/filter-table.ts | Delegates verifyDataIntegrity() via nested FlatMap implementation. |
| apps/backend/src/lib/bulldozer/db/tables/concat-table.ts | Implements verifyDataIntegrity() as a no-op for concat. |
| apps/backend/src/lib/bulldozer/db/index.ts | Adds verifyDataIntegrity() to Table and verifyAllTablesIntegrity(). |
| apps/backend/src/lib/bulldozer/db/example-schema.ts | Adds an example schema file demonstrating Bulldozer operators. |
| apps/backend/src/app/api/latest/payments/purchases/purchase-session/route.tsx | Dual-writes subscription updates for purchase-session flows. |
| apps/backend/src/app/api/latest/payments/products/[customer_type]/[customer_id]/switch/route.ts | Dual-writes subscription updates for product switch flows. |
| apps/backend/src/app/api/latest/payments/products/[customer_type]/[customer_id]/[product_id]/route.ts | Dual-writes subscription updates for product cancellation. |
| apps/backend/src/app/api/latest/payments/items/[customer_type]/[customer_id]/[item_id]/update-quantity/route.ts | Dual-writes item quantity changes after Prisma writes. |
| apps/backend/src/app/api/latest/internal/payments/transactions/refund/route.tsx | Dual-writes refunded purchases/subscriptions after refund updates. |
| apps/backend/src/app/api/latest/integrations/stripe/webhooks/route.tsx | Dual-writes OneTimePurchase upserts from Stripe webhooks. |
| apps/backend/src/app/api/latest/auth/passkey/register/verification-code-handler.tsx | Adds a missing null check for registrationInfo. |
| apps/backend/src/app/api/latest/auth/passkey/initiate-passkey-registration/route.tsx | Avoids relying on hints typing by using Reflect and deletion. |
| apps/backend/scripts/run-cron-jobs.ts | Adds an initial wait before starting cron loop. |
| apps/backend/scripts/db-migrations.ts | Runs payments Bulldozer init/ingress after seed/init/migrate. |
| apps/backend/scripts/bulldozer-payments-init.ts | Adds payments schema initialization + Prisma→Bulldozer ingress script. |
| apps/backend/prisma/schema.prisma | Adds Bulldozer tables to Prisma schema; adds endedAt/revokedAt. |
| apps/backend/prisma/migrations/20260413043028_add_revoked_at_to_otp/migration.sql | Adds revokedAt to OneTimePurchase. |
| apps/backend/prisma/migrations/20260413040008_add_subscription_ended_at/migration.sql | Adds endedAt to Subscription. |
| apps/backend/prisma/migrations/20260323150000_add_bulldozer_timefold_queue/tests/process-queue.ts | Adds migration test for timefold queue processing. |
| apps/backend/prisma/migrations/20260323150000_add_bulldozer_timefold_queue/migration.sql | Adds timefold queue + worker function + pg_cron best-effort setup. |
| apps/backend/prisma/migrations/20260323120000_add_bulldozer_data/tests/ltree-queries.ts | Adds migration test coverage for BulldozerStorageEngine semantics. |
| apps/backend/prisma/migrations/20260323120000_add_bulldozer_data/migration.sql | Adds BulldozerStorageEngine table + indexes + seeded roots. |
| apps/backend/package.json | Adds elkjs and runs run-bulldozer-studio in pnpm dev. |
| AGENTS.md | Updates lint instructions and adds an agent guideline. |
| .vscode/settings.json | Reorders/adds cSpell words. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Greptile SummaryThis PR adds a Two implementations diverge from the PR's stated "full re-derivation" goal in ways that leave real gaps:
Confidence Score: 4/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[verifyAllTablesIntegrity] -->|UNION ALL| B[stored-table\nno-op ✓]
A -->|UNION ALL| C[concat-table\nno-op ✓]
A -->|UNION ALL| D[flat-map-table\nfull re-derive ✓]
A -->|UNION ALL| E[filter-table\ndelegates to flat-map ✓]
A -->|UNION ALL| F[map-table\ndelegates to flat-map ✓]
A -->|UNION ALL| G[group-by-table\nfull re-derive ✓]
A -->|UNION ALL| H[left-join-table\nfull re-derive ✓]
A -->|UNION ALL| I[compact-table\nfull re-derive ✓]
A -->|UNION ALL| J[sort-table\nrowData only ⚠️]
A -->|UNION ALL| K[limit-table\nextra rows + count only ⚠️]
A -->|UNION ALL| L[reduce-table\nstructural only 📋]
A -->|UNION ALL| M[l-fold-table\ngroup membership only 📋]
A -->|UNION ALL| N[time-fold-table\nextra groups only 📋]
style J fill:#ff9999
style K fill:#ff9999
style L fill:#ffffcc
style M fill:#ffffcc
style N fill:#ffffcc
Prompt To Fix All With AIThis is a comment left during a code review.
Path: apps/backend/src/lib/bulldozer/db/tables/sort-table.ts
Line: 417-452
Comment:
**Sort key never verified**
Both `"expected"` and `"actual"` CTEs select only `rowdata`, not `rowsortkey`. A stored sort key that is corrupted (wrong value, wrong row, stale after an incremental update) passes silently. The sort key is the primary product of this operator; downstream `LFold` tables fold in sort-key order, so a silent sort-key mismatch here propagates undetected to those tables. The PR description claims "full re-derivation" for sort, but the check only verifies row-data propagation.
The fix is to also compute the expected sort key via `getSortKey` and compare:
```
"expected" AS (
SELECT
"r"."groupkey" AS "groupKey",
"r"."rowidentifier" AS "rowIdentifier",
"r"."rowdata" AS "rowData",
to_jsonb("sk"."newSortKey") AS "computedSortKey"
FROM (${allInputRows}) AS "r"
CROSS JOIN LATERAL (
SELECT ${options.getSortKey}
FROM (
SELECT "r"."rowidentifier" AS "rowIdentifier",
"r"."rowsortkey" AS "oldSortKey",
"r"."rowdata" AS "rowData"
) AS "sortKeyInput"
) AS "sk"
),
"actual" AS (
SELECT "r"."groupkey" AS "groupKey", "r"."rowidentifier" AS "rowIdentifier",
"r"."rowdata" AS "rowData",
"r"."rowsortkey" AS "computedSortKey"
FROM (${allActualRows}) AS "r"
)
-- then JOIN on groupKey + rowIdentifier and flag differences in rowData OR computedSortKey
```
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: apps/backend/src/lib/bulldozer/db/tables/limit-table.ts
Line: 496-535
Comment:
**Under-selection goes undetected**
The check catches (1) rows in `actual` that don't exist in the input (`extraRows`) and (2) groups whose count exceeds the limit (`overLimit`), but it never detects the case where the materialized table stores *fewer* rows than it should. For example, if the input has 5 rows and the limit is 3, a bug that only materializes 2 rows produces no errors — count 2 ≤ 3 and no ghost rows. Similarly, if the wrong 3 rows are selected (all valid inputs, just not the "first 3 by sort key"), neither check fires.
The PR description claims "full re-derivation" for limit, which requires re-computing the expected selected rows per group (re-applying the same sort-key ordering + LIMIT N logic) and diffing that against `actual`. The current check is a one-sided "sanity guard" rather than a full re-derivation.
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: apps/backend/src/lib/bulldozer/db/tables/l-fold-table.ts
Line: 802-835
Comment:
**Structural-only check doesn't verify folded row data**
The verification only confirms that the set of output groups matches the set of input groups (`missing_group` / `extra_group`). Wrong data values in any output row — for example, a stale accumulated state that was never recomputed after an upstream change — would pass this check entirely. While `reduce` and `time-fold` face similar constraints (the custom aggregate / recursive WITH can't easily be embedded in a bare `SqlQuery`), it's worth documenting this coverage gap explicitly in the method body or a code comment, so future maintainers know these table types only get a structural guard rather than a full re-derivation.
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: apps/backend/src/lib/bulldozer/db/index.test.ts
Line: 239-251
Comment:
**Opt-in tracking could silently miss tables**
`allInitializedTables` is populated only when a test (or its helper) explicitly calls `trackTable`. Any test that declares and initializes a table directly — without going through one of the helper functions — won't have that table integrity-checked in `afterEach`. A future author adding a test might not notice this requirement. Consider adding a comment here warning that every initialized table must be wrapped with `trackTable`, or explore hooking into table `init()` so tracking is automatic.
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "feat(bulldozer): add verifyDataIntegrity..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+417
to
+452
| verifyDataIntegrity: () => { | ||
| const allInputRows = options.fromTable.listRowsInGroup({ | ||
| start: "start", end: "end", startInclusive: true, endInclusive: true, | ||
| }); | ||
| const allActualRows = table.listRowsInGroup({ | ||
| start: "start", end: "end", startInclusive: true, endInclusive: true, | ||
| }); | ||
| return sqlQuery` | ||
| WITH "expected" AS ( | ||
| SELECT "r"."groupkey" AS "groupKey", "r"."rowidentifier" AS "rowIdentifier", "r"."rowdata" AS "rowData" | ||
| FROM (${allInputRows}) AS "r" | ||
| ), | ||
| "actual" AS ( | ||
| SELECT "r"."groupkey" AS "groupKey", "r"."rowidentifier" AS "rowIdentifier", "r"."rowdata" AS "rowData" | ||
| FROM (${allActualRows}) AS "r" | ||
| ) | ||
| SELECT | ||
| CASE | ||
| WHEN "expected"."rowIdentifier" IS NULL THEN 'extra_row' | ||
| WHEN "actual"."rowIdentifier" IS NULL THEN 'missing_row' | ||
| ELSE 'data_mismatch' | ||
| END AS errortype, | ||
| COALESCE("expected"."groupKey", "actual"."groupKey") AS groupkey, | ||
| COALESCE("expected"."rowIdentifier", "actual"."rowIdentifier") AS rowidentifier, | ||
| "expected"."rowData" AS expected, | ||
| "actual"."rowData" AS actual | ||
| FROM "expected" | ||
| FULL OUTER JOIN "actual" | ||
| ON "expected"."groupKey" IS NOT DISTINCT FROM "actual"."groupKey" | ||
| AND "expected"."rowIdentifier" = "actual"."rowIdentifier" | ||
| WHERE ("expected"."rowIdentifier" IS NULL | ||
| OR "actual"."rowIdentifier" IS NULL | ||
| OR "expected"."rowData" IS DISTINCT FROM "actual"."rowData") | ||
| AND ${isInitializedExpression} | ||
| `; | ||
| }, |
Contributor
There was a problem hiding this comment.
Both "expected" and "actual" CTEs select only rowdata, not rowsortkey. A stored sort key that is corrupted (wrong value, wrong row, stale after an incremental update) passes silently. The sort key is the primary product of this operator; downstream LFold tables fold in sort-key order, so a silent sort-key mismatch here propagates undetected to those tables. The PR description claims "full re-derivation" for sort, but the check only verifies row-data propagation.
The fix is to also compute the expected sort key via getSortKey and compare:
"expected" AS (
SELECT
"r"."groupkey" AS "groupKey",
"r"."rowidentifier" AS "rowIdentifier",
"r"."rowdata" AS "rowData",
to_jsonb("sk"."newSortKey") AS "computedSortKey"
FROM (${allInputRows}) AS "r"
CROSS JOIN LATERAL (
SELECT ${options.getSortKey}
FROM (
SELECT "r"."rowidentifier" AS "rowIdentifier",
"r"."rowsortkey" AS "oldSortKey",
"r"."rowdata" AS "rowData"
) AS "sortKeyInput"
) AS "sk"
),
"actual" AS (
SELECT "r"."groupkey" AS "groupKey", "r"."rowidentifier" AS "rowIdentifier",
"r"."rowdata" AS "rowData",
"r"."rowsortkey" AS "computedSortKey"
FROM (${allActualRows}) AS "r"
)
-- then JOIN on groupKey + rowIdentifier and flag differences in rowData OR computedSortKey
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/backend/src/lib/bulldozer/db/tables/sort-table.ts
Line: 417-452
Comment:
**Sort key never verified**
Both `"expected"` and `"actual"` CTEs select only `rowdata`, not `rowsortkey`. A stored sort key that is corrupted (wrong value, wrong row, stale after an incremental update) passes silently. The sort key is the primary product of this operator; downstream `LFold` tables fold in sort-key order, so a silent sort-key mismatch here propagates undetected to those tables. The PR description claims "full re-derivation" for sort, but the check only verifies row-data propagation.
The fix is to also compute the expected sort key via `getSortKey` and compare:
```
"expected" AS (
SELECT
"r"."groupkey" AS "groupKey",
"r"."rowidentifier" AS "rowIdentifier",
"r"."rowdata" AS "rowData",
to_jsonb("sk"."newSortKey") AS "computedSortKey"
FROM (${allInputRows}) AS "r"
CROSS JOIN LATERAL (
SELECT ${options.getSortKey}
FROM (
SELECT "r"."rowidentifier" AS "rowIdentifier",
"r"."rowsortkey" AS "oldSortKey",
"r"."rowdata" AS "rowData"
) AS "sortKeyInput"
) AS "sk"
),
"actual" AS (
SELECT "r"."groupkey" AS "groupKey", "r"."rowidentifier" AS "rowIdentifier",
"r"."rowdata" AS "rowData",
"r"."rowsortkey" AS "computedSortKey"
FROM (${allActualRows}) AS "r"
)
-- then JOIN on groupKey + rowIdentifier and flag differences in rowData OR computedSortKey
```
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+496
to
+535
| verifyDataIntegrity: () => { | ||
| const allInputRows = options.fromTable.listRowsInGroup({ | ||
| start: "start", end: "end", startInclusive: true, endInclusive: true, | ||
| }); | ||
| const allActualRows = table.listRowsInGroup({ | ||
| start: "start", end: "end", startInclusive: true, endInclusive: true, | ||
| }); | ||
| return sqlQuery` | ||
| WITH "inputRows" AS ( | ||
| SELECT "r"."groupkey" AS "groupKey", "r"."rowidentifier" AS "rowIdentifier", "r"."rowdata" AS "rowData" | ||
| FROM (${allInputRows}) AS "r" | ||
| ), | ||
| "actual" AS ( | ||
| SELECT "r"."groupkey" AS "groupKey", "r"."rowidentifier" AS "rowIdentifier", "r"."rowdata" AS "rowData" | ||
| FROM (${allActualRows}) AS "r" | ||
| ), | ||
| "extraRows" AS ( | ||
| SELECT 'extra_row' AS errortype, | ||
| "actual"."groupKey" AS groupkey, "actual"."rowIdentifier" AS rowidentifier, | ||
| NULL::jsonb AS expected, "actual"."rowData" AS actual | ||
| FROM "actual" | ||
| LEFT JOIN "inputRows" | ||
| ON "inputRows"."groupKey" IS NOT DISTINCT FROM "actual"."groupKey" | ||
| AND "inputRows"."rowIdentifier" = "actual"."rowIdentifier" | ||
| WHERE "inputRows"."rowIdentifier" IS NULL | ||
| ), | ||
| "overLimit" AS ( | ||
| SELECT 'over_limit' AS errortype, | ||
| "counts"."groupKey" AS groupkey, NULL::text AS rowidentifier, | ||
| to_jsonb("counts"."cnt") AS expected, to_jsonb(${normalizedLimit}) AS actual | ||
| FROM ( | ||
| SELECT "groupKey", COUNT(*)::int AS "cnt" FROM "actual" GROUP BY "groupKey" | ||
| ) AS "counts" | ||
| WHERE "counts"."cnt" > ${normalizedLimit} | ||
| ) | ||
| SELECT * FROM "extraRows" WHERE ${isInitializedExpression} | ||
| UNION ALL | ||
| SELECT * FROM "overLimit" WHERE ${isInitializedExpression} | ||
| `; | ||
| }, |
Contributor
There was a problem hiding this comment.
The check catches (1) rows in actual that don't exist in the input (extraRows) and (2) groups whose count exceeds the limit (overLimit), but it never detects the case where the materialized table stores fewer rows than it should. For example, if the input has 5 rows and the limit is 3, a bug that only materializes 2 rows produces no errors — count 2 ≤ 3 and no ghost rows. Similarly, if the wrong 3 rows are selected (all valid inputs, just not the "first 3 by sort key"), neither check fires.
The PR description claims "full re-derivation" for limit, which requires re-computing the expected selected rows per group (re-applying the same sort-key ordering + LIMIT N logic) and diffing that against actual. The current check is a one-sided "sanity guard" rather than a full re-derivation.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/backend/src/lib/bulldozer/db/tables/limit-table.ts
Line: 496-535
Comment:
**Under-selection goes undetected**
The check catches (1) rows in `actual` that don't exist in the input (`extraRows`) and (2) groups whose count exceeds the limit (`overLimit`), but it never detects the case where the materialized table stores *fewer* rows than it should. For example, if the input has 5 rows and the limit is 3, a bug that only materializes 2 rows produces no errors — count 2 ≤ 3 and no ghost rows. Similarly, if the wrong 3 rows are selected (all valid inputs, just not the "first 3 by sort key"), neither check fires.
The PR description claims "full re-derivation" for limit, which requires re-computing the expected selected rows per group (re-applying the same sort-key ordering + LIMIT N logic) and diffing that against `actual`. The current check is a one-sided "sanity guard" rather than a full re-derivation.
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+802
to
+835
| verifyDataIntegrity: () => { | ||
| const allInputGroups = options.fromTable.listGroups({ | ||
| start: "start", end: "end", startInclusive: true, endInclusive: true, | ||
| }); | ||
| const allActualGroups = table.listGroups({ | ||
| start: "start", end: "end", startInclusive: true, endInclusive: true, | ||
| }); | ||
| return sqlQuery` | ||
| WITH "inputGroups" AS ( | ||
| SELECT "g"."groupkey" AS "groupKey" FROM (${allInputGroups}) AS "g" | ||
| ), | ||
| "actualGroups" AS ( | ||
| SELECT "g"."groupkey" AS "groupKey" FROM (${allActualGroups}) AS "g" | ||
| ), | ||
| "missingGroups" AS ( | ||
| SELECT 'missing_group' AS errortype, | ||
| "inputGroups"."groupKey" AS groupkey, NULL::text AS rowidentifier, | ||
| NULL::jsonb AS expected, NULL::jsonb AS actual | ||
| FROM "inputGroups" | ||
| LEFT JOIN "actualGroups" ON "actualGroups"."groupKey" IS NOT DISTINCT FROM "inputGroups"."groupKey" | ||
| WHERE "actualGroups"."groupKey" IS NULL | ||
| ), | ||
| "extraGroups" AS ( | ||
| SELECT 'extra_group' AS errortype, | ||
| "actualGroups"."groupKey" AS groupkey, NULL::text AS rowidentifier, | ||
| NULL::jsonb AS expected, NULL::jsonb AS actual | ||
| FROM "actualGroups" | ||
| LEFT JOIN "inputGroups" ON "inputGroups"."groupKey" IS NOT DISTINCT FROM "actualGroups"."groupKey" | ||
| WHERE "inputGroups"."groupKey" IS NULL | ||
| ) | ||
| SELECT * FROM "missingGroups" WHERE ${isInitializedExpression} | ||
| UNION ALL SELECT * FROM "extraGroups" WHERE ${isInitializedExpression} | ||
| `; | ||
| }, |
Contributor
There was a problem hiding this comment.
The verification only confirms that the set of output groups matches the set of input groups (missing_group / extra_group). Wrong data values in any output row — for example, a stale accumulated state that was never recomputed after an upstream change — would pass this check entirely. While reduce and time-fold face similar constraints (the custom aggregate / recursive WITH can't easily be embedded in a bare SqlQuery), it's worth documenting this coverage gap explicitly in the method body or a code comment, so future maintainers know these table types only get a structural guard rather than a full re-derivation.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/backend/src/lib/bulldozer/db/tables/l-fold-table.ts
Line: 802-835
Comment:
**Structural-only check doesn't verify folded row data**
The verification only confirms that the set of output groups matches the set of input groups (`missing_group` / `extra_group`). Wrong data values in any output row — for example, a stale accumulated state that was never recomputed after an upstream change — would pass this check entirely. While `reduce` and `time-fold` face similar constraints (the custom aggregate / recursive WITH can't easily be embedded in a bare `SqlQuery`), it's worth documenting this coverage gap explicitly in the method body or a code comment, so future maintainers know these table types only get a structural guard rather than a full re-derivation.
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+239
to
+251
| const allInitializedTables: Table<any, any, any>[] = []; | ||
| function trackTable<T extends Table<any, any, any>>(t: T): T { | ||
| allInitializedTables.push(t); | ||
| return t; | ||
| } | ||
|
|
||
| afterEach(async () => { | ||
| if (allInitializedTables.length > 0) { | ||
| const errors = await readRows(verifyAllTablesIntegrity(allInitializedTables)); | ||
| expect(errors).toEqual([]); | ||
| } | ||
| allInitializedTables.length = 0; | ||
| }); |
Contributor
There was a problem hiding this comment.
allInitializedTables is populated only when a test (or its helper) explicitly calls trackTable. Any test that declares and initializes a table directly — without going through one of the helper functions — won't have that table integrity-checked in afterEach. A future author adding a test might not notice this requirement. Consider adding a comment here warning that every initialized table must be wrapped with trackTable, or explore hooking into table init() so tracking is automatic.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/backend/src/lib/bulldozer/db/index.test.ts
Line: 239-251
Comment:
**Opt-in tracking could silently miss tables**
`allInitializedTables` is populated only when a test (or its helper) explicitly calls `trackTable`. Any test that declares and initializes a table directly — without going through one of the helper functions — won't have that table integrity-checked in `afterEach`. A future author adding a test might not notice this requirement. Consider adding a comment here warning that every initialized table must be wrapped with `trackTable`, or explore hooking into table `init()` so tracking is automatic.
How can I resolve this? If you propose a fix, please make it concise.
…ments-bulldozer-txn-rework
N2D4
approved these changes
Apr 14, 2026
Contributor
There was a problem hiding this comment.
there is a verify-data-integrity.ts script — can you update that one so it runs all of these integrity checks?
Comment on lines
+245
to
+251
| afterEach(async () => { | ||
| if (allInitializedTables.length > 0) { | ||
| const errors = await readRows(verifyAllTablesIntegrity(allInitializedTables)); | ||
| expect(errors).toEqual([]); | ||
| } | ||
| allInitializedTables.length = 0; | ||
| }); |
Contributor
There was a problem hiding this comment.
can you also do this in index.fuzz.test.ts and index.perf.test.ts?
Comment on lines
+145
to
+157
| // any is used here because the verifier works with heterogeneous table types | ||
| export function verifyAllTablesIntegrity(tables: Table<any, any, any>[]): SqlQuery<Iterable<{ tableId: string, errorType: string, groupKey: Json | null, rowIdentifier: RowIdentifier | null, expected: Json | null, actual: Json | null }>> { | ||
| if (tables.length === 0) { | ||
| return sqlQuery`SELECT NULL::text AS tableid, NULL::text AS errortype, NULL::jsonb AS groupkey, NULL::text AS rowidentifier, NULL::jsonb AS expected, NULL::jsonb AS actual WHERE false`; | ||
| } | ||
| const combined: { sql: string } = { | ||
| sql: tables.map(t => { | ||
| const label = tableIdToDebugString(t.tableId).replaceAll("'", "''"); | ||
| return `SELECT '${label}' AS tableid, "v".* FROM (${t.verifyDataIntegrity().sql}) AS "v"`; | ||
| }).join("\nUNION ALL\n"), | ||
| }; | ||
| return sqlQuery`${combined}`; | ||
| } |
Contributor
There was a problem hiding this comment.
is this necessary? feels like it could just be a normal for-loop inside the verify-data-integrity script
…tables Added a recursive verification process for all payments tables in the Bulldozer system. This includes the implementation of a `verifyDataIntegrity()` method for each table, which checks for data consistency and throws errors if discrepancies are found. The verification is integrated into the main execution flow, ensuring that data integrity is maintained across all operations. Additionally, updated tests to validate the integrity of initialized tables after each test execution.
…ments-bulldozer-txn-rework
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
This was referenced Apr 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
verifyDataIntegrity()method to the BulldozerTableinterface that returns a SQL query producing error rows when materialized data diverges from re-derivation from inputs (empty result = healthy).verifyAllTablesIntegrity(tables)helper that UNION ALLs all tables' queries with atableidcolumn.afterEachhook — all 124 existing tests now automatically verify data integrity after each run.Test plan
isInitialized)Made with Cursor
Summary by CodeRabbit
Release Notes
New Features
Bug Fixes
Documentation
Tests