Skip to content

Fix cron jobs using dev env instead of test env in CI workflows #1319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BilalG1 merged 2 commits into from
Apr 10, 2026
Merged

Fix cron jobs using dev env instead of test env in CI workflows #1319

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4e402e2
Fix cron jobs using dev env instead of test env in CI workflows
BilalG1 Apr 10, 2026
1d1dd0f
Fix ClickHouse sync verification race condition
BilalG1 Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/db-migration-backwards-compatibility.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ jobs:
uses: JarvusInnovations/background-action@v1.0.7
if: ${{ hashFiles('apps/backend/scripts/run-cron-jobs.ts') != '' }}
with:
run: pnpm -C apps/backend run with-env:dev tsx scripts/run-cron-jobs.ts --log-order=stream &
run: pnpm -C apps/backend run run-cron-jobs:test --log-order=stream &
wait-on: |
http://localhost:8102
tail: true
Expand Down Expand Up @@ -394,7 +394,7 @@ jobs:
uses: JarvusInnovations/background-action@v1.0.7
if: ${{ hashFiles('apps/backend/scripts/run-cron-jobs.ts') != '' }}
with:
run: pnpm -C apps/backend run with-env:dev tsx scripts/run-cron-jobs.ts --log-order=stream &
run: pnpm -C apps/backend run run-cron-jobs:test --log-order=stream &
wait-on: |
http://localhost:8102
tail: true
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/e2e-custom-base-port-api-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ jobs:
- name: Start run-cron-jobs in background
uses: JarvusInnovations/background-action@v1.0.7
with:
run: pnpm -C apps/backend run run-cron-jobs --log-order=stream &
run: pnpm -C apps/backend run run-cron-jobs:test --log-order=stream &
wait-on: |
http://localhost:6702
tail: true
Expand Down
2 changes: 1 addition & 1 deletion apps/backend/scripts/verify-data-integrity/clickhouse-sync-verifier.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ const SORT_KEYS = {
team_invitations: ["id"],
email_outboxes: ["id"],
project_permissions: ["user_id", "permission_id"],
notification_preferences: ["id"],
notification_preferences: ["user_id", "notification_category_id"],
refresh_tokens: ["id"],
connected_accounts: ["user_id", "provider", "provider_account_id"],
} satisfies Record<keyof typeof DEFAULT_DB_SYNC_MAPPINGS, string[]>;
Expand Down
5 changes: 5 additions & 0 deletions apps/backend/scripts/verify-data-integrity/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { syncExternalDatabases } from "@/lib/external-db-sync";
import { DEFAULT_BRANCH_ID, getSoleTenancyFromProjectBranch } from "@/lib/tenancies";
import { getPrismaClientForTenancy, globalPrismaClient } from "@/prisma-client";
import type { OrganizationRenderedConfig } from "@stackframe/stack-shared/dist/config/schema";
Expand Down Expand Up @@ -228,6 +229,10 @@ async function main() {

if (!shouldSkipClickhouse && clickhouseAvailable && tenancy) {
await recurse("[clickhouse sync]", async (recurse) => {
// Flush any pending ClickHouse syncs by running a direct sync before verifying.
// This avoids race conditions where QStash hasn't delivered all sync callbacks yet.
await syncExternalDatabases(tenancy);

Comment on lines +232 to +235
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Handle syncExternalDatabases() return value before verifying ClickHouse.

Line 234 currently ignores the needsResync signal, so verification can still run with pending/throttled sync work.

🔧 Proposed fix (bounded retry + fail fast) 
         await recurse("[clickhouse sync]", async (recurse) => {
           // Flush any pending ClickHouse syncs by running a direct sync before verifying.
           // This avoids race conditions where QStash hasn't delivered all sync callbacks yet.
-          await syncExternalDatabases(tenancy);
+          const MAX_SYNC_ATTEMPTS = 5;
+          for (let attempt = 1; attempt <= MAX_SYNC_ATTEMPTS; attempt++) {
+            const needsResync = await syncExternalDatabases(tenancy);
+            if (!needsResync) break;
+            if (attempt === MAX_SYNC_ATTEMPTS) {
+              throw new StackAssertionError(
+                `ClickHouse sync still pending after ${MAX_SYNC_ATTEMPTS} attempts for project ${projectId}.`
+              );
+            }
+            await wait(250);
+          }

           await verifyClickhouseSync({
             tenancy,
             projectId,
             branchId: DEFAULT_BRANCH_ID,

As per coding guidelines: "Fail early, fail loud; fail fast with an error instead of silently continuing" and "Validate all assumptions through type system (preferred), assertions, or tests; ideally two out of three".

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@apps/backend/scripts/verify-data-integrity/index.ts` around lines 232 - 235,
syncExternalDatabases(tenancy) currently has its return value ignored which lets
verify run despite pending/throttled sync work; update the call site to inspect
the returned flag (e.g., needsResync) and implement a bounded retry loop (with a
small backoff and max attempts) that re-invokes syncExternalDatabases and only
proceeds to ClickHouse verification when it returns false, otherwise fail fast
by throwing an error after exhausting retries; reference the
syncExternalDatabases return handling and the subsequent ClickHouse verification
call so the change affects the awaiting logic before verification.

await verifyClickhouseSync({
tenancy,
projectId,
Expand Down
Loading