Add optional PostgreSQL backend support with migration safeguards and regression coverage

[kevingatera]

Why

Audiobookshelf currently defaults to SQLite, which is simple for most installs, but some deployments need an external DB backend for reliability and infrastructure fit.

This PR adds PostgreSQL as an optional backend while preserving SQLite as the default and maintaining backward compatibility.
It also includes targeted safety fixes for real-world PostgreSQL edge cases discovered during migration and testing.

What this changes

1) Optional PostgreSQL backend support (SQLite remains default)

• Add dialect and config selection:
  • DB_DIALECT=sqlite|postgres (optional)
  • DATABASE_URL for PostgreSQL
• Keep SQLite behavior unchanged when PostgreSQL is not configured.
• Add PostgreSQL dependencies (pg, pg-hstore).

2) Dialect-aware query compatibility

• Introduce server/utils/sqlDialectHelpers.js for cross-dialect SQL generation.
• Refactor key query paths to avoid SQLite-specific syntax in PostgreSQL, including:
  • case-insensitive sort behavior
  • JSON array and value operations
  • numeric casting behavior for text-backed fields
• Guard sequence and year sorting with safe casts to avoid runtime failures from non-numeric legacy values.

3) Migration and schema-safety hardening

• Add server/scripts/migrateSqliteToPostgres.js for SQLite to PostgreSQL data migration.
• Add preflight checks to prevent data loss:
  • detect overlong varchar source values
  • detect non-integer source values for integer target columns
• Harden JSON normalization for malformed and double-encoded payloads.
• Improve migration metadata handling and case-insensitive table and column mapping for PostgreSQL.

4) PostgreSQL runtime bug fixes found during validation

• Fix legacy user lookup behavior for PostgreSQL (oldUserId path).
• Fix media progress updatedAt sync path to avoid raw SQL table-name mismatches.
• Fix collapse-series edge cases:
  • empty include list (IN ()) generation
  • aliasing issues in PostgreSQL SQL
  • display title fallback alias compatibility

5) Focused regression tests (high signal)

Added tests for the exact risky paths that caused failures:

• test/server/utils/queries/libraryItemsBookFilters.test.js
  • collapse-series empty ID set
  • safe ID quoting
  • PostgreSQL join alias safety
  • failure-path query diagnostics
• test/server/models/MediaProgress.test.js
  • valid and invalid lastUpdate handling
• test/server/models/User.test.js
  • UUID and legacy old-id lookup behavior on PostgreSQL and SQLite
• test/server/scripts/migrateSqliteToPostgres.test.js
  • JSON normalization and integer and varchar preflight checks
• test/server/utils/sqlDialectHelpers.test.js
  • dialect helper correctness

Backward compatibility

• SQLite remains the default behavior.
• Existing SQLite users are unaffected unless PostgreSQL is explicitly configured.
• PostgreSQL code paths are opt-in and covered by targeted regression tests.

Validation

Local:

• npm test passed (356 passing)
• targeted compatibility suite passed (69 passing)

CI (branch):

• Unit tests passed
• Integration tests passed

Observed runtime impact during migration validation (same host, from app logs):

• Discover load time improved from about 5.23s (SQLite period average) to about 0.08s (PostgreSQL period average).
• Personalized shelves load time improved from about 7.35s to about 0.12s.

Notes

• This PR intentionally focuses on safe, minimal compatibility and migration behavior.
• It does not force external DB usage. PostgreSQL is optional.

Beta image tag for testing:

• ghcr.io/kevingatera/audiobookshelf:homelab-postgres-5871b01f6dd9f1cd690b9a70bdc227a2d5db2643

[kevingatera]

Related context for reviewers:

• PostgreSQL support request tracking: [Enhancement]: PostgreSQL support #2046
• Newer duplicate and request thread: [Enhancement]: Postgres support #5018
• Earlier external database backend discussion: [Enhancement]: Are you open to a PR for MongoDB? #1584
• Prior compatibility-related PR (ID handling groundwork): set library id lowercase make more compatible #3600

This PR keeps SQLite as the default, adds optional PostgreSQL support, and includes regression coverage for the edge cases hit during validation (JSON normalization, numeric cast safety, aliasing, legacy oldUserId lookup, and migration safety preflight checks).

[kevingatera]

I pushed a tightening pass focused on reviewer-risk areas and sqlite/postgres parity.

Changes in this pass:

• Added dialect/default and table-detection coverage in test/server/Database.test.js.
• Added podcast and series query parity coverage in:
  • test/server/utils/queries/libraryItemsPodcastFilters.test.js
  • test/server/utils/queries/seriesFilters.test.js
• Expanded postgres-specific migration manager coverage in test/server/managers/MigrationManager.test.js.
• Added uppercase UUID regression coverage in test/server/models/User.test.js.
• Reduced collapse-series fallback log noise (warn to debug) in server/utils/queries/libraryItemsBookFilters.js.

[kevingatera]

This thread is also relevant to the design choices here: #5070

I share the concern about keeping migration/model complexity under control if both backends are supported.

That is why this PR tries to keep the blast radius small:

• SQLite stays the default path unless Postgres is explicitly enabled.
• Postgres is opt-in only (DB_DIALECT and DATABASE_URL).
• Dialect differences are kept in helper/query layers with targeted regression tests, instead of splitting core model behavior.
• Migration execution is guarded with preflight checks and dry-run-first behavior.

If this is still too broad for a first step, I can split follow-up work into smaller PRs (core dialect/query parity first, migration tooling after).