Add experimental Content-Addressable Storage (CAS) backups

[filimonov]

Add experimental Content-Addressable Storage (CAS) backups

⚠️ Experimental. Layout may change incompatibly before stable. See docs/cas-design.md and docs/cas-operator-runbook.md.

Summary

Adds seven new commands — cas-{upload,download,restore,delete,verify,prune,status} — that store backups in a content-addressable layout where every file is keyed by the CityHash128 already in ClickHouse's per-part checksums.txt. Identical content is stored once and referenced from any number of backups, across mutations, days, and tables. There is no RequiredBackup chain — every backup is independently restorable; deleting one never affects another. Storage grows with new content, not with backup count.

The feature lives side-by-side with v1 (upload/download/restore) under a separate root prefix in the bucket. v1 commands stay byte-for-byte unchanged.

Why

Three operational pain points motivated this:

1. Mutations re-upload everything. A single ALTER TABLE … UPDATE rewrites one column file and renames the part. Today's incremental dedup is name-based, so the new part looks unrelated and 99% of identical bytes get re-transferred. Content-addressing dedups them.
2. Incremental chains are fragile. v1 incremental backups depend on a base; restore unrolls the chain. Operators avoid the chain by taking expensive full backups. CAS removes the dependency: each backup is independent.
3. Storage scales with backup count. Keeping 30 daily snapshots of a slowly-changing dataset is roughly the same cost as keeping one.

How to enable

cas:
  enabled: true
  cluster_id: my-prod-cluster   # required, unique per source cluster

Then:

clickhouse-backup create my_backup
clickhouse-backup cas-upload my_backup
clickhouse-backup cas-restore my_backup

All seven commands also work over the existing daemon REST API (POST /backup/cas-upload/{name} etc.; cas-* verbs in /backup/actions).

What's new

Operator-facing:

• 7 new CLI commands, 7 new REST endpoints
• New cas: config block with enabled, cluster_id, root_prefix, inline_threshold, grace_blob, abandon_threshold, wait_for_prune, allow_unsafe_markers, skip_conditional_put_probe, allow_unsafe_object_disk_skip
• cas-prune mark-and-sweep GC; cas-verify for HEAD+size integrity checks
• cas-status daemon endpoint for monitoring

Implementation:

• pkg/cas/ — core engine (~6 KLOC + ~6 KLOC tests)
• pkg/checksumstxt/ — new parser for ClickHouse's part-file checksums format (versions 2/3/4)
• pkg/storage/ — new PutFileAbsoluteIfAbsent / PutFileIfAbsent on all 6 backends (S3, Azure, GCS, COS, SFTP, FTP) for atomic markers
• One-shot startup probe verifies S3-compatible backends honor If-None-Match: * (refuses MinIO < 2024-11)

Breaking changes

See ChangeLog.md vNEXT section. Top three:

1. Don't downgrade with CAS data in the bucket. A pre-CAS binary will treat the cas/ namespace as a broken v1 backup and silently delete it on the next clean remote_broken or BackupsToKeepRemote run. Recovery procedure in the operator runbook.
2. pkg/storage.RemoteStorage interface gains two methods (PutFileAbsoluteIfAbsent, PutFileIfAbsent). Third-party implementations need to add them.
3. pkg/storage.BackupDestination.BackupList signature gains a skipPrefixes []string parameter.

A v1 backup literally named "cas" (matching the default root_prefix) will be filtered after upgrade; rename before upgrading.

How to review (suggested reading order)

The branch has many small commits because it landed in 8 phases plus 6 review waves. The cleanest way through it is by package, not by commit. Suggested order:

1. Read the design first

• docs/cas-design.md — full design (~700 lines). §1–§3 = motivation/scope, §6 = the protocol, §7 = what's reused, §8 = risk register, §9 = the v2 backlog of known deferrals.
• docs/cas-operator-runbook.md — what an operator sees: first-deployment walkthrough, recovery procedures, REST endpoints.

2. Read the data model

• pkg/checksumstxt/ (smallest; standalone) — parser for the ClickHouse on-disk format. The hash CAS uses comes from here. format.md is the reference spec.
• pkg/cas/types.go + pkg/cas/blobpath.go + pkg/cas/paths.go — blob-key derivation and the on-bucket layout (where everything lives under cas/<cluster>/).
• pkg/cas/config.go — the cas: config block, validation rules, the SkipPrefixes() contract that protects CAS from v1 retention.

3. Storage primitives

• pkg/storage/structs.go — interface-level: PutFileAbsoluteIfAbsent, ErrConditionalPutNotSupported, the BackupList signature change.
• pkg/storage/{s3,gcs,azblob,cos,sftp,ftp}.go (and tests) — per-backend conditional-create implementations. Read S3's IfNoneMatch: "*" first (canonical), then SFTP's O_EXCL, then FTP (the refuse-by-default one with the allow_unsafe_markers opt-in best-effort fallback).
• pkg/cas/casstorage/backend_storage.go — the thin adapter that wraps pkg/storage.BackupDestination as the narrower cas.Backend interface. This is the import-cycle breaker; everything in pkg/cas/ talks to cas.Backend, never to pkg/storage directly.

4. The hot path — upload

• pkg/cas/upload.go — the 13-step upload protocol. Each step is commented with its number and intent. Steps 5 (write inprogress marker) → 11 (pre-commit re-checks) → 13 (commit metadata.json) are the load-bearing sequence; everything between is parallel blob/archive uploads. Note the single deferred cleanup at step 5 (added in wave-6 review Support incremental backups #3): on any error path including panic, the marker is released via a detached context.
• pkg/cas/coldlist.go + pkg/cas/cache.go — the parallel 256-shard LIST that seeds the existence set so the same blob isn't re-uploaded.
• pkg/cas/archive.go — per-table tar.zstd builder for the inline files.
• pkg/cas/markers.go + pkg/cas/probe.go + pkg/cas/wait.go — atomic mutual exclusion, conditional-put probe, polite waiting on prune-marker.

5. Download and restore

• pkg/cas/download.go — materializes a backup into a v1-shaped local directory. Note the staging-dir + atomic-rename pattern (added in an earlier review wave) — partial downloads never leave a "looks valid" directory at the final path.
• pkg/cas/restore.go — thin wrapper: cas-download into a staging dir, then hand off to v1's b.Restore. The handoff metadata sets BackupMetadata.CAS.Handoff = true so v1's cross-mode guards distinguish "raw CAS backup" (refuse) from "CAS handoff layout" (allow + skip object-disk fetch).

6. Lifecycle: delete and prune

• pkg/cas/delete.go — ordered delete (metadata.json first, then subtree). Writes its own inprogress marker tagged Tool: cas-delete to lock out concurrent same-name uploads on other hosts.
• pkg/cas/prune.go + pkg/cas/sweep.go + pkg/cas/markset.go — mark-and-sweep GC. External-mergesort mark set spilled to disk; container/heap-based shard iterator for the sweep phase. Holds an exclusive prune.marker; explicit --unlock escape hatch when stranded.

7. Verify, list, status

• pkg/cas/verify.go — HEAD + size check (no content re-hash; that's a v2 deferred item).
• pkg/cas/list.go — backup catalog walker; surfaces CAS backups in clickhouse-backup list remote and the REST list endpoint.
• pkg/cas/status.go — bucket health summary (LIST-only, cheap).

8. The integration glue

• pkg/backup/cas_methods.go — Backuper.CASUpload/CASDownload/CASRestore/CASDelete/CASVerify/CASPrune/CASStatus. Object-disk preflight (fail-closed on disk-query failure), pidlock around cas-download phase, probe-singleton injection for daemon mode.
• cmd/clickhouse-backup/cas_commands.go — CLI flag definitions and action wiring.
• pkg/server/cas_handlers.go + pkg/server/actions_cas.go — REST handlers (mostly async with the existing status.Current pattern; cas-status is sync).
• pkg/server/server.go — route registration; new casProbeState field on APIServer so the conditional-put probe fires once per daemon lifetime, not once per request.
• pkg/backup/list.go — CollectRemoteCASBackups for the merged /backup/list response.

9. Tests

• pkg/cas/internal/fakedst/ — in-memory cas.Backend fake with hooks for error injection.
• pkg/cas/internal/testfixtures/ — synthetic local-backup builder (parts with deterministic hashes).
• test/integration/cas_*.go — testcontainer-driven end-to-end coverage across MinIO, Azurite, fake-gcs-server, OpenSSH-server SFTP, and proftpd FTP. The full CAS integration suite passes 19 PASS / 2 SKIP / 0 FAIL.

Test plan

Running on this branch:

• go test ./pkg/cas/... ./pkg/storage/... ./pkg/backup/... ./pkg/server/... -race -count=1 — ✅
• go vet ./... — ✅
• go test ./... whole tree, -short — ✅
• RUN_PARALLEL=1 go test -tags=integration ./test/integration/ -run TestCAS -timeout 90m — ✅ (19 PASS, 2 SKIP)

CI in .github/workflows/build.yaml will run the integration suite across the ClickHouse-version matrix on this PR.

What's deferred

The docs/cas-design.md §9 backlog tracks ~30 known follow-up items. None block v1 of CAS. The largest categories:

• §9.1 Major features (deferred): hash-verification on download (cas-verify --deep), object-disk parts, convergent encryption, cas-fsck, parallel cas-verify, distributed locking via S3 conditional create (beyond the same-name race we already close), local-disk / NFS target.
• §9.2 Performance / scalability: prune mark-phase parallelism (already partial; further gains possible), SweepOrphans spill-to-disk, streaming archive upload via io.Pipe, heap-merge for shardIter, replace ColdList with per-blob PutFileIfAbsent (architectural alternative).
• §9.3 Operability: --log-format=json for prune, populate BlobsTotal/OrphansHeldByGrace (already partial), per-blob progress logging.
• §9.4 Correctness defenses: pkg/pidlock TOCTOU (whole-tool, not CAS-specific), probe-key cleanup on prune, S3 IfNoneMatch startup probe (already shipped — listed for clarity).
• §9.5 Tests: tighten GCS/COS/FTP not-found tests to call real production code (mirror functions in test file today).
• §9.6 UX polish: cas-delete --force, wait_for_prune defaults docs, etc.

Known limitations (operator-facing)

In docs/cas-operator-runbook.md "Known limitations (v1)":

• --tables patterns are filepath.Match glob, not regex
• Object-disk tables refused (S3/Azure/HDFS/encrypted-over-S3)
• Multi-host concurrent upload to the same backup name unsupported (use unique names per writer)
• Hash verification on download is HEAD + size only
• No per-blob resumable uploads
• FTP requires opt-in allow_unsafe_markers for best-effort markers
• MinIO < RELEASE.2024-11-07 rejected by conditional-put probe
• No cross-cluster blob sharing

Phasing of the work in this branch

For maintainers who do want to see how the work landed: 8 implementation phases + 6 external review-feedback waves. Commit groups roughly correspond to:

• Phases 1–2: MVP upload+restore, then prune
• Phase 3: planner correctness (projections, encoded-vs-decoded names, empty tables)
• Phase 4: atomic markers across all 6 backends
• Phase 5: per-backend integration smoke tests
• Phase 6: P1 defects from external review wave 1 (marker-leak, dry-run-unlock, zero-ModTime classification, etc.)
• Phase 7 (cleanup round): ColdList TOCTOU re-validation, PruneReport counters, cfg.Validate() defensive guards
• Phase 8: wait_for_prune knob + REST API wiring
• Wave 5 + 6: external-review feedback rounds covering atomic-download, fail-closed object-disk preflight, encrypted-over-S3 detection, conditional-put probe per-process key, blob-upload byte verification, name-collision guards, async cas-delete, streaming verify, dry-run candidate API, and many smaller hardenings

git log --oneline master..HEAD shows the full sequence; commit subjects are tagged with the relevant fix-id.