Recoverable jobs

[jqnatividad]

For an existing resource, when a job fails after it has already been deleted from the datastore, there is no easy way to recover.

To avoid this, DP+ should:

• do a "soft-delete" of the table (perhaps, by temporarily renaming it?)
• do the DP+ job as usual
• if the job is successful, drop the "soft-deleted" table. If the job failed, restore the old table

[jqnatividad]

Status update against current main (59e08d7): partially addressed in two distinct dimensions.

Job-level recoverability (Prefect): DP+ v3 runs every ingest as a Prefect flow with per-task retry policies (@task(retries=N, retry_delay_seconds=...) on download_task / format_convert_task / etc.) and a _retry_if_transient callback that distinguishes deterministic 4xx errors and JobError (no retry) from transient failures (retry with backoff). So job-level resilience is now a Prefect concern — if a worker dies mid-run, Prefect re-queues the flow.

Datastore-level soft-delete (the actual ask): NOT done — the database stage still does a hard delete of the existing datastore table before COPY. The original idea ("rename → ingest → drop old on success / restore old on failure") would survive a failure where the data itself is unrecoverable but the operator's annotations are intact, but PostgreSQL doesn't have a free RENAME TABLE round-trip for tables with foreign-key views, and the in-flight datastore views would need re-pointing.

Partial workaround that DID land: #307 / issue #265 (just merged) stashes the Data Dictionary (per-field info dicts — labels, descriptions, type_overrides) to disk before the delete and restores it on failure. The data is still lost on failure, but the annotations are preserved, which was the most painful re-do.

Leaving open as the full table soft-delete is still a valid future ask, but priority is lower now that the worst-case data loss is just "row data" and not "annotations + row data".