DP+ 3.0 tracking issue

[jqnatividad]

DP+ 3.0 will make the Data Resource Upload Flow (DRUF) that was developed for the Texas Water Data Hub generally available.

The DRUF is interactive FAIRification, featuring:

• ckanext-scheming 3.0 powered Dataset Form Pages with a standard Upload page in the beginning, and a Review page at the end.
• metadata formulas - Excel like formula DSL using jinja2 that can calculate/suggest metadata. They can either be pre-calculated and assigned automatically or offered as suggestions.
• Extensible formula helper library - since the formulas are written in jinja2, additional helpers can be easily added. The formula helpers will prioritize precalculating DCAT-US 3 recommended optional properties that are otherwise too hard to calculate manually (DCAT-US 3 in DP+ tracking issue #218)
• suggestion UX/UI - a Suggestion UI using Bootstrap popovers that show the Suggestion and the Formula.

---

• apply scheming 3.0 changes
• reorganize project for maintainability Feature/reorganize project structure #264
  (DO THIS FIRST, BEFORE ITEMS BELOW)
• Commit after truncate to reduce lock contention #263 — merged 2025-11-23 (69f5481); truncate in its own txn so the AccessExclusive lock is held for the shortest possible window, eliminating UI hangs on large COPY-into-datastore jobs.
• incorporate TX Form Pages implementation - specifically, the Registration Formpage and the Upload formpage in the beginning, and the Review Formpage at the end.
• incorporate AI-powered suggestions for Description and Data Dictionary using qsv describegpt — backend in feat(ai-suggestions): optional qsv describegpt stage for LLM-derived metadata #301 (fdc3714), scheming UI in feat(ai-suggestions): scheming UI for AI-derived metadata (PR #253 follow-up) #302 (ad28ebc), qsv-flag + envelope-shape fixes from LM Studio E2E in fix(ai-suggestions): real-qsv schema + CLI flag corrections (PR #301/#302 follow-up) #303, and Vitest + jsdom unit suite in test(js): vitest + jsdom unit tests for scheming-ai-suggestions.js (+ fixes 1 polling bug) #304 (which also caught a production bug where the polling JS was reading STATUS from the wrong path and would never have terminated). Opt-in via ckanext.datapusher_plus.enable_ai_suggestions (off by default; needs an OpenAI-compatible endpoint configured in qsv's describegpt prompt-file). Per-field opt-in via ai_suggestion: true on the scheming field config; AI buttons appear next to opted-in fields on markdown / text / select form snippets, polling JS reads suggestions from package["dpp_suggestions"]["ai_suggestions"] and applies on click.
• extend qsv describegpt so that it can use a controlled vocabulary for suggested tags. The controlled vocabulary will be configurable and point to the specified CKAN resource with the following columns tag, description, AI guidance. AI guidance will be customizable instructions when/how to use the associated tag.
• leverage qsv's delimiter inferencing (see Failing to detect separator for semicolon separated file #256) — done in efb254f; format_converter.py:319-340 now sniffs and normalizes non-comma CSVs
• Automatic Empty Date Range handling #261
• make safenames configurable #262
• Stash existing data dictionary before doing a DP+ job. If DP+job fails, the old Data Dictionary should be restored. #265 — stash + restore Data Dictionary across DP+ job failures, landed in fix(#265): stash Data Dictionary before delete, restore on rollback #307 (59e08d7). Analysis stage writes a small per-resource JSON snapshot to disk before deleting the existing datastore; _rollback_database reads it back on failure and re-creates the datastore with the original info dicts (and proper type derived from type_override) and zero rows. A retry branch in AnalysisStage._parse_stats re-loads the stash when no live datastore exists, so the dictionary survives even when the failure happened outside the database transaction. Stash mtime is surfaced in restore logs so operators can spot stale-restore vs. genuine retry. New knob ckanext.datapusher_plus.dictionary_stash_dir (defaults to <tempdir>/dpp_dict_stash).

[tino097]

with #264 changes, we will reorganize the project structure which will help in better maintainability and testability. It will help the whole project to move into better direction and help to identify what can be improved in future.

In the upcoming period we will merge this into main branch so please sync the forks which will help with the merging of all PR's from the users

cc @jqnatividad

[tino097]

Two development branches have been created:

• Dev 2.0
• Dev 3.0

For the next 2–3 months, dev-v2.0 will be kept in sync with main. After that, dev-v3.0 will be merged into main, becoming the primary codebase. The dev-v2.0 branch will remain available for a limited transition period.

My intention is to move Datapusher Plus to the refactored code (v3.0) within the next 2 months, then allow some time for users to migrate to the new version. After the transition period, v1.0 and v2.0 will be deprecated and dropped.

@jqnatividad

Merry Christmas 🎄