Vector storage

[minhajuddin2510]

No description provided.

[Copilot]

Pull Request Overview

This PR adds end-to-end vector store support to DataPusher Plus, embedding resource data via a local SentenceTransformer model and querying via ChromaDB and OpenRouter.

• Introduces a new DataPusherVectorStore class for embedding, querying, and managing vector data.
• Integrates vector embedding into the upload job pipeline with optional temporal coverage extraction.
• Adds configuration settings and a helper to check embedding status.

Reviewed Changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.

File	Description
ckanext/datapusher_plus/vector_store.py	New module implementing vector store integration
ckanext/datapusher_plus/jobs.py	Hooks embedding into the datapusher job pipeline
ckanext/datapusher_plus/helpers.py	Adds helper to query embedding status
ckanext/datapusher_plus/config.py	Adds configuration flags and defaults for vector store

Comments suppressed due to low confidence (2)

ckanext/datapusher_plus/jobs.py:1599

• The new vector store embedding workflow in the job pipeline lacks corresponding unit or integration tests. Consider adding tests to cover DataPusherVectorStore.embed_resource and the job integration path.

    if conf.ENABLE_VECTOR_STORE and VECTOR_STORE_AVAILABLE:

ckanext/datapusher_plus/jobs.py:1638

• The function parsedate is not imported in this module, causing a NameError at runtime. Add the appropriate import (e.g., from dateutil.parser import parse as parsedate).

                                        min_year = parsedate(str(min_date)).year

[Copilot]

A default OpenRouter API key is hard-coded in source. This poses a security risk; consider loading it exclusively from a secure environment variable.

[jqnatividad]

Thanks @minhajuddin2510 for this work. Before this can land, a few concerns to flag for a follow-up version:

Description. The PR body is empty and the change adds a new vector_store.py module with ~730 lines plus modifications to config.py, plugin.py, and requirements.txt. For a feature addition this size, a description covering (a) what the vector store is for in DP+'s pipeline, (b) which embedding/store backend it uses and why, (c) operator-facing config (any new ckanext.datapusher_plus.* keys), (d) integration points (Prefect tasks? CKAN actions? CLI?) is essential before review.

Rebase needed. The PR modifies ckanext/datapusher_plus/jobs.py, which the Prefect 3 migration (#277) deleted. Vector-storage hooks now belong in either an existing stage (jobs/stages/) or — if it's a per-domain side concern — a Prefect subflow primitive in jobs/subflows.py (which #289 added precisely for this kind of optional per-domain step).

Could you close + reopen with a description + rebase against current main? Vector embeddings on ingestion is a real and welcome feature; it just needs the design surfaced so reviewers can engage with it.