VEDA Data Ingest

This application is to allow users to create PRs in a data repo such as veda-data to ingest data into the staging environment.

Quick Start

# Install dependencies
yarn install

# Set up local environment (see Environment Setup below)
cp .env.local.example .env.local
# Edit .env.local with your credentials

# Start development server
yarn dev

Feature Tour

The latest Playwright test report is published after each merge to main. This provides screenshots and descriptions of features in the veda-ingest-ui. https://nasa-impact.github.io/veda-ingest-ui/

Tech Stack

• Framework: Next.js 15+ with App Router
• Form Generation: react-jsonschema-form for dynamic forms
• UI Components: Ant Design React framework
• GitHub API: @octokit/rest for GitHub operations
• Authentication: Keycloak via NextAuth.js
• Testing: Vitest + Playwright for comprehensive testing
• Deployment: AWS Amplify with serverless architecture

📁 Project Structure

├── app/                    # Next.js App Router pages
│   ├── api/               # API routes for GitHub operations and STAC API
│   ├── collections/       # Collection management pages
│   ├── datasets/          # Dataset management pages
│   ├── edit-existing-collection/ # STAC collection editing interface
│   └── upload/            # File upload functionality
├── components/            # Reusable React components
│   ├── ingestion/        # Form components for data ingestion and editing
│   ├── layout/           # Layout components (header, sidebar, etc.)
│   ├── rjsf-components/  # Custom RJSF form components
│   ├── thumbnails/       # Thumbnail upload components
│   └── ui/               # General UI components
├── FormSchemas/          # JSON schemas for form generation
├── hooks/                # Custom React hooks
├── lib/                  # Utility libraries and configurations
├── types/                # TypeScript type definitions
├── utils/                # Helper functions and GitHub utilities
└── __tests__/            # Test suites (unit, integration, e2e)

Architecture

The application supports two primary workflows:

1. Data Ingestion

The application allows users to create and edit PRs in the data repository for data ingestion. New PRs are created with a prefix of '[collection/dataset] Ingest Request for [collectionName]'. The branch name and file name of the json for these new PRs is set by the Collection Name field in the form after any non-alphanumeric characters are removed from the collection name:

const fileName = 'ingestion-data/staging/dataset-config/${collectionName}.json';
const branchName = `feat/${collectionName}`;

Users are allowed to edit open PRs that are modifying json files in the standard filepath for each ingestion type. The existing values in the json will be loaded into a form. A user can update those values and a new commit will be added to the PR with the new values.

2. Collection Editing

The application also provides direct editing of existing STAC collections through the STAC API. User must have stac:collection:update scope from keycloak for editing permissions.

• Collection Discovery: Browse existing collections from /api/stac/collections
• Real-time Editing: Modify collection metadata directly without GitHub PRs
• Data Sanitization: Automatic STAC schema compliance with null-to-array/object conversion and datetime format fixes. This helps clean legacy formatting errors in veda-data.

Authentication & Authorization

All API calls require users to be authenticated via Keycloak.

• GitHub Operations: Uses GitHub token for repository operations
• STAC Operations: Uses access token for STAC API calls with tenant-based permissions
• Tenant Filtering: Support for multi-tenant environments with proper access controls

Creation Component Architecture

graph TD
    subgraph "Creation Flow"
        A[CreationFormManager] -->|Sends POST Request| API_POST[API];
        A -->|Receives 'ingestionType' prop| B{Render based on type};

        subgraph "UI & Validation"
            C[DatasetIngestionForm]
            C --> C_E{Render UI Tabs};
            C_E --> C_F[RJSF Form];
            C_E --> C_G[JSON Editor];
            C_F --> C_H["FormSchemas/datasets"];
            C_G --> C_H;
        end

        subgraph "UI & Validation"
            D[CollectionIngestionForm]
            D --> D_E{Render UI Tabs};
            D_E --> D_F[RJSF Form];
            D_E --> D_G[JSON Editor];
            D_F --> D_I["FormSchemas/collections"];
            D_G --> D_I;
        end

        B -- "dataset" --> C;
        B -- "collection" --> D;
    end

    style A fill:#0B3D91,stroke:#fff,stroke-width:2px,color:#fff
    style B fill:#BCC6CC,stroke:#333,stroke-width:2px,color:#000
    style C fill:#A4D3EE,stroke:#333,stroke-width:2px,color:#000
    style D fill:#A4D3EE,stroke:#333,stroke-width:2px,color:#000
    style API_POST fill:#FC3D21,stroke:#333,stroke-width:2px

Loading

Edit Component Architecture

graph TD
    subgraph "Edit Flow"
        A[EditFormManager] -->|Sends PUT Request| API_PUT[API];
        A -->|Receives 'ingestionType' & 'disableName' props| B{Render based on type};

        subgraph "Dataset Form"
            C_Edit[DatasetIngestionForm]
            C_Edit --Name fields disabled--> C_E_Edit{Render UI Tabs};
            C_E_Edit --> C_F_Edit[RJSF Form];
            C_E_Edit --> C_G_Edit[JSON Editor];
            C_F_Edit --> C_H_Edit["FormSchemas/datasets"];
            C_G_Edit --> C_H_Edit;
        end

        subgraph "Collection Form"
            D_Edit[CollectionIngestionForm]
            D_Edit --Name fields disabled--> D_E_Edit{Render UI Tabs};
            D_E_Edit --> D_F_Edit[RJSF Form];
            D_E_Edit --> D_G_Edit[JSON Editor];
            D_F_Edit --> D_I_Edit["FormSchemas/collections"];
            D_G_Edit --> D_I_Edit;
        end

        B -- "dataset" --> C_Edit;
        B -- "collection" --> D_Edit;
    end

    style A fill:#0B3D91,stroke:#fff,stroke-width:2px,color:#fff
    style B fill:#BCC6CC,stroke:#333,stroke-width:2px,color:#000
    style C_Edit fill:#A4D3EE,stroke:#333,stroke-width:2px,color:#000
    style D_Edit fill:#A4D3EE,stroke:#333,stroke-width:2px,color:#000
    style API_PUT fill:#FC3D21,stroke:#333,stroke-width:2px

Loading

Requirements

To set up the development environment for this website, you'll need to install the following on your system:

• Node (see version in .nvmrc) (To manage multiple node versions we recommend nvm)
• Yarn Package manager

If you use nvm, activate the desired Node version:

Installation

Install Node + package manager this repo depends on.

nvm install
npm -g install yarn

Then install project dependencies by running the yarn install.

yarn install

Usage

🔐 Environment Setup

Local Development

Configuration uses environment files that are never committed to version control for security.

1. Create your local environment file:

   cp .env.local.example .env.local

2. Configure required variables in .env.local:

   GitHub App Configuration:

   APP_ID=your-app-id                    # GitHub App ID
   INSTALLATION_ID=your-installation-id   # GitHub App installation ID
   GITHUB_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
   ...your private key here...
   -----END RSA PRIVATE KEY-----"

   AWS Configuration:

   ASSUME_ROLE_ARN="arn:aws:iam::account-id:role/role-name"
   INGEST_UI_EXTERNAL_ID="your-external-id"

   Authentication:

   NEXTAUTH_SECRET="your-secret-key"     # Generate with: openssl rand -base64 32
   NEXTAUTH_URL="http://localhost:3000"
   KEYCLOAK_CLIENT_ID="ingest-ui"
   KEYCLOAK_CLIENT_SECRET="your-client-secret"
   NEXT_PUBLIC_KEYCLOAK_ISSUER="https://your-keycloak-server/realms/veda"

   Development Options:

   NEXT_PUBLIC_APP_ENV="local"           # or "veda", "eic", "disasters"
   NEXT_PUBLIC_DISABLE_AUTH=true         # Bypass Keycloak for local dev without keycloak scope configuration
   NEXT_PUBLIC_MOCK_SCOPES="dataset:update,stac:collection:update"
   # NEXT_PUBLIC_MOCK_TENANTS=tenant1,tenant2

3. Verify .env.local is gitignored:

   git check-ignore .env.local  # Should output: .env.local

AWS Amplify Deployment

For production deployments, this app loads runtime secrets from AWS Secrets Manager.

1. Navigate to: AWS Amplify Console -> Your App -> App settings -> Environment variables

2. Add required Amplify environment variables (plain env vars):

• APP_RUNTIME_SECRET_ID - ARN of the Secrets Manager secret containing runtime JSON
• APP_ID - GitHub App ID
• INSTALLATION_ID - GitHub App installation ID
• ASSUME_ROLE_ARN - AWS IAM role ARN used for STS AssumeRole for uploading Thumbnails
• KEYCLOAK_CLIENT_ID - Keycloak client ID
• NEXTAUTH_URL - Base URL for NextAuth
• NEXT_PUBLIC_KEYCLOAK_ISSUER - Keycloak issuer URL
• NEXT_PUBLIC_APP_ENV - Environment profile (local, veda, eic, disasters)

3. Create a Secrets Manager secret (in the same account/region as your Amplify compute) with a JSON object in this shape:

{
  "GITHUB_PRIVATE_KEY": "-----BEGIN RSA PRIVATE KEY-----\\n...\\n-----END RSA PRIVATE KEY-----\\n",
  "KEYCLOAK_CLIENT_SECRET": "your-keycloak-client-secret",
  "NEXTAUTH_SECRET": "your-nextauth-secret",
  "INGEST_UI_EXTERNAL_ID": "your-external-id"
}

Notes:

• Keep these keys at the top level (no wrapper object).
• Use escaped newlines (\\n) for the private key value.
• Set APP_RUNTIME_SECRET_ID environment variable to this secret ARN or name.

4. Add IAM permission to the Amplify server runtime role (the SSR Lambda execution role, not only the build role). Ensure the ARN matches your actual roles.

  {
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::12345:role/thumbnail-uploader"
},
{
  "Sid": "ReadVedaIngestUiSecrets",
  "Effect": "Allow",
  "Action": [
   "secretsmanager:GetSecretValue",
   "secretsmanager:DescribeSecret"
  ],
  "Resource": "arn:aws:secretsmanager:us-west-2:12345:secret:veda-ingest-ui/*"
}

5. Runtime fallback behavior: If Secrets Manager is unavailable, the app falls back to matching environment variables (GITHUB_PRIVATE_KEY, KEYCLOAK_CLIENT_SECRET, NEXTAUTH_SECRET, INGEST_UI_EXTERNAL_ID).

Github Access

GitHub access is handled via a GitHub App installed on the target repository. See Github Destination Repo Configuration section below for setup.

Running the app

To preview the app use:

yarn dev

This will start the app and make it available at http://localhost:3000/.

To bypass the keycloak login, set the NEXT_PUBLIC_DISABLE_AUTH environment variable to true. This variable is also leveraged for Playwright testing.

🛠️ STAC Data Sanitization

To fix incorrect, previously ingested data, the application includes a data sanitization system to ensure STAC schema compliance:

Sanitization Features

• Null Handling: Converts null values to appropriate empty arrays or objects
• Datetime Format: Fixes timezone and separator issues (e.g., +00 → +00:00, space → T)

Implementation

Sanitization logic is located in utils/stacSanitization.ts and includes:

// Main sanitization function
import { sanitizeFormData } from '@/utils/stacSanitization';

const cleanedData = sanitizeFormData(formData);

Field Type Rules

• Arrays: stac_extensions, keywords, providers, links
• Objects: assets, item_assets, summaries
• Datetime Strings: Temporal extent fields with format fixes

Configuring the Validation Form

The fields in the Validation Form are configured by a combination of the json schema in the jsonschema.json file and the UI Schema in the uischema.json file. To modify fields in the form, a developer must update the json schema to include the proper JSON schema data fields and then modify the ui Schema to have any new or renamed fields in the desired location.

The Form uses a 24 column grid format and the layout of each row is dictated by the "ui:grid" array in that json. Each row is defined as an object with each field allowed up to 24 columns wide. For example:

  "ui:grid": [
    {
      "collection": 4,
      "title": 4,
      "license": 4,
      "description": 12
    },
    ...
  ]

the new first row has 4 fields with a combined width of 24. Nested objects in the field can be defined with their own grid. For example,

  "spatial_extent": {
    "ui:grid": [
      {
        "xmin": 12,
        "ymin": 12
      },
      {
        "xmax": 12,
        "ymax": 12
      }
    ]
  },

Github Destination Repo Configuration

To allow the veda-ingest-ui to open PRs in a repo, a Github app must be installed on the destination repo and several environment variables are needed from that Github app installation. Follow the Installing your own GitHub App guide from github to get started:

1. Uncheck the "Active" checkbox under webhook. No webhook is required.
2. Ensure the app has Read and Write permissions to Contents and Pull Requests.
3. Create and save a Private Key to place in your env variables.
4. Copy the App ID and Client ID from the new github app's overview.
5. Copy the Installation ID from the repo's list of Installed GitHub Apps. The Installation ID is found in the URL for that application. For example, https://github.com/settings/installations/[Installation ID] or https://github.com/organizations/[ORGANIZATION]/settings/installations/[Installation ID]