Docker Compose environments for developing modern data platform architectures using Kafka, Flink, Spark, Iceberg, OpenLineage, OpenMetadata, Pinot, ClickHouse, StarRocks + Kpow & Flex by Factor House

Tools for processing sql queries metadata.

NextGen DataMotion Lineage

Covid19 and Iowa Liquor Sales analysis at BigQuery using dbt, Airflow, Marquez, Google Cloud and other modern data stack tools

Simple python script for easy local airflow deployment with docker. Packed with additional components. Will be adding more going forward.

A walkthrough of Correlator — an incident correlation engine that connects test failures to producing job runs to downstream impact across your data stack. The demo runs a Jaffle Shop pipeline with Airflow, dbt, and Great Expectations.

This Project aims to enhance Spark Lineage Generation in Open Metadata by creating Lineage edges from Table-to-Table with Column level lineage as well as Container to Table Lineage.

Testing a Docker deployment of Marquez and OpenLineage

Incident correlation engine for data teams — connects test failures to job runs via OpenLineage

Deploy MWAA with OpenLineage data lineage tracking using AWS CDK - Airflow 3.0.6, Marquez HA

dbt plugin that accelerates incident resolution with automated test failure correlation

Toy metadata-first catalog for preclinical biotech/pharma data — DuckDB catalog, MCP tool surface, Anthropic-SDK agent, Streamlit UI.

Exploring how to surface dlt pipeline metadata via Airflow's OpenLineage integration.

Data reliability and governance framework for quality controls, telemetry evidence, incident response, and ISO-aligned audit readiness.

Production-grade monitoring and feedback layer for a real-time ML platform: lineage, metrics, freshness checks, validation, and retraining signals.

Scan AWS Glue PySpark jobs in GitHub repositories to generate data lineage using AST analysis and LLMs.

Apache Airflow 2.9 multi-cloud medallion platform — Debezium CDC + Kafka + PySpark + Scala + Delta Lake across AWS/Azure/GCP via LocalStack/Azurite. OpenLineage + Marquez + DataHub + Great Expectations. Zero-cost docker-compose stack.

Airflow plugin that accelerates incident resolution with automated pipeline correlation

Accelerate Great Expectations incident resolution with automated correlation. Links validation failures to upstream data issues and provides direct navigation from incident to root cause.