You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat: Add AUDIT_ONLY model kind for multi-table validation
Introduces a new model kind that validates data relationships across
multiple tables without materializing results. Combines model benefits
(DAG participation, dependencies) with audit behavior (validation only).
- Add AUDIT_ONLY to ModelKindName enum and create AuditOnlyKind class
- Implement AuditOnlyStrategy for execution without materialization
- Add comprehensive unit and integration tests
- Update documentation with usage examples and best practices
- Add three example models to sushi project demonstrating use cases
Copy file name to clipboardExpand all lines: docs/concepts/audits.md
+101Lines changed: 101 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -722,3 +722,104 @@ MODEL (
722
722
)
723
723
);
724
724
```
725
+
726
+
### AUDIT_ONLY Models
727
+
728
+
In addition to traditional audits, SQLMesh provides a special model kind called `AUDIT_ONLY` for validating data relationships across multiple tables without materializing any results.
729
+
730
+
#### When to Use AUDIT_ONLY Models
731
+
732
+
Use `AUDIT_ONLY` models when you need to:
733
+
- Validate relationships between multiple tables (e.g., referential integrity)
734
+
- Run complex validation queries that don't belong to a single model
735
+
- Create validation logic that participates in the model DAG with proper dependencies
736
+
- Avoid creating unnecessary materialized tables just for validation
737
+
738
+
Unlike traditional audits that are scoped to a single model, `AUDIT_ONLY` models can depend on multiple models and validate relationships between them.
739
+
740
+
#### Creating AUDIT_ONLY Models
741
+
742
+
AUDIT_ONLY models are defined like regular models but with `kind AUDIT_ONLY`:
743
+
744
+
```sql
745
+
MODEL (
746
+
name data_quality.order_validation,
747
+
kind AUDIT_ONLY (
748
+
blocking TRUE, -- Fail pipeline if validation fails (default)
749
+
max_failing_rows 20-- Number of sample rows to show in error (default: 10)
750
+
),
751
+
depends_on [orders, customers],
752
+
cron '@daily',
753
+
owner 'data_quality_team'
754
+
);
755
+
756
+
-- Query should return 0 rows for success
757
+
-- Any returned rows indicate validation failures
758
+
SELECT
759
+
o.order_id,
760
+
o.customer_id,
761
+
'Missing customer record'as issue_type
762
+
FROM orders o
763
+
LEFT JOIN customers c ONo.customer_id=c.customer_id
764
+
WHEREc.customer_id IS NULL;
765
+
```
766
+
767
+
#### Key Differences from Regular Audits
768
+
769
+
| Feature | Traditional Audits | AUDIT_ONLY Models |
Copy file name to clipboardExpand all lines: docs/concepts/models/model_kinds.md
+109Lines changed: 109 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -860,6 +860,115 @@ SELECT DISTINCT
860
860
FROM db.employees;
861
861
```
862
862
863
+
## AUDIT_ONLY
864
+
865
+
The `AUDIT_ONLY` model kind is designed for data validation across multiple tables without materializing any results. These models execute validation queries and fail if any rows are returned, similar to [audits](../audits.md#audit_only-models) but with the ability to participate as full models in the DAG.
866
+
867
+
### Purpose
868
+
869
+
`AUDIT_ONLY` models are ideal for:
870
+
- Validating referential integrity between multiple tables
871
+
- Detecting data quality issues across different models
872
+
- Running complex validation queries that don't belong to a single model
873
+
- Avoiding unnecessary table materialization for validation purposes
874
+
875
+
### Configuration
876
+
877
+
The `AUDIT_ONLY` kind supports two configuration parameters:
Copy file name to clipboardExpand all lines: docs/reference/model_configuration.md
+11Lines changed: 11 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -306,6 +306,17 @@ Configuration options for [`SCD_TYPE_2_BY_COLUMN` models](../concepts/models/mod
306
306
307
307
Python model kind `name` enum value: [ModelKindName.SCD_TYPE_2_BY_COLUMN](https://sqlmesh.readthedocs.io/en/stable/_readthedocs/html/sqlmesh/core/model/kind.html#ModelKindName)
308
308
309
+
### `AUDIT_ONLY` models
310
+
311
+
Configuration options for [`AUDIT_ONLY` models](../concepts/models/model_kinds.md#audit_only) (in addition to [general model properties](#general-model-properties)).
|`blocking`| If set to true, the pipeline will fail when the validation query returns any rows. If false, only warnings are logged. (Default: `True`) | bool | N |
316
+
|`max_failing_rows`| Maximum number of failing rows to display in error messages when a validation fails. (Default: `10`) | int | N |
317
+
318
+
Python model kind `name` enum value: [ModelKindName.AUDIT_ONLY](https://sqlmesh.readthedocs.io/en/stable/_readthedocs/html/sqlmesh/core/model/kind.html#ModelKindName)
319
+
309
320
### `SEED` models
310
321
311
322
Configuration options for [`SEED` models](../concepts/models/model_kinds.md#seed). `SEED` models do not support all the general properties supported by other models; they only support the properties listed in this table.
0 commit comments