|
1 | | -# data-caterer-example |
| 1 | +# Data Caterer Examples moved to main repository found [here](https://github.com/data-catering/data-caterer/tree/main/example) |
2 | 2 |
|
3 | 3 |  |
4 | 4 |
|
5 | | -Data Caterer is a metadata driven test data management tool that aids in creating production like data across batch and |
6 | | -event data systems. Run data validations to ensure your systems have ingested it as expected. Use the Java, Scala API, |
7 | | -UI, or YAML files to help with setup or customisation that are all run via Docker. |
8 | | - |
9 | | -This repo contains example Java and Scala API usage for Data Caterer. |
10 | | - |
11 | | - |
12 | | - |
13 | | -## How |
14 | | - |
15 | | -Check out any of [the Scala examples](src/main/scala/io/github/datacatering/plan) or |
16 | | -[Java examples](src/main/java/io/github/datacatering/plan). If you are looking for more information, you can follow |
17 | | -[**detailed documentation found here**](https://data.catering/latest/docs/). |
18 | | - |
19 | | -Not comfortable with Java or Scala? No worries. |
20 | | -[You can use the UI via these steps](https://data.catering/latest/get-started/quick-start/). |
21 | | - |
22 | | -Want some YAML instead? Also, no worries. Check the example [plan](docker/data/custom/plan) and |
23 | | -[task](docker/data/custom/task) YAML files here. |
24 | | - |
25 | | -### Java |
26 | | - |
27 | | -1. Create new Java class similar |
28 | | - to [DocumentationJavaPlanRun.java](src/main/java/io/github/datacatering/plan/DocumentationJavaPlanRun.java) |
29 | | - 1. Needs to extend `io.github.datacatering.datacaterer.javaapi.api.PlanRun` |
30 | | - |
31 | | -### Scala |
32 | | - |
33 | | -1. Create new Scala class similar |
34 | | - to [DocumentationPlanRun.scala](src/main/scala/io/github/datacatering/plan/DocumentationPlanRun.scala) |
35 | | - 1. Needs to extend `io.github.datacatering.datacaterer.api.PlanRun` |
36 | | - |
37 | | -### YAML |
38 | | - |
39 | | -1. Copy existing plan file (such as [foreign-key.yaml](docker/data/custom/plan/foreign-key.yaml)) in directory |
40 | | -[docker/data/custom/plan](docker/data/custom/plan) |
41 | | -2. Copy existing task file (such as [json-account-task.yaml](docker/data/custom/task/file/json/json-account-task.yaml)) |
42 | | -in directory [docker/data/custom/task](docker/data/custom/task) |
43 | | - 1. If you want to run data validations, copy the file [simple-validation.yaml](docker/data/custom/validation/simple-validation.yaml) |
44 | | - and add validation to plan via: |
45 | | - ```yaml |
46 | | - validations: |
47 | | - - "<name of validation (i.e. account_checks)>" |
48 | | - ``` |
49 | | -3. [Use JSON schema to help creating metadata for plan, tasks or validations](schema/data-caterer-latest.json). |
50 | | -You can import this schema into your IDE for validation of your YAML files. Links below show how you can import the schema: |
51 | | -- [IntelliJ](https://www.jetbrains.com/help/idea/json.html#ws_json_schema_add_custom) |
52 | | -- [VS Code](https://code.visualstudio.com/docs/languages/json#_json-schemas-and-settings) |
53 | | -
|
54 | | -
|
55 | | -## Run |
56 | | -
|
57 | | -Requires: |
58 | | -
|
59 | | -- Docker |
60 | | -
|
61 | | -```shell |
62 | | -./run.sh |
63 | | -#check results under docker/sample/report/index.html folder |
64 | | -``` |
65 | | - |
66 | | -## Docker |
67 | | - |
68 | | -Create your own Docker image via: |
69 | | - |
70 | | -```shell |
71 | | -./gradlew clean build |
72 | | -docker build -t <my_image_name>:<my_image_tag> . |
73 | | -docker run -e PLAN_CLASS=io.github.datacatering.plan.DocumentationPlanRun -v ${PWD}/docs/run:/opt/app/data <my_image_name>:<my_image_tag> |
74 | | -#check results under docs/run folder |
75 | | -``` |
76 | | - |
77 | | -## Docker Compose |
78 | | - |
79 | | -Run with own class from either Java or Scala API: |
80 | | - |
81 | | -```shell |
82 | | -./gradlew clean build |
83 | | -cd docker |
84 | | -PLAN_CLASS=io.github.datacatering.plan.DocumentationPlanRun DATA_SOURCE=postgres docker-compose up -d datacaterer |
85 | | -``` |
86 | | - |
87 | | -[Details from docs](https://data.catering/latest/get-started/quick-start/). |
88 | | -Docker compose sample found under `docker` folder. |
89 | | - |
90 | | -```shell |
91 | | -cd docker |
92 | | -docker-compose up -d datacaterer |
93 | | -``` |
94 | | - |
95 | | -Check result under [here](docker/data/custom). |
96 | | - |
97 | | -Change to another data source via: |
98 | | - |
99 | | -- postgres |
100 | | -- mysql |
101 | | -- cassandra |
102 | | -- solace |
103 | | -- kafka |
104 | | -- http |
105 | | - |
106 | | -```shell |
107 | | -DATA_SOURCE=cassandra docker-compose up -d datacaterer |
108 | | -``` |
109 | | - |
110 | | -### Run with YAML files |
111 | | - |
112 | | -Example YAML files can be found here: |
113 | | -- [Plan](docker/data/custom/plan): Define tasks, data sources, foreign keys, etc. to run |
114 | | -- [Task](docker/data/custom/task): Define data generation details such as schema and number of records |
115 | | -- [Validation](docker/data/custom/validation): Define data validation details to run on data sources |
116 | | - |
117 | | -If you want to use a [different YAML plan](docker/data/custom/plan) for the data source, you can run: |
118 | | - |
119 | | -```shell |
120 | | -PLAN=plan/postgres-multiple-tables DATA_SOURCE=postgres docker-compose up -d datacaterer |
121 | | -``` |
122 | | - |
123 | | -## Helm |
124 | | - |
125 | | -```shell |
126 | | -helm install data-caterer ./data-caterer-example/helm/data-caterer |
127 | | -``` |
128 | | - |
129 | | -## Benchmarks |
130 | | - |
131 | | -Base benchmark tests can be run via: |
132 | | - |
133 | | -```shell |
134 | | -bash benchmark/run_benchmark.sh |
135 | | -``` |
136 | | - |
137 | | -Results can be found under [benchmark/results](benchmark/results). |
0 commit comments