Skip to content

Commit 7e86140

Browse files
mildbytemildbyte
committed
Fix autogenerated README for the project
Elaborate on needing to edit the `splitgraph.yml` file
1 parent 65e9d43 commit 7e86140

2 files changed

Lines changed: 224 additions & 0 deletions

File tree

  • splitgraph/cloud/project
  • test/splitgraph/cloud/project/snapshots/test_generation/test_generate_project_with_dbt/generate_project_dbt

‎splitgraph/cloud/project/templates.py‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,118 @@
132132
"SQL credentials"). You can get them at https://www.splitgraph.com/settings/sql-credentials (or
133133
your deployment URL if you're on a private deployment).
134134
135+
### Edit `splitgraph.yml`
136+
137+
We generated a [`splitgraph.yml`](./splitgraph.yml) file from your chosen plugins'
138+
parameters JSONSchema. You should review it and add suitable plugin settings:
139+
140+
- set `tables` to `tables: {}` to let the plugin automatically infer the schema and the
141+
options of the data source (by default, it adds a sample table into the project file)
142+
- change and customize the `metadata` block
143+
- set up the plugin parameters in `external.params`. Where the comment says `CHOOSE ONE`
144+
and offers a list of alternative subobjects, choose one entry from the list and delete
145+
the list itself, leaving the object at the top level.
146+
147+
Example:
148+
149+
```yaml
150+
- namespace: my_namespace
151+
repository: csv
152+
# Catalog-specific metadata for the repository. Optional.
153+
metadata:
154+
readme:
155+
text: Readme
156+
description: Description of the repository
157+
topics:
158+
- sample_topic
159+
# Data source settings for the repository. Optional.
160+
external:
161+
# Name of the credential that the plugin uses. This can also be a credential_id if the
162+
# credential is already registered on Splitgraph.
163+
credential: csv
164+
plugin: csv
165+
# Plugin-specific parameters matching the plugin's parameters schema
166+
params:
167+
connection: # Choose one of:
168+
- connection_type: http # REQUIRED. Constant
169+
url: '' # REQUIRED. HTTP URL to the CSV file
170+
- connection_type: s3 # REQUIRED. Constant
171+
s3_endpoint: '' # REQUIRED. S3 endpoint (including port if required)
172+
s3_bucket: '' # REQUIRED. Bucket the object is in
173+
s3_region: '' # Region of the S3 bucket
174+
s3_secure: false # Whether to use HTTPS for S3 access
175+
s3_object: '' # Limit the import to a single object
176+
s3_object_prefix: '' # Prefix for object in S3 bucket
177+
autodetect_header: true # Detect whether the CSV file has a header automatically
178+
autodetect_dialect: true # Detect the CSV file's dialect (separator, quoting characters etc) automatically
179+
autodetect_encoding: true # Detect the CSV file's encoding automatically
180+
autodetect_sample_size: 65536 # Sample size, in bytes, for encoding/dialect/header detection
181+
schema_inference_rows: 100000 # Number of rows to use for schema inference
182+
encoding: utf-8 # Encoding of the CSV file
183+
ignore_decode_errors: false # Ignore errors when decoding the file
184+
header: true # First line of the CSV file is its header
185+
delimiter: ',' # Character used to separate fields in the file
186+
quotechar: '"' # Character used to quote fields
187+
tables:
188+
sample_table:
189+
# Plugin-specific table parameters matching the plugin's schema
190+
options:
191+
url: '' # HTTP URL to the CSV file
192+
s3_object: '' # S3 object of the CSV file
193+
autodetect_header: true # Detect whether the CSV file has a header automatically
194+
autodetect_dialect: true # Detect the CSV file's dialect (separator, quoting characters etc) automatically
195+
autodetect_encoding: true # Detect the CSV file's encoding automatically
196+
autodetect_sample_size: 65536 # Sample size, in bytes, for encoding/dialect/header detection
197+
schema_inference_rows: 100000 # Number of rows to use for schema inference
198+
encoding: utf-8 # Encoding of the CSV file
199+
ignore_decode_errors: false # Ignore errors when decoding the file
200+
header: true # First line of the CSV file is its header
201+
delimiter: ',' # Character used to separate fields in the file
202+
quotechar: '"' # Character used to quote fields
203+
# Schema of the table, a list of objects with `name` and `type`. If set to `[]`, will infer.
204+
schema: []
205+
# Whether live querying is enabled for the plugin (creates a "live" tag in the
206+
# repository proxying to the data source). The plugin must support live querying.
207+
is_live: true
208+
# Ingestion schedule settings. Disable this if you're using GitHub Actions or other methods
209+
# to trigger ingestion.
210+
schedule:
211+
```
212+
213+
becomes:
214+
215+
```yaml
216+
- namespace: my_namespace
217+
repository: csv
218+
metadata:
219+
readme:
220+
text: Readme
221+
description: Description of the repository
222+
topics:
223+
- sample_topic
224+
external:
225+
# No credential required since we're querying a CSV file over HTTP
226+
plugin: csv
227+
# Plugin-specific parameters matching the plugin's parameters schema
228+
params:
229+
connection:
230+
connection_type: http # REQUIRED. Constant
231+
url: 'https://people.sc.fsu.edu/~jburkardt/data/csv/airtravel.csv' # REQUIRED. HTTP URL to the CSV file
232+
autodetect_header: true # Detect whether the CSV file has a header automatically
233+
autodetect_dialect: true # Detect the CSV file's dialect (separator, quoting characters etc) automatically
234+
autodetect_encoding: true # Detect the CSV file's encoding automatically
235+
autodetect_sample_size: 65536 # Sample size, in bytes, for encoding/dialect/header detection
236+
schema_inference_rows: 100000 # Number of rows to use for schema inference
237+
encoding: utf-8 # Encoding of the CSV file
238+
ignore_decode_errors: false # Ignore errors when decoding the file
239+
header: true # First line of the CSV file is its header
240+
delimiter: ',' # Character used to separate fields in the file
241+
quotechar: '"' # Character used to quote fields
242+
# Automatically infer table parameters
243+
tables: {}
244+
is_live: true
245+
```
246+
135247
### Set up GitHub Actions
136248
137249
Because this repository was itself generated by a GitHub Actions job, we can't edit the workflow

‎test/splitgraph/cloud/project/snapshots/test_generation/test_generate_project_with_dbt/generate_project_dbt/README.md‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,118 @@ repository and create the following secrets:
3939
"SQL credentials"). You can get them at https://www.splitgraph.com/settings/sql-credentials (or
4040
your deployment URL if you're on a private deployment).
4141

42+
### Edit `splitgraph.yml`
43+
44+
We generated a [`splitgraph.yml`](./splitgraph.yml) file from your chosen plugins'
45+
parameters JSONSchema. You should review it and add suitable plugin settings:
46+
47+
- set `tables` to `tables: {}` to let the plugin automatically infer the schema and the
48+
options of the data source (by default, it adds a sample table into the project file)
49+
- change and customize the `metadata` block
50+
- set up the plugin parameters in `external.params`. Where the comment says `CHOOSE ONE`
51+
and offers a list of alternative subobjects, choose one entry from the list and delete
52+
the list itself, leaving the object at the top level.
53+
54+
Example:
55+
56+
```yaml
57+
- namespace: my_namespace
58+
repository: csv
59+
# Catalog-specific metadata for the repository. Optional.
60+
metadata:
61+
readme:
62+
text: Readme
63+
description: Description of the repository
64+
topics:
65+
- sample_topic
66+
# Data source settings for the repository. Optional.
67+
external:
68+
# Name of the credential that the plugin uses. This can also be a credential_id if the
69+
# credential is already registered on Splitgraph.
70+
credential: csv
71+
plugin: csv
72+
# Plugin-specific parameters matching the plugin's parameters schema
73+
params:
74+
connection: # Choose one of:
75+
- connection_type: http # REQUIRED. Constant
76+
url: '' # REQUIRED. HTTP URL to the CSV file
77+
- connection_type: s3 # REQUIRED. Constant
78+
s3_endpoint: '' # REQUIRED. S3 endpoint (including port if required)
79+
s3_bucket: '' # REQUIRED. Bucket the object is in
80+
s3_region: '' # Region of the S3 bucket
81+
s3_secure: false # Whether to use HTTPS for S3 access
82+
s3_object: '' # Limit the import to a single object
83+
s3_object_prefix: '' # Prefix for object in S3 bucket
84+
autodetect_header: true # Detect whether the CSV file has a header automatically
85+
autodetect_dialect: true # Detect the CSV file's dialect (separator, quoting characters etc) automatically
86+
autodetect_encoding: true # Detect the CSV file's encoding automatically
87+
autodetect_sample_size: 65536 # Sample size, in bytes, for encoding/dialect/header detection
88+
schema_inference_rows: 100000 # Number of rows to use for schema inference
89+
encoding: utf-8 # Encoding of the CSV file
90+
ignore_decode_errors: false # Ignore errors when decoding the file
91+
header: true # First line of the CSV file is its header
92+
delimiter: ',' # Character used to separate fields in the file
93+
quotechar: '"' # Character used to quote fields
94+
tables:
95+
sample_table:
96+
# Plugin-specific table parameters matching the plugin's schema
97+
options:
98+
url: '' # HTTP URL to the CSV file
99+
s3_object: '' # S3 object of the CSV file
100+
autodetect_header: true # Detect whether the CSV file has a header automatically
101+
autodetect_dialect: true # Detect the CSV file's dialect (separator, quoting characters etc) automatically
102+
autodetect_encoding: true # Detect the CSV file's encoding automatically
103+
autodetect_sample_size: 65536 # Sample size, in bytes, for encoding/dialect/header detection
104+
schema_inference_rows: 100000 # Number of rows to use for schema inference
105+
encoding: utf-8 # Encoding of the CSV file
106+
ignore_decode_errors: false # Ignore errors when decoding the file
107+
header: true # First line of the CSV file is its header
108+
delimiter: ',' # Character used to separate fields in the file
109+
quotechar: '"' # Character used to quote fields
110+
# Schema of the table, a list of objects with `name` and `type`. If set to `[]`, will infer.
111+
schema: []
112+
# Whether live querying is enabled for the plugin (creates a "live" tag in the
113+
# repository proxying to the data source). The plugin must support live querying.
114+
is_live: true
115+
# Ingestion schedule settings. Disable this if you're using GitHub Actions or other methods
116+
# to trigger ingestion.
117+
schedule:
118+
```
119+
120+
becomes:
121+
122+
```yaml
123+
- namespace: my_namespace
124+
repository: csv
125+
metadata:
126+
readme:
127+
text: Readme
128+
description: Description of the repository
129+
topics:
130+
- sample_topic
131+
external:
132+
# No credential required since we're querying a CSV file over HTTP
133+
plugin: csv
134+
# Plugin-specific parameters matching the plugin's parameters schema
135+
params:
136+
connection:
137+
connection_type: http # REQUIRED. Constant
138+
url: 'https://people.sc.fsu.edu/~jburkardt/data/csv/airtravel.csv' # REQUIRED. HTTP URL to the CSV file
139+
autodetect_header: true # Detect whether the CSV file has a header automatically
140+
autodetect_dialect: true # Detect the CSV file's dialect (separator, quoting characters etc) automatically
141+
autodetect_encoding: true # Detect the CSV file's encoding automatically
142+
autodetect_sample_size: 65536 # Sample size, in bytes, for encoding/dialect/header detection
143+
schema_inference_rows: 100000 # Number of rows to use for schema inference
144+
encoding: utf-8 # Encoding of the CSV file
145+
ignore_decode_errors: false # Ignore errors when decoding the file
146+
header: true # First line of the CSV file is its header
147+
delimiter: ',' # Character used to separate fields in the file
148+
quotechar: '"' # Character used to quote fields
149+
# Automatically infer table parameters
150+
tables: {}
151+
is_live: true
152+
```
153+
42154
### Set up GitHub Actions
43155
44156
Because this repository was itself generated by a GitHub Actions job, we can't edit the workflow

0 commit comments

Comments
 (0)