Skip to content

Commit 1390f0f

Browse files
committed
add example documentation
1 parent 290dc31 commit 1390f0f

1 file changed

Lines changed: 35 additions & 1 deletion

File tree

docs/integrations/engines/duckdb.md

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
| `extensions` | Extension to load into duckdb. Only autoloadable extensions are supported. | list | N |
1919
| `connector_config` | Configuration to pass into the duckdb connector. | dict | N |
2020
| `secrets` | Configuration for authenticating external sources (e.g., S3) using DuckDB secrets. | dict | N |
21+
| `file_systems` | Configuration for registering `fsspec` filesystems to the DuckDB connection. secrets. | dict | N |
2122

2223
#### DuckDB Catalogs Example
2324

@@ -256,4 +257,37 @@ After configuring the secrets, you can directly reference S3 paths in your catal
256257

257258
Refer to the official DuckDB documentation for the full list of [supported S3 secret parameters](https://duckdb.org/docs/stable/extensions/httpfs/s3api.html#overview-of-s3-secret-parameters) and for more information on the [Secrets Manager configuration](https://duckdb.org/docs/configuration/secrets_manager.html).
258259

259-
> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh.
260+
> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh.
261+
262+
##### File system configuration example for Microsoft Onelake
263+
264+
The `file_systems` accepts a list of file systems to register in the DuckDB connection. This is especially useful for Azure Storage Accounts, as it adds write support for DuckDB which is not natively supported by DuckDB (yet).
265+
266+
267+
=== "YAML"
268+
269+
```yaml linenums="1"
270+
gateways:
271+
ducklake:
272+
connection:
273+
type: duckdb
274+
catalogs:
275+
ducklake:
276+
type: ducklake
277+
path: myducklakecatalog.duckdb
278+
data_path: abfs://MyFabricWorkspace/MyFabricLakehouse.Lakehouse/Files/DuckLake.Files
279+
extensions:
280+
- ducklake
281+
file_systems:
282+
- protocol: abfs
283+
storage_options:
284+
account_name: onelake
285+
account_host: onelake.blob.fabric.microsoft.com
286+
client_id: {{ env_var('AZURE_CLIENT_ID') }}
287+
client_secret: {{ env_var('AZURE_CLIENT_SECRET') }}
288+
tenant_id: {{ env_var('AZURE_TENANT_ID') }}
289+
# anon: False # To use azure.identity.DefaultAzureCredential authentication
290+
```
291+
292+
293+
Refer to the documentation for `fsspec` [fsspec.filesystem](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.filesystem) and `adlfs` [adlfs.AzureBlobFileSystem](https://fsspec.github.io/adlfs/api/#api-reference) for a full list of storage options.

0 commit comments

Comments
 (0)