Merge branch 'main' of https://github.com/TobikoData/sqlmesh

Author: sungchun12

README.md

@@ -145,8 +145,7 @@ python -m venv .venv
 source .venv/bin/activate
 pip install 'sqlmesh[lsp]' # install the sqlmesh package with extensions to work with VSCode
 source .venv/bin/activate # reactivate the venv to ensure you're using the right installation
-sqlmesh init duckdb # get started right away with a local duckdb instance
-sqlmesh plan # see the plan for the changes you're making
+sqlmesh init # follow the prompts to get started (choose DuckDB)
 ```
 
 </details>
@@ -163,13 +162,12 @@ python -m venv .venv
 .\.venv\Scripts\Activate.ps1
 pip install 'sqlmesh[lsp]' # install the sqlmesh package with extensions to work with VSCode
 .\.venv\Scripts\Activate.ps1 # reactivate the venv to ensure you're using the right installation
-sqlmesh init duckdb # get started right away with a local duckdb instance
-sqlmesh plan # see the plan for the changes you're making
+sqlmesh init # follow the prompts to get started (choose DuckDB)
 ```
 </details>
 
 
-Follow the [quickstart guide](https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/#1-create-the-sqlmesh-project) to learn how to use SQLMesh. You already have a head start!
+Follow the [quickstart guide](https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/) to learn how to use SQLMesh. You already have a head start!
 
 Follow the [crash course](https://sqlmesh.readthedocs.io/en/stable/examples/sqlmesh_cli_crash_course/) to learn the core movesets and use the easy to reference cheat sheet. 
 

docs/integrations/engines/duckdb.md

@@ -18,6 +18,7 @@
 | `extensions`       | Extension to load into duckdb. Only autoloadable extensions are supported.                                                                                                                                                                      |  list  |    N     |
 | `connector_config` | Configuration to pass into the duckdb connector.                                                                                                                                                                                                |  dict  |    N     |
 | `secrets`   | Configuration for authenticating external sources (e.g., S3) using DuckDB secrets.                                                                                                                                                                                                |  dict  |    N     |
+| `filesystems`   | Configuration for registering `fsspec` filesystems to the DuckDB connection.                                                                                                                                                                                                |  dict  |    N     |
 
 #### DuckDB Catalogs Example
 
@@ -256,4 +257,36 @@ After configuring the secrets, you can directly reference S3 paths in your catal
 
 Refer to the official DuckDB documentation for the full list of [supported S3 secret parameters](https://duckdb.org/docs/stable/extensions/httpfs/s3api.html#overview-of-s3-secret-parameters) and for more information on the [Secrets Manager configuration](https://duckdb.org/docs/configuration/secrets_manager.html).
 
-> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh.
+> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh.
+
+##### File system configuration example for Microsoft Onelake
+
+The `filesystems` accepts a list of file systems to register in the DuckDB connection. This is especially useful for Azure Storage Accounts, as it adds write support for DuckDB which is not natively supported by DuckDB (yet).
+
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    gateways:
+      ducklake:
+        connection:
+          type: duckdb
+          catalogs:
+            ducklake:
+            type: ducklake
+            path: myducklakecatalog.duckdb
+            data_path: abfs://MyFabricWorkspace/MyFabricLakehouse.Lakehouse/Files/DuckLake.Files
+        extensions:
+          - ducklake
+        filesystems:
+          - fs: abfs
+            account_name: onelake
+            account_host: onelake.blob.fabric.microsoft.com
+            client_id: {{ env_var('AZURE_CLIENT_ID') }}
+            client_secret: {{ env_var('AZURE_CLIENT_SECRET') }}
+            tenant_id: {{ env_var('AZURE_TENANT_ID') }}
+            # anon: False # To use azure.identity.DefaultAzureCredential authentication 
+    ```
+
+
+Refer to the documentation for `fsspec` [fsspec.filesystem](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.filesystem) and `adlfs` [adlfs.AzureBlobFileSystem](https://fsspec.github.io/adlfs/api/#api-reference) for a full list of storage options. 

docs/integrations/overview.md

@@ -9,20 +9,20 @@ SQLMesh supports integrations with the following tools:
 * [Kestra](https://kestra.io/plugins/plugin-sqlmesh/tasks/cli/io.kestra.plugin.sqlmesh.cli.sqlmeshcli)
 
 ## Execution engines
-SQLMesh supports the following execution engines for running SQLMesh projects:
+SQLMesh supports the following execution engines for running SQLMesh projects (engine `type` in parentheses - example usage: `pip install "sqlmesh[databricks]"`):
 
-* [Athena](./engines/athena.md)
-* [Azure SQL](./engines/azuresql.md)
-* [BigQuery](./engines/bigquery.md)
-* [ClickHouse](./engines/clickhouse.md)
-* [Databricks](./engines/databricks.md)
-* [DuckDB](./engines/duckdb.md)
-* [MotherDuck](./engines/motherduck.md)
-* [MSSQL](./engines/mssql.md)
-* [MySQL](./engines/mysql.md)
-* [Postgres](./engines/postgres.md)
-* [GCP Postgres](./engines/gcp-postgres.md)
-* [Redshift](./engines/redshift.md)
-* [Snowflake](./engines/snowflake.md)
-* [Spark](./engines/spark.md)
-* [Trino](./engines/trino.md)
+* [Athena](./engines/athena.md) (athena)
+* [Azure SQL](./engines/azuresql.md) (azuresql)
+* [BigQuery](./engines/bigquery.md) (bigquery)
+* [ClickHouse](./engines/clickhouse.md) (clickhouse)
+* [Databricks](./engines/databricks.md) (databricks)
+* [DuckDB](./engines/duckdb.md) (duckdb)
+* [MotherDuck](./engines/motherduck.md) (motherduck)
+* [MSSQL](./engines/mssql.md) (mssql)
+* [MySQL](./engines/mysql.md) (mysql)
+* [Postgres](./engines/postgres.md) (postgres)
+* [GCP Postgres](./engines/gcp-postgres.md) (gcppostgres)
+* [Redshift](./engines/redshift.md) (redshift)
+* [Snowflake](./engines/snowflake.md) (snowflake)
+* [Spark](./engines/spark.md) (spark)
+* [Trino](./engines/trino.md) (trino)