Enable unified host support without flag (#1358)

hectorcast-db · web-flow · commit 78914fa48d07 · 2026-03-31T12:08:05.000Z
## 🥞 Stacked PR Use this [link](https://github.com/databricks/databricks-sdk-py/pull/1358/files) to review incremental changes. - [**stack/unified-host-ga**](#1358) [[Files changed](https://github.com/databricks/databricks-sdk-py/pull/1358/files)] ---------  ## Summary Remove the `experimental_is_unified_host` flag, which was not used anymore. Update README to document the new support, the updates to default authentication flow and auto detection. This PR includes no behavioral changes. Co-authored-by: Isaac ## How is this tested? N/A
diff --git a/NEXT_CHANGELOG.md b/NEXT_CHANGELOG.md
@@ -3,6 +3,7 @@
 ## Release v0.103.0
 
 ### New Features and Improvements
+* Add support for unified hosts. A single configuration profile can now be used for both account-level and workspace-level operations when the host supports it and both `account_id` and `workspace_id` are available. The `experimental_is_unified_host` flag has been removed; unified host detection is now automatic.
 * Accept `DATABRICKS_OIDC_TOKEN_FILEPATH` environment variable for consistency with other Databricks SDKs (Go, CLI, Terraform). The previous `DATABRICKS_OIDC_TOKEN_FILE` is still supported as an alias.
 
 ### Security
@@ -27,4 +28,4 @@
 * Add `cascade` field for `databricks.sdk.service.pipelines.DeletePipelineRequest`.
 * Add `default_branch` field for `databricks.sdk.service.postgres.ProjectSpec`.
 * Add `default_branch` field for `databricks.sdk.service.postgres.ProjectStatus`.
-* Add `ingress` and `ingress_dry_run` fields for `databricks.sdk.service.settings.AccountNetworkPolicy`.
+* Add `ingress` and `ingress_dry_run` fields for `databricks.sdk.service.settings.AccountNetworkPolicy`.
diff --git a/README.md b/README.md
@@ -92,6 +92,7 @@ The conventional name for the variable that holds the workspace-level client of
 ### In this section
 
 - [Default authentication flow](#default-authentication-flow)
+- [Unified host support](#unified-host-support)
 - [Databricks native authentication](#databricks-native-authentication)
 - [Azure native authentication](#azure-native-authentication)
 - [Overriding .databrickscfg](#overriding-databrickscfg)
@@ -107,10 +108,19 @@ in the following order, until it succeeds:
 
 1. [Databricks native authentication](#databricks-native-authentication)
 2. [Azure native authentication](#azure-native-authentication)
+3. [GCP native authentication](#google-cloud-platform-native-authentication)
 4. If the SDK is unsuccessful at this point, it returns an authentication error and stops running.
 
-You can instruct the Databricks SDK for Python to use a specific authentication method by setting the `auth_type` argument
-as described in the following sections.
+Each authentication method requires specific configuration attributes (e.g., `token` for PAT auth, `azure_client_id` for Azure service principal auth). The SDK automatically detects the cloud provider and skips authentication methods whose required configuration attributes are not present. This means that Azure-specific methods like `azure-cli` are automatically skipped when connecting to an AWS or GCP workspace, and vice versa for GCP-specific methods.
+
+To force a specific authentication method instead of relying on auto-detection, set the `auth_type` argument:
+
+```python
+from databricks.sdk import WorkspaceClient
+# Force Azure CLI authentication — skip all other methods
+w = WorkspaceClient(host='https://mycompany.databricks.com', auth_type='azure-cli', cloud='AZURE')
+```
+This is useful when your environment has credentials for multiple authentication methods and you want to ensure a specific one is used or when auto detection is not accurate.
 
 For each authentication method, the SDK searches for compatible authentication credentials in the following locations,
 in the following order. Once the SDK finds a compatible set of credentials that it can use, it stops searching:
@@ -125,6 +135,38 @@ in the following order. Once the SDK finds a compatible set of credentials that
 
 Depending on the Databricks authentication method, the SDK uses the following information. Presented are the `WorkspaceClient` and `AccountClient` arguments (which have corresponding `.databrickscfg` file fields), their descriptions, and any corresponding environment variables.
 
+### Unified host support
+
+Certain Databricks host types support both account-level and workspace-level API operations from a single endpoint. When using such a unified host, a single configuration profile can be used to create both `WorkspaceClient` and `AccountClient` instances without changing the `host`.
+
+For this to work, the following conditions must be met:
+
+1. The host must support unified operations.
+2. Both `account_id` and `workspace_id` must be available — either set explicitly in the configuration or auto-discovered.
+
+When both values are present, the SDK uses `workspace_id` to route workspace-level requests and `account_id` to route account-level requests, all through the same host.
+
+```ini
+# .databrickscfg
+[unified]
+host         = https://mycompany.databricks.com
+account_id   = 00000000-0000-0000-0000-000000000000
+workspace_id = 1234567890
+```
+
+```python
+from databricks.sdk import WorkspaceClient, AccountClient
+
+# Both clients share the same host and profile
+w = WorkspaceClient(profile='unified')
+a = AccountClient(profile='unified')
+
+# A WorkspaceClient for a different workspace under the same host and account
+w = WorkspaceClient(profile='unified', workspace_id='2345678901')
+```
+
+If the host supports it, `account_id` and `workspace_id` may be auto-discovered, reducing the required explicit configuration.
+
 ### Databricks native authentication
 
 By default, the Databricks SDK for Python initially tries [Databricks token authentication](https://docs.databricks.com/dev-tools/api/latest/authentication.html) (`auth_type='pat'` argument). If the SDK is unsuccessful, it then tries Workload Identity Federation (WIF). See [Supported WIF](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-federation-provider) for the supported JWT token providers.
@@ -133,10 +175,15 @@ By default, the Databricks SDK for Python initially tries [Databricks token auth
 - For Databricks OIDC authentication, you must provide the `host`, `client_id` and `token_audience` _(optional)_ either directly, through the corresponding environment variables, or in your `.databrickscfg` configuration file.
 - For Azure DevOps OIDC authentication, the `token_audience` is irrelevant as the audience is always set to `api://AzureADTokenExchange`. Also, the `System.AccessToken` pipeline variable required for OIDC request must be exposed as the `SYSTEM_ACCESSTOKEN` environment variable, following [Pipeline variables](https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#systemaccesstoken)
 
+During initialization, the SDK automatically resolves missing configuration fields (`account_id`, `workspace_id`, `cloud`, and `discovery_url`). Any explicitly provided values take precedence and are never overwritten. If the auto discovery fails, the SDK falls back to the explicit configuration. It is recommended to always set explicit configuration.
+
 | Argument         | Description                                                                                                                                                                                                                                                               | Environment variable    |
 |------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|
-| `host`           | _(String)_ The Databricks host URL for either the Databricks workspace endpoint or the Databricks accounts endpoint.                                                                                                                                                      | `DATABRICKS_HOST`       |     
-| `account_id`     | _(String)_ The Databricks account ID for the Databricks accounts endpoint. Only has effect when `Host` is either `https://accounts.cloud.databricks.com/` _(AWS)_, `https://accounts.azuredatabricks.net/` _(Azure)_, or `https://accounts.gcp.databricks.com/` _(GCP)_.  | `DATABRICKS_ACCOUNT_ID` |
+| `host`           | _(String)_ The Databricks host URL for either the Databricks workspace endpoint or the Databricks accounts endpoint.                                                                                                                                                      | `DATABRICKS_HOST`       |
+| `account_id`     | _(String)_ The Databricks account ID for the Databricks accounts endpoint. Auto-discovered if not provided.  | `DATABRICKS_ACCOUNT_ID` |
+| `workspace_id`   | _(String)_ The Databricks workspace ID for the Databricks workspace endpoint. Auto-discovered if not provided. | `DATABRICKS_WORKSPACE_ID` |
+| `cloud`          | _(String)_ The cloud provider for the Databricks workspace (`AWS`, `AZURE`, or `GCP`). Auto-discovered if not provided. When set, `is_aws`, `is_azure`, and `is_gcp` use this value directly instead of inferring from hostname. | `DATABRICKS_CLOUD` |
+| `discovery_url`  | _(String)_ The OpenID Connect discovery URL. Auto-discovered if not provided. When set, OIDC endpoints are fetched directly from this URL instead of using the default host-based well-known endpoint logic. | `DATABRICKS_DISCOVERY_URL` |
 | `token`          | _(String)_ The Databricks personal access token (PAT) _(AWS, Azure, and GCP)_ or Azure Active Directory (Azure AD) token _(Azure)_.                                                                                                                                       | `DATABRICKS_TOKEN`      |
 | `client_id`      | _(String)_ The Databricks Service Principal Application ID.                                                                                                                                                                                                               | `DATABRICKS_CLIENT_ID`  |
 | `token_audience` | _(String)_ When using Workload Identity Federation, the audience to specify when fetching an ID token from the ID token supplier.                                                                                                                                         | `TOKEN_AUDIENCE`        |
@@ -528,7 +575,7 @@ useragent.with_partner("partner-xyz")
 
 `with_product()` can be used to define the name and version of the product that is built with the Databricks SDK for Python. The product name has the same restrictions as the partner name above, and the product version must be a valid [SemVer](https://semver.org/). Subsequent calls to `with_product()` replace the original product with the new user-specified one.
 
-```go
+```python
 from databricks.sdk import useragent
 useragent.with_product("databricks-example-product", "1.2.0")
 ```
diff --git a/databricks/sdk/config.py b/databricks/sdk/config.py
@@ -91,14 +91,11 @@ class Config:
     account_id: str = ConfigAttribute(env="DATABRICKS_ACCOUNT_ID")
     workspace_id: str = ConfigAttribute(env="DATABRICKS_WORKSPACE_ID")
 
-    # Experimental flag to indicate if the host is a unified host (supports both workspace and account APIs)
-    experimental_is_unified_host: bool = ConfigAttribute(env="DATABRICKS_EXPERIMENTAL_IS_UNIFIED_HOST")
-
-    # [Experimental] Cloud provider. When set, is_aws/is_azure/is_gcp use this value directly
+    # Cloud provider. When set, is_aws/is_azure/is_gcp use this value directly
     # instead of inferring from hostname. Populated automatically from /.well-known/databricks-config.
     cloud: Cloud = ConfigAttribute(env="DATABRICKS_CLOUD", transform=_parse_cloud)
 
-    # [Experimental] OpenID Connect discovery URL. When set, OIDC endpoints are fetched directly
+    # OpenID Connect discovery URL. When set, OIDC endpoints are fetched directly
     # from this URL instead of the default host-type-based well-known endpoint logic.
     discovery_url: str = ConfigAttribute(env="DATABRICKS_DISCOVERY_URL")
 
diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py
@@ -95,7 +95,7 @@ def unified_config(env_or_skip) -> Config:
     config = Config()
     config.host = env_or_skip("UNIFIED_HOST")
     config.workspace_id = env_or_skip("TEST_WORKSPACE_ID")
-    config.experimental_is_unified_host = True
+
     config._fix_host_if_needed()
     return config
 
diff --git a/tests/test_config.py b/tests/test_config.py
@@ -317,7 +317,6 @@ def test_client_type_workspace():
         host="https://unified.databricks.com",
         workspace_id="test-workspace",
         account_id="test-account",
-        experimental_is_unified_host=True,
         token="test-token",
     )
     assert config.client_type == ClientType.WORKSPACE
@@ -352,7 +351,6 @@ def test_is_account_client_does_not_raise_on_unified_host():
     """Test that is_account_client raises ValueError when used with unified hosts."""
     config = Config(
         host="https://unified.databricks.com",
-        experimental_is_unified_host=True,
         workspace_id="test-workspace",
         token="test-token",
     )
@@ -464,7 +462,6 @@ def test_workspace_org_id_header_on_unified_host(requests_mock):
         host="https://unified.databricks.com",
         account_id="test-account",
         workspace_id="test-workspace-123",
-        experimental_is_unified_host=True,
         token="test-token",
     )
 
@@ -487,7 +484,6 @@ def test_not_workspace_org_id_header_on_unified_host_on_account_endpoint(request
         host="https://unified.databricks.com",
         account_id="test-account",
         workspace_id="test-workspace-123",
-        experimental_is_unified_host=True,
         token="test-token",
     )
 
@@ -727,7 +723,7 @@ def test_databricks_oidc_endpoints_uses_discovery_url(requests_mock):
                 "account_id": _DUMMY_ACCOUNT_ID,
                 "workspace_id": _DUMMY_WORKSPACE_ID,
             },
-            {"experimental_is_unified_host": True},
+            {},
             {
                 "account_id": _DUMMY_ACCOUNT_ID,
                 "workspace_id": _DUMMY_WORKSPACE_ID,
@@ -738,7 +734,7 @@ def test_databricks_oidc_endpoints_uses_discovery_url(requests_mock):
         pytest.param(
             _DUMMY_ACC_HOST,
             {"oidc_endpoint": f"{_DUMMY_ACC_HOST}/oidc/accounts/{{account_id}}"},
-            {"account_id": _DUMMY_ACCOUNT_ID, "experimental_is_unified_host": True},
+            {"account_id": _DUMMY_ACCOUNT_ID},
             {
                 "discovery_url": f"{_DUMMY_ACC_HOST}/oidc/accounts/{_DUMMY_ACCOUNT_ID}/.well-known/oauth-authorization-server"
             },
@@ -750,7 +746,6 @@ def test_databricks_oidc_endpoints_uses_discovery_url(requests_mock):
             {
                 "account_id": _DUMMY_ACCOUNT_ID,
                 "workspace_id": _DUMMY_WORKSPACE_ID,
-                "experimental_is_unified_host": True,
             },
             {"account_id": _DUMMY_ACCOUNT_ID, "workspace_id": _DUMMY_WORKSPACE_ID},
             id="unified-does-not-overwrite-existing-fields",
@@ -771,7 +766,7 @@ def test_resolve_host_metadata_missing_account_id(mocker):
         return_value=oauth.HostMetadata.from_dict({"oidc_endpoint": f"{_DUMMY_ACC_HOST}/oidc/accounts/{{account_id}}"}),
     )
     with pytest.raises(ValueError, match="account_id is required to resolve discovery_url"):
-        Config(host=_DUMMY_ACC_HOST, token="t", experimental_is_unified_host=True)
+        Config(host=_DUMMY_ACC_HOST, token="t")
 
 
 def test_resolve_host_metadata_no_oidc_endpoint(mocker):
@@ -780,7 +775,7 @@ def test_resolve_host_metadata_no_oidc_endpoint(mocker):
         "databricks.sdk.config.get_host_metadata",
         return_value=oauth.HostMetadata.from_dict({"account_id": _DUMMY_ACCOUNT_ID}),
     )
-    config = Config(host=_DUMMY_WS_HOST, token="t", experimental_is_unified_host=True)
+    config = Config(host=_DUMMY_WS_HOST, token="t")
     assert config.account_id == _DUMMY_ACCOUNT_ID
     assert config.discovery_url is None
 
@@ -791,7 +786,7 @@ def test_resolve_host_metadata_http_error(mocker):
         "databricks.sdk.config.get_host_metadata",
         side_effect=ValueError(f"Failed to fetch host metadata from {_DUMMY_WS_HOST}/.well-known/databricks-config"),
     )
-    config = Config(host=_DUMMY_WS_HOST, token="t", experimental_is_unified_host=True)
+    config = Config(host=_DUMMY_WS_HOST, token="t")
     assert config.account_id is None
     assert config.discovery_url is None
 
@@ -887,7 +882,7 @@ def test_resolve_host_metadata_populates_cloud(mocker):
             }
         ),
     )
-    config = Config(host=_DUMMY_WS_HOST, token="t", experimental_is_unified_host=True)
+    config = Config(host=_DUMMY_WS_HOST, token="t")
     assert config.cloud == Cloud.AWS
 
 
@@ -905,7 +900,6 @@ def test_resolve_host_metadata_cloud_not_overwritten(mocker):
     config = Config(
         host=_DUMMY_WS_HOST,
         token="t",
-        experimental_is_unified_host=True,
         cloud="AWS",
     )
     assert config.cloud == Cloud.AWS
@@ -917,7 +911,7 @@ def test_resolve_host_metadata_cloud_missing_in_response(mocker):
         "databricks.sdk.config.get_host_metadata",
         return_value=oauth.HostMetadata.from_dict({"oidc_endpoint": f"{_DUMMY_WS_HOST}/oidc"}),
     )
-    config = Config(host=_DUMMY_WS_HOST, token="t", experimental_is_unified_host=True)
+    config = Config(host=_DUMMY_WS_HOST, token="t")
     assert config.cloud is None
 
 
@@ -941,7 +935,6 @@ def test_resolve_host_metadata_sets_token_audience_for_account_host(mocker):
         host=_DUMMY_ACC_HOST,
         token="t",
         account_id=_DUMMY_ACCOUNT_ID,
-        experimental_is_unified_host=True,
     )
     assert config.token_audience == _DUMMY_ACCOUNT_ID
 
@@ -958,7 +951,7 @@ def test_resolve_host_metadata_no_token_audience_for_workspace_host(mocker):
             }
         ),
     )
-    config = Config(host=_DUMMY_WS_HOST, token="t", experimental_is_unified_host=True)
+    config = Config(host=_DUMMY_WS_HOST, token="t")
     assert config.token_audience is None
 
 
@@ -977,7 +970,6 @@ def test_resolve_host_metadata_does_not_overwrite_token_audience(mocker):
         host=_DUMMY_ACC_HOST,
         token="t",
         account_id=_DUMMY_ACCOUNT_ID,
-        experimental_is_unified_host=True,
         token_audience="custom-audience",
     )
     assert config.token_audience == "custom-audience"
diff --git a/tests/test_credentials_provider.py b/tests/test_credentials_provider.py
@@ -321,7 +321,7 @@ def test_account_client_passes_account_id(self, mocker):
         mock_cfg = Mock()
         mock_cfg.profile = None
         mock_cfg.host = "https://accounts.cloud.databricks.com"
-        mock_cfg.experimental_is_unified_host = False
+
         mock_cfg.account_id = "test-account-id"
         mock_cfg.client_type = ClientType.ACCOUNT
         mock_cfg.databricks_cli_path = "/path/to/databricks"
@@ -348,7 +348,7 @@ def test_profile_uses_profile_flag_with_host_fallback(self, mocker):
         mock_cfg = Mock()
         mock_cfg.profile = "my-profile"
         mock_cfg.host = "https://workspace.databricks.com"
-        mock_cfg.experimental_is_unified_host = False
+
         mock_cfg.databricks_cli_path = "/path/to/databricks"
         mock_cfg.disable_async_token_refresh = False
 
