[ISSUE] Intermittent 403 "Failed to retrieve tenant ID for given token" with Azure service principal is reported as an SDK parsing bug and not retried

Open
  • Is something wrong?
#1,628 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
57/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
azure, python
Domain
authentication, backend-api-design

Research direction

Start by reading _base_client._is_retryable, which the issue identifies as the retry decision point, and trace how plain-text 403 responses become PermissionDenied. The report has no deterministic reproduction or debug logs, so first look for existing retry and response parsing tests covering these paths. Done when a test captures the reported transient response behavior and the resulting retry or error handling matches the intended behavior.

Written by the indexing model from the issue text.

Description

Description

We intermittently get a 403 Forbidden with a plain-text body from the Unity Catalog API when authenticating as an Azure service principal (client secret). Most requests succeed; a fraction fail on the first call made by a freshly created WorkspaceClient.

The SDK raises PermissionDenied with the message below. The response body is plain text rather than JSON, so the SDK reports "unable to parse response ... likely a bug in the Databricks SDK":

databricks.sdk.errors.platform.PermissionDenied: unable to parse response. This is likely a bug in the Databricks SDK for Python or the underlying API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-py/issues. Request log:```GET /api/2.1/unity-catalog/tables/<catalog>.<schema>.<table>
< 403 Forbidden
< Failed to retrieve tenant ID for given token```

Two things seem worth looking at on the SDK side:

  1. The message is misleading. A plain-text 403 from the control plane is reported as a probable SDK bug, and as PermissionDenied, which suggests a missing privilege. The cause looks like an intermittent failure to validate the token, not a permissions problem, because the identical request succeeds moments later in other runs.
  2. It is not retried. _base_client._is_retryable only matches connection errors, timeouts, Retry-After responses and a fixed list of message substrings, and Failed to retrieve tenant ID for given token is not in that list. The error therefore surfaces immediately. We now work around it with our own retry around tables.get.

Reproduction

I do not have a deterministic reproduction; it is intermittent. The pattern is:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(
    host="https://<workspace>.azuredatabricks.net",
    azure_tenant_id="<tenant-id>",
    azure_client_id="<sp-client-id>",
    azure_client_secret="<sp-client-secret>",
)
w.tables.get("<catalog>.<schema>.<table>")  # occasionally fails with the 403 above

Each run is a new process that builds a new client and makes this call right away. The tenant ID is set explicitly and is correct, because the same configuration works for most runs.

Expected behavior

Either the request succeeds, or the SDK retries the transient token rejection. If it still fails, the error would ideally say the token could not be validated, not suggest an SDK bug or a missing permission.

Is it a regression?

Unknown. We upgraded from 0.139.0 to 0.156.0 around the time we first noticed it, but I cannot tell whether the errors started before that. I read the changelog for 0.140.0 to 0.162.0 and found nothing about Azure auth or error handling in that range.

Debug Logs

Not captured yet. The request log included in the exception is shown above.

Other Information

  • OS: Linux (Alpine-based container)
  • Python: 3.12
  • Version: databricks-sdk 0.156.0
  • Auth: Azure service principal with client secret (azure_client_id, azure_client_secret, azure_tenant_id)
  • Observed in two different Azure Databricks workspaces, each with its own service principal and tenant configuration.

Additional context

  • Similar message, different cause: terraform-provider-databricks#2723 reported the same text, but there the account ID was simply wrong, so it failed every time. Ours works most of the time with the same configuration.
  • Is this a known transient condition on the Databricks side, for example shortly after a token is issued? If so, treating it as retryable in the SDK would help every Azure service principal user.
Dominant language
Python
Stars
577
Forks
229
Avg merge
40m
Merged PRs (30d)
12

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/databricks-sdk-py

All issues in databricks/databricks-sdk-py

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.