[ISSUE] Intermittent 403 "Failed to retrieve tenant ID for given token" with Azure service principal is reported as an SDK parsing bug and not retried

Đang mở
  • Có gì không ổn?
#1,628 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
57/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
azure, python
Lĩnh vực
authentication, backend-api-design

Hướng nghiên cứu

Start by reading _base_client._is_retryable, which the issue identifies as the retry decision point, and trace how plain-text 403 responses become PermissionDenied. The report has no deterministic reproduction or debug logs, so first look for existing retry and response parsing tests covering these paths. Done when a test captures the reported transient response behavior and the resulting retry or error handling matches the intended behavior.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Description

We intermittently get a 403 Forbidden with a plain-text body from the Unity Catalog API when authenticating as an Azure service principal (client secret). Most requests succeed; a fraction fail on the first call made by a freshly created WorkspaceClient.

The SDK raises PermissionDenied with the message below. The response body is plain text rather than JSON, so the SDK reports "unable to parse response ... likely a bug in the Databricks SDK":

databricks.sdk.errors.platform.PermissionDenied: unable to parse response. This is likely a bug in the Databricks SDK for Python or the underlying API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-py/issues. Request log:```GET /api/2.1/unity-catalog/tables/<catalog>.<schema>.<table>
< 403 Forbidden
< Failed to retrieve tenant ID for given token```

Two things seem worth looking at on the SDK side:

  1. The message is misleading. A plain-text 403 from the control plane is reported as a probable SDK bug, and as PermissionDenied, which suggests a missing privilege. The cause looks like an intermittent failure to validate the token, not a permissions problem, because the identical request succeeds moments later in other runs.
  2. It is not retried. _base_client._is_retryable only matches connection errors, timeouts, Retry-After responses and a fixed list of message substrings, and Failed to retrieve tenant ID for given token is not in that list. The error therefore surfaces immediately. We now work around it with our own retry around tables.get.

Reproduction

I do not have a deterministic reproduction; it is intermittent. The pattern is:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(
    host="https://<workspace>.azuredatabricks.net",
    azure_tenant_id="<tenant-id>",
    azure_client_id="<sp-client-id>",
    azure_client_secret="<sp-client-secret>",
)
w.tables.get("<catalog>.<schema>.<table>")  # occasionally fails with the 403 above

Each run is a new process that builds a new client and makes this call right away. The tenant ID is set explicitly and is correct, because the same configuration works for most runs.

Expected behavior

Either the request succeeds, or the SDK retries the transient token rejection. If it still fails, the error would ideally say the token could not be validated, not suggest an SDK bug or a missing permission.

Is it a regression?

Unknown. We upgraded from 0.139.0 to 0.156.0 around the time we first noticed it, but I cannot tell whether the errors started before that. I read the changelog for 0.140.0 to 0.162.0 and found nothing about Azure auth or error handling in that range.

Debug Logs

Not captured yet. The request log included in the exception is shown above.

Other Information

  • OS: Linux (Alpine-based container)
  • Python: 3.12
  • Version: databricks-sdk 0.156.0
  • Auth: Azure service principal with client secret (azure_client_id, azure_client_secret, azure_tenant_id)
  • Observed in two different Azure Databricks workspaces, each with its own service principal and tenant configuration.

Additional context

  • Similar message, different cause: terraform-provider-databricks#2723 reported the same text, but there the account ID was simply wrong, so it failed every time. Ours works most of the time with the same configuration.
  • Is this a known transient condition on the Databricks side, for example shortly after a token is issued? If so, treating it as retryable in the SDK would help every Azure service principal user.
Ngôn ngữ chính
Python
Star
577
Fork
229
Merge trung bình
40 phút
Pull request đã merge (30 ngày)
12

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của databricks/databricks-sdk-py

Tất cả issue của databricks/databricks-sdk-py

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.