[Bug] Python client not able to connect to self-hosted Temporal server via proxy using authorization header

从报告中的 src/utils.py 内的 Client.connect 调用和 src/run_worker.py 中的 worker 流程开始，然后使用所述的服务器和 SDK 版本通过代理复现故障。将 60 秒连接失败及后续的轮询错误与 issue #631 进行比较；完成标准是 Python worker 保持连接，并通过授权标头成功进行轮询。

What are you really trying to do?

We’re trying to connect Python client to our self-hosted Temporal server via a proxy using an authorization header.

Describe the bug

We see that it is able to connect to the worker at startup, but after exactly 60 seconds, it fails with the below error. Note that we do see some transient 503 errors while connecting to the server; but after that it’s able to connect successfully (verified in server logs).

Minimal Reproduction

Here’ the set of errors we see:
1.

Error: Client connection not established Failed client connect: `get_system_info` call error after connection: Status { code: PermissionDenied, message: "encountered c2c service api error response 503", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:11:39 GMT"} }, source: None }.
Traceback (most recent call last):
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 202, in <module>
    asyncio.run(main())
  File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 157, in main
    await conn.create_client()
  File "/home/temporal/temporal_sample_worker_python/src/utils.py", line 105, in create_client
    self.client = await Client.connect("dev.api.data.falcon.sfdcfc.net:7443",
  File "/usr/local/lib64/python3.9/site-packages/temporalio/client.py", line 176, in connect
    await temporalio.service.ServiceClient.connect(connect_config),
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 209, in connect
    return await _BridgeServiceClient.connect(config)
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 1117, in connect
    await client._connected_client()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 1130, in _connected_client
    self._bridge_client = await temporalio.bridge.client.Client.connect(
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/client.py", line 97, in connect
    await temporalio.bridge.temporal_sdk_bridge.connect_client(
RuntimeError: Failed client connect: `get_system_info` call error after connection: Status { code: PermissionDenied, message: "encountered c2c service api error response 503", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:11:39 GMT"} }, source: None }

2025-03-19T02:17:48.974543Z  WARN temporal_sdk_core::worker::workflow::wft_poller: Error while polling for workflow tasks error=Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }
2025-03-19T02:17:48.974547Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None })
2025-03-19T02:17:48.974551Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None })
2025-03-19 02:17:48,975 - ERROR - Worker failed, shutting down
Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 154, in run
    act = await self._bridge_worker().poll_workflow_activation()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/worker.py", line 147, in poll_workflow_activation
    await self._ref.poll_workflow_activation()
RuntimeError: Poll failure: Unhandled grpc error when polling: Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 164, in run
    raise RuntimeError("Workflow worker failed") from err
RuntimeError: Workflow worker failed
2025-03-19 02:17:48,975 - INFO - Beginning worker shutdown, will wait 0:00:00 before cancelling activities

Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 154, in run
    act = await self._bridge_worker().poll_workflow_activation()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/worker.py", line 147, in poll_workflow_activation
    await self._ref.poll_workflow_activation()
RuntimeError: Poll failure: Unhandled grpc error when polling: Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 202, in <module>
    asyncio.run(main())
  File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 179, in main
    await run_worker(client, taskqueue_name, args.worker_type)
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 142, in run_worker
    await worker.run()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_worker.py", line 571, in run
    raise exception
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 164, in run
    raise RuntimeError("Workflow worker failed") from err
RuntimeError: Workflow worker failed

Environment/Versions

• Temporal Version: Server v1.26.2, SDK v1.10.0

Additional context

Our issue looks similar to this this open bug: [Bug] Worker hangs after polling workflow task queue · Issue #631 · temporalio/sdk-python · GitHub
Wanted to check if it is same issue and if not we need help in understanding the fixing it.