HTTP/2 PROTOCOL_ERROR and TaskCanceledException during FCM requests FCM 请求出现 HTTP/2 PROTOCOL_ERROR 和 TaskCanceledException

Open
#591 1 comment 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
backend

Research direction

Start by tracing the FCM request send and retry path in the SDK, then inspect its HTTP client factory and handler configuration. Identify which diagnostics can distinguish cancellation sources and correlate retries with the final send result; done means providing evidence-based guidance on configuration, logging severity, and when these failures need investigation. The issue names no source files or tests.

Written by the indexing model from the issue text.

Description

Environment / 环境

  • FirebaseAdmin: Custom build based on 3.0.1 / 基于 3.0.1 的自定义构建
  • Product: FCM
  • OS: Linux (Kubernetes)
  • .NET: 10

Description / 问题描述

Over the past few days, we have frequently observed the following exceptions during FCM requests. Repeated cancellation exceptions are concentrated on one pod. No application cancellation token is passed in the normal send path.

最近几天,FCM 请求频繁出现以下异常,其中取消异常集中在一个 Pod 持续发生。普通发送路径未传入业务取消令牌。

HTTP/2 protocol error / HTTP/2 协议错误
System.Net.Http.HttpRequestException:
The HTTP/2 server sent invalid data on the connection.
HTTP/2 error code 'PROTOCOL_ERROR' (0x1).
 ---> System.Net.Http.HttpProtocolException
 at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(...)
 at System.Net.Http.Http2Connection.SendAsync(...)
 at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(...)
Cancellation / 取消异常
System.Threading.Tasks.TaskCanceledException:
The operation was canceled.
 at System.Net.Http.Http2Connection.Http2Stream.<>c.<WaitForDataAsync>b__88_0(...)
 at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(...)
 at System.Threading.TimerQueue.FireNextTimers()
--- End of stack trace from previous location ---
 at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(...)
 at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(...)

Stack traces are shortened and sanitized. / 堆栈已精简并脱敏。

HTTP configuration / HTTP 配置

We provide a custom Google.Apis.Http.IHttpClientFactory backed by SocketsHttpHandler:

我们通过自定义 Google.Apis.Http.IHttpClientFactory 使用以下 SocketsHttpHandler 配置:

new SocketsHttpHandler
{
    AllowAutoRedirect = false,
    AutomaticDecompression = decompression,
    EnableMultipleHttp2Connections = true,
    PooledConnectionLifetime = TimeSpan.FromMinutes(5),
};

decompression is either GZip | Deflate or None, depending on the SDK factory options.

decompression 根据 SDK 工厂选项设置为 GZip | Deflate 或 None。

A custom DelegatingHandler sets the version policy and logs exceptions before rethrowing them:

自定义 DelegatingHandler 设置版本策略,并记录异常后重新抛出:

try
{
    request.VersionPolicy = HttpVersionPolicy.RequestVersionOrHigher;
    return await base.SendAsync(request, cancellationToken);
}
catch (Exception ex)
{
    logger.LogWarning(ex, "Firebase HTTP request failed");
    throw;
}

Questions / 希望确认

Could our custom HTTP configuration contribute to these exceptions? Are there recommended configuration changes or fixes?
自定义 HTTP 配置是否可能导致这些异常?是否有推荐的配置调整或修复方案?

What diagnostics should we collect to identify the cancellation source and correlate retry attempts with the final send result?
建议收集哪些诊断信息,以定位取消来源并关联重试与最终发送结果?

If the SDK retries successfully and the final send succeeds, can these inner-handler exceptions be treated as transient and logged at a lower severity? Under what conditions should they require further investigation?
如果 SDK 重试后最终发送成功,是否可以将这些内部 Handler 异常视为瞬时故障并降低日志级别?哪些情况下仍需要进一步排查?

Dominant language
C#
Stars
407
Forks
149
Avg merge
2d 10h
Merged PRs (30d)
14

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from firebase/firebase-admin-dotnet

All issues in firebase/firebase-admin-dotnet

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.