[FR] Expose retry-attempt observability (hook/event) — retried 503s and per-attempt latency are invisible to callers
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 28/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- nodejs, typescript
- Domain
- api, observability
Research direction
Start with the retry behavior referenced in the api-request code comment and review the messaging or app options discussed in the request, along with related issue #1615. Done means callers can observe retry attempts, status or error details, elapsed time, and whether another retry will occur, without patching HTTP/2 internals.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem?
The HTTP clients retry transient failures — per the code comment in api-request, "Retries up to 4 times on connection reset and timeout errors as well as 503 errors" — which is great, but the retry lifecycle is completely invisible to the caller. You observe only the final result and total elapsed time.
In production this matters a lot for FCM:
- A
sendEach()call that "took 15s" is indistinguishable from one that made three 5s attempts. Capacity planning, SLO attribution, and incident analysis need to tell these apart. - 503s that the SDK retries away never appear anywhere. During an FCM backend hiccup, our error rate looked flat while the wire was full of retried 503s — we only learned this after instrumenting below the SDK.
What we did as a workaround (and what it revealed)
We attached listeners at the HTTP/2 layer to record, per attempt, the :status header the SDK already receives, plus per-attempt request→response duration. Two things became visible immediately:
- Swallowed 503s during backend episodes (invisible at the SDK surface).
- Batches with ~15s total latency decomposed cleanly into 3 × ~5s attempts — i.e. the tail was retry behavior, not slow single requests.
We validated the accounting at scale: in a ~650k-message load run at ~2,000 rps, wire-level attempt counts reconciled exactly with SDK-level result counts (596,663 = 596,663).
Monkey-patching works but is version-fragile and clearly not the intended way.
Describe the solution you'd like
Any of these would solve it (in rough order of preference):
- An
onRetryAttempt(info)callback / EventEmitter on the messaging or app options, with{ attempt, statusCode?, errorCode?, elapsedMs, willRetry }. - Attempt metadata attached to the final response/error (e.g.
attempts: [{status, elapsedMs}, ...]). - At minimum, a debug logging hook for retry decisions.
Describe alternatives you've considered
- Runtime-patching the HTTP/2 request path to observe response headers (works, but couples us to SDK internals).
enableLegacyHttpTransport()+ external proxy metrics (gives up HTTP/2).
Additional context
Related: #1615 (custom RetryConfig) — configuration and observability of the same mechanism. Verified against 12.7.0 and 14.1, on Node 16 and 24.
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 419
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 16
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from firebase/firebase-admin-node
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
firebase/firebase-admin-node#3234 ·
-
firebase/firebase-admin-node#3221 · 3 comments · 1 assignee ·
-
api: messaging
Difficulty 3/5 1-2 days Newbie friendliness 70/100
firebase/firebase-admin-node#3215 ·
-
api: firestore type: feature request
firebase/firebase-admin-node#3183 · 1 comment · 1 assignee ·
-
api: appcheck
firebase/firebase-admin-node#3181 · 2 comments · 5 reactions · 1 assignee ·
All issues in firebase/firebase-admin-node
Similar issues
-
VerificationGate: ATTRIBUTION quote guard never matches a normal quotation (\b around the quote) Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
danielmiessler/LifeOS#2234 ·
-
T: Bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
Mend: dependency security vulnerability untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 70/100