Disaggregated TiFlash: FetchDisaggPages UNAVAILABLE aborts Q1 during rolling upgrade with two available replicas
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 35/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- cpp, grpc
- Ambito
- databases, distributed-systems
Direzione di ricerca
Start with SegmentReadTask.cpp around line 412, where FetchDisaggPages returns gRPC UNAVAILABLE, then read the source analysis in the issue for the retry and error-classification paths at the exact running commit. Identify the affected recovery coverage and add or update the relevant tests; done means transient page-fetch transport failures follow the intended retry or recovery semantics without aborting the query.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Bug Report
Summary
A disaggregated TiFlash rolling upgrade caused a CH Q1 query to fail with MySQL Error 1105 / FetchDisaggPages gRPC UNAVAILABLE(14), despite two Write nodes, two Compute nodes, and test.order_line having two available TiFlash replicas before the upgrade.
The first failing page-fetch happened after the first Write node reported ready and before the second Write node received its termination signal. The same query successfully established snapshots on both Write stores and fetched other segments from the first Write. This is not evidence of both Write nodes being shut down simultaneously.
Source inspection identifies an error-classification/retry coverage gap matching the observed failure. The underlying reason for the initial connection/handshake failures remains unconfirmed; please do not treat a particular connection-pool, DNS, TLS, or network cause as established.
1. Reproduction / observed execution
Observed once in the following real environment; no additional reproduction or upgrade replay was performed.
- AWS us-west-2, TiDB Cloud Premium, disaggregated LEGACY TiFlash topology.
- 2 Write nodes + 2 Compute nodes; Write stores 1004 and 1006.
- Source/downstream logical groups:
cluster1andcluster2. - Environment-only TCMS execution: 8214336.
- The TCMS step itself is only
testbed_readyand succeeded. The failure reported here belongs to a separately launched full local-binary case on that retained environment, not a failed TCMStestbed_readystep. - Case:
premium_upgrade_all_with_ticdc_ch, focus#premium-upgrade-all-with-ticdc-ch#,testcase/upgrade/tidbx_upgrade_all. - Endless source: PR #4179, commit
4838f24eefd74487ff80fa1ca1e665ec4ca939cd. - Test binary SHA-256:
dce51ede36249098a65e86b1a187abc1b7066f548f720163d1bec6cd98ca456b. - Run ID:
smoke-8214336-pr4179-4838f24e; 2026-10-29 15:13:35–15:46:12 UTC+08.
Steps followed by that single complete-case attempt:
- Restore CH-150 into source and downstream.
- Set source
test.order_lineto TiFlash REPLICA 2 and wait for availability. Verify Q1 actually executes through TiFlash. - Verify initial source/downstream CRC consistency.
- Create a case-owned MySQL TiCDC changefeed in
FORCE_SYNCmode; align TiCDC to source imagev26.3.4-nextgen. - Run source CH TP concurrency 2 + AP Q1 concurrency 1, with the case's consistency checking and TiCDC checkpoint monitoring.
- Complete the original 10-minute pre-upgrade observation.
- Upgrade TiCDC to
release-nextgen-202609; then execute the TiFlash Compute/Write upgrade using the same target tag and logical versionv26.9.99. - The AP query fails during the TiFlash rolling window. After the SDK operation completes, the runtime gate reports the workload failure and stops the case. PD and later component upgrades are not reached.
Original observation durations were not shortened. The remaining intended flow was later component upgrades, 10-minute post-observation, checkpoint catch-up, final CRC/MPP checks and cleanup. The complete case did not pass.
Replica evidence:
- TiDB server DDL logs at 15:20:35.250 / 15:20:35.345 UTC+08 record:
ALTER TABLE test.order_line SET TIFLASH REPLICA 2. - Pre-upgrade observation at 15:22:12, subsequent rolling-window observations, and the post-failure observation at 15:46:36 UTC+08 report:
REPLICA_COUNT=2, AVAILABLE=1, PROGRESS=1. - These are table-level readiness observations, not proof that every physical replica or every RPC connection was healthy at every instant.
2. Expected behavior
The rolling upgrade should preserve successful AP reads under the supported two-Write/two-Compute configuration, or recover a transient page-fetch transport failure within the product's retry/recovery semantics. A healthy replica count should not be mistaken for proof that this transport failure is impossible.
3. Actual result and first-error timeline
All times below are 2026-10-29 UTC+08 (Asia/Shanghai). Log bodies use UTC, eight hours earlier.
| Time UTC+08 | Evidence |
|---|---|
| 15:43:24.296 | TiFlash SDK upgrade step starts. |
| 15:43:43.509 | Write 0glner / store 1004 receives a termination signal. |
| 15:43:46.188 | Write 1004 closes listening sockets. |
| 15:44:25.631 | Compute 6zypjl reports ready; Write 1004 is still starting. |
| 15:44:56.307 | Write 1004 reports Ready for connections. |
| 15:45:04.780 etc. | Compute-to-Write1004 gRPC logs still show connection timeout / FD Shutdown and internal reconnect attempts. |
| 15:45:06.911 | Previous AP Q1 completes successfully. |
| 15:45:07.577 | Failing Q1 starts; query TSO 470092325199806622. |
| 15:45:07.592 / .597 | Compute receives successful EstablishDisaggTask responses from stores 1004 and 1006. |
| 15:45:07.663–.684 | Write1004 serves other segments (14, 5, 23) for this same query. |
| 15:45:07.706 | First page-fetch ERROR: Compute 6zypjl, SegmentReadTask.cpp:412, segment s1004_ks1_t80_1_5_3658451676704_4939483, gRPC status 14. |
| 15:45:07.708–.731 | Compute ExchangeReceiver/Pipeline cancels the MPP query. |
| 15:45:07.733 / .737 | TiDB records the MPP error; client Q1 receives Error 1105. |
| 15:45:08.399 | AP workload exits; cancellation errors in other workload tasks follow. |
| 15:45:10.079 | Second Write 5rx4pc / store 1006 receives termination signal, after the first AP failure. |
| 15:45:11.202 | Compute logs a further handshake timeout to Write1004. |
| 15:45:53.038 | Second Write reports ready. |
| 15:46:12.369 | SDK marks TiFlash operation completed. The following runtime gate fails because the workload already exited; process ends with exit code 1. |
Condensed first-error evidence (server log timestamp is UTC):
[2026/10/29 07:45:07.706 +00:00] [ERROR] [SegmentReadTask.cpp:412]
Code: 49, DB::Exception: Failed to fetch all pages for
s1004_ks1_t80_1_5_3658451676704_4939483,
wn_address=<Write1004>:3930 status=14
message=failed to connect to all addresses
The subsequent client error is Error 1105 (HY000): other error for mpp stream ... Receiver state: ERROR ... Failed to fetch all pages ....
4. Actual versions
Upgrade version boundary (Compute and Write)
Both TiFlash Compute and TiFlash Write were upgraded from image tag v26.3.10-nextgen (logical v26.3.99) to image tag release-nextgen-202609 (requested logical v26.9.99).
| Version identity | Before upgrade | After restart during the failing upgrade |
|---|---|---|
| Image tag (Compute and Write) | v26.3.10-nextgen |
release-nextgen-202609 |
| Logical version | v26.3.99 |
requested v26.9.99 |
| Actual TiFlash executable self-reported version | 26.3.10 |
v27.0.0-pre |
| Actual TiFlash git commit | d89f347e5331ad47abe277764f67ff21f6a6976c |
4c437ce7f7b0cc8927266e04f1b3b194a15ab4ef |
Thus the tested image transition is v26.3.10-nextgen → release-nextgen-202609, and the observed executable transition is 26.3.10 / d89f347e → v27.0.0-pre / 4c437ce7. The target's self-reported version does not equal the requested logical version; both identities are retained rather than inferred from the image tag. This report does not claim every component completed the broader source-to-target upgrade: the case stopped after the TiFlash step's workload gate failed.
Please distinguish requested tags/logical versions from the executable's self-reported identity.
| Item | Observed value |
|---|---|
| Source TiFlash Compute and Write runtime | 26.3.10, git d89f347e5331ad47abe277764f67ff21f6a6976c |
| Requested target image tag, both Compute and Write | release-nextgen-202609 |
| Requested target logical version | v26.9.99 |
| Restarted Compute and Write self-reported release | v27.0.0-pre |
| Actual restarted TiFlash git | 4c437ce7f7b0cc8927266e04f1b3b194a15ab4ef |
| Reported build time | 2026-10-25 08:10:36 UTC |
| client-c gitlink at that TiFlash commit | 55aede24990ec5f92a1d74768d65a11eacc5f7d5 |
| gRPC gitlink | 591d56e1300b6d11948e1b821efac785a295989c |
The pre-release version string is an observation from the deployed executable; it does not by itself prove that an unrelated master image was selected.
5. Clinic logs
Full Clinic entry and complete run window
- Clinic cluster entry — search by shared pool ID
- Full-run logs — all components, no error/text filter
- Full-run TiFlash logs — Compute + Write, all log streams, no error/text filter
Full run window: 2026-10-29 15:13:00–15:48:00 UTC+08 (07:13:00–07:48:00 UTC). This covers preparation, the pre-upgrade workload, both upgrade steps reached, the first error and terminal failure/cleanup attempt. The process ran 15:13:35–15:46:12 UTC+08. The links below retain narrower incident windows for convenience; the full-run links above do not restrict logs to selected error keywords. Clinic login/internal access is required.
These links contain filters and absolute times, no credentials. They require internal Clinic access.
- Compute first error and handshake/connection errors — 15:44:50–15:45:20 UTC+08; focus on 15:45:07.706 (log body 07:45:07.706 UTC).
- Write rolling restart lifecycle — 15:43:20–15:46:20 UTC+08.
Routing: shared pool 643782309392196954, namespace tidb643782309392196954. C++ logs are in ti-serverlog / ti-errorlog, rather than only the tiflash proxy stdout stream.
6. Source analysis: confirmed recovery gap vs. open hypotheses
Confirmed from the exact running commit and matching failure location:
fetchPages()has up to three attempts, but only retriespingcap::Exception.finishPagesPacketStream()handles non-OK stream completion. StatusUNAVAILABLE(14)goes to the default branch and throwsDB::Exception(LOGICAL_ERROR).DB::Exceptionandpingcap::Exceptionare distinct exception types. Therefore this failure reachesfetchPages()'scatch (...), which logs at line 412 and immediately rethrows, bypassing its retry loop. This matches the observed server ERROR exactly.- client-c's connection array selects connections round-robin; each address has five clients. This selection does not check channel readiness.
dropConnIfNeeded()exists, but theFetchDisaggPagesstream-finish error path does not call it. The background cleanup checks whether addresses remain in PD, rather than whether individual channels are healthy.- gRPC does perform internal reconnect attempts; the logs show them. The finding is not that there is no reconnect at all, but that this query still fails without entering the page-fetch retry branch.
Not yet established:
- The original cause of connection/handshake timeout: channel/pool recovery, DNS/IP lifecycle, TLS/handshaking, or underlying network.
- A permanent stale-channel state or a specific failing channel mapped to every successful/failed RPC.
- Whether a bounded retry alone is sufficient. Retry correctness must consider partial packets, memtable data, snapshot release and ownership; switching a pinned segment snapshot to another replica is not simply replacing its address.
- Data loss, permanent unavailability, or behavior on a repeated query. No extra workload replay was performed.
7. Related issues and fixes
- #10513: similar WN rolling-restart symptom, but its documented cause was a long query exceeding the 600-second shutdown grace period; it was closed after scan optimizations #10522 / #10555. This report's short Q1 failure after the first Write was ready needs separate triage.
- #10994: related disaggregated fetch/establish recovery failures under WN network partition; different injected fault and timeout error.
- #10425 / #10430: existing WN graceful-shutdown work; not evidence that this post-recovery connection error is covered.
- tikv/client-c#233: connection-cleanup changes are already an ancestor of this run's client-c dependency; this is not simply evidence that #233 was absent.
Current default master inspected at f3b7883955fc7127476c913ead6ebbdd835c9caf has the same SegmentReadTask.cpp blob (188f88acc996854622e32f0f31f41ed405d6077a) as the running commit. No matching fix for the exception/retry gap was found in the scoped GitHub review. This is source-level verification, not a fixed-build regression test.
Current state
Subsequent upgrades are stopped and the existing environment was retained for diagnostics. No automatic rerun, downgrade, new environment or product-code change was performed. The full-case attempt failed; it must not be reported as a complete smoke pass.
- Lingua principale
- C++
- Stelle
- 1k
- Fork
- 428
- Merge medio
- 2g 7h
- PR unite (30g)
- 100
Preparare l'ambiente
Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Nessuna guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di pingcap/tiflash
-
component/compute type/bug
Difficoltà 2/5 1-2 giorni Idoneità per principianti 82/100
I maintainer di solito rispondono entro 1 giorno
-
Out-of-bounds read in MyDateTime formatting from unvalidated packed valuesForse già presa @ChangRui-Ryan l’ha presa oggi. Apertaaffects-25.10 affects-26.3 affects-26.9 affects-7.5 affects-8.1 affects-8.5 severity/major type/bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
I maintainer di solito rispondono entro 1 giorno
-
component/storage severity/moderate type/bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
pingcap/tiflash#11335 · 5 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Missing validation in JSON constructors, LEFT expressions, and CH-block decodingForse già presa @windtalker l’ha presa oggi. Apertatype/bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 25/100
I maintainer di solito rispondono entro 1 giorno
-
[disagg] Query fails with Code: 11000 NoSuchKey when S3 GC (ScanThenDelete) removes a DMFile that a compute node still readsForse già presa @JinheLin l’ha presa 6 giorni fa. Apertamay-affects-25.10 may-affects-26.3 may-affects-26.9 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/moderate type/bug
pingcap/tiflash#11291 · 1 commento · 1 assegnatario ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di pingcap/tiflash
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno
-
useless statementAperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 86/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
grizzlypeak3d/DJV#962 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 1 giorno