Disaggregated TiFlash: FetchDisaggPages UNAVAILABLE aborts Q1 during rolling upgrade with two available replicas
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 35/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- cpp, grpc
- Área
- databases, distributed-systems
Línea de trabajo
Start with SegmentReadTask.cpp around line 412, where FetchDisaggPages returns gRPC UNAVAILABLE, then read the source analysis in the issue for the retry and error-classification paths at the exact running commit. Identify the affected recovery coverage and add or update the relevant tests; done means transient page-fetch transport failures follow the intended retry or recovery semantics without aborting the query.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Bug Report
Summary
A disaggregated TiFlash rolling upgrade caused a CH Q1 query to fail with MySQL Error 1105 / FetchDisaggPages gRPC UNAVAILABLE(14), despite two Write nodes, two Compute nodes, and test.order_line having two available TiFlash replicas before the upgrade.
The first failing page-fetch happened after the first Write node reported ready and before the second Write node received its termination signal. The same query successfully established snapshots on both Write stores and fetched other segments from the first Write. This is not evidence of both Write nodes being shut down simultaneously.
Source inspection identifies an error-classification/retry coverage gap matching the observed failure. The underlying reason for the initial connection/handshake failures remains unconfirmed; please do not treat a particular connection-pool, DNS, TLS, or network cause as established.
1. Reproduction / observed execution
Observed once in the following real environment; no additional reproduction or upgrade replay was performed.
- AWS us-west-2, TiDB Cloud Premium, disaggregated LEGACY TiFlash topology.
- 2 Write nodes + 2 Compute nodes; Write stores 1004 and 1006.
- Source/downstream logical groups:
cluster1andcluster2. - Environment-only TCMS execution: 8214336.
- The TCMS step itself is only
testbed_readyand succeeded. The failure reported here belongs to a separately launched full local-binary case on that retained environment, not a failed TCMStestbed_readystep. - Case:
premium_upgrade_all_with_ticdc_ch, focus#premium-upgrade-all-with-ticdc-ch#,testcase/upgrade/tidbx_upgrade_all. - Endless source: PR #4179, commit
4838f24eefd74487ff80fa1ca1e665ec4ca939cd. - Test binary SHA-256:
dce51ede36249098a65e86b1a187abc1b7066f548f720163d1bec6cd98ca456b. - Run ID:
smoke-8214336-pr4179-4838f24e; 2026-10-29 15:13:35–15:46:12 UTC+08.
Steps followed by that single complete-case attempt:
- Restore CH-150 into source and downstream.
- Set source
test.order_lineto TiFlash REPLICA 2 and wait for availability. Verify Q1 actually executes through TiFlash. - Verify initial source/downstream CRC consistency.
- Create a case-owned MySQL TiCDC changefeed in
FORCE_SYNCmode; align TiCDC to source imagev26.3.4-nextgen. - Run source CH TP concurrency 2 + AP Q1 concurrency 1, with the case's consistency checking and TiCDC checkpoint monitoring.
- Complete the original 10-minute pre-upgrade observation.
- Upgrade TiCDC to
release-nextgen-202609; then execute the TiFlash Compute/Write upgrade using the same target tag and logical versionv26.9.99. - The AP query fails during the TiFlash rolling window. After the SDK operation completes, the runtime gate reports the workload failure and stops the case. PD and later component upgrades are not reached.
Original observation durations were not shortened. The remaining intended flow was later component upgrades, 10-minute post-observation, checkpoint catch-up, final CRC/MPP checks and cleanup. The complete case did not pass.
Replica evidence:
- TiDB server DDL logs at 15:20:35.250 / 15:20:35.345 UTC+08 record:
ALTER TABLE test.order_line SET TIFLASH REPLICA 2. - Pre-upgrade observation at 15:22:12, subsequent rolling-window observations, and the post-failure observation at 15:46:36 UTC+08 report:
REPLICA_COUNT=2, AVAILABLE=1, PROGRESS=1. - These are table-level readiness observations, not proof that every physical replica or every RPC connection was healthy at every instant.
2. Expected behavior
The rolling upgrade should preserve successful AP reads under the supported two-Write/two-Compute configuration, or recover a transient page-fetch transport failure within the product's retry/recovery semantics. A healthy replica count should not be mistaken for proof that this transport failure is impossible.
3. Actual result and first-error timeline
All times below are 2026-10-29 UTC+08 (Asia/Shanghai). Log bodies use UTC, eight hours earlier.
| Time UTC+08 | Evidence |
|---|---|
| 15:43:24.296 | TiFlash SDK upgrade step starts. |
| 15:43:43.509 | Write 0glner / store 1004 receives a termination signal. |
| 15:43:46.188 | Write 1004 closes listening sockets. |
| 15:44:25.631 | Compute 6zypjl reports ready; Write 1004 is still starting. |
| 15:44:56.307 | Write 1004 reports Ready for connections. |
| 15:45:04.780 etc. | Compute-to-Write1004 gRPC logs still show connection timeout / FD Shutdown and internal reconnect attempts. |
| 15:45:06.911 | Previous AP Q1 completes successfully. |
| 15:45:07.577 | Failing Q1 starts; query TSO 470092325199806622. |
| 15:45:07.592 / .597 | Compute receives successful EstablishDisaggTask responses from stores 1004 and 1006. |
| 15:45:07.663–.684 | Write1004 serves other segments (14, 5, 23) for this same query. |
| 15:45:07.706 | First page-fetch ERROR: Compute 6zypjl, SegmentReadTask.cpp:412, segment s1004_ks1_t80_1_5_3658451676704_4939483, gRPC status 14. |
| 15:45:07.708–.731 | Compute ExchangeReceiver/Pipeline cancels the MPP query. |
| 15:45:07.733 / .737 | TiDB records the MPP error; client Q1 receives Error 1105. |
| 15:45:08.399 | AP workload exits; cancellation errors in other workload tasks follow. |
| 15:45:10.079 | Second Write 5rx4pc / store 1006 receives termination signal, after the first AP failure. |
| 15:45:11.202 | Compute logs a further handshake timeout to Write1004. |
| 15:45:53.038 | Second Write reports ready. |
| 15:46:12.369 | SDK marks TiFlash operation completed. The following runtime gate fails because the workload already exited; process ends with exit code 1. |
Condensed first-error evidence (server log timestamp is UTC):
[2026/10/29 07:45:07.706 +00:00] [ERROR] [SegmentReadTask.cpp:412]
Code: 49, DB::Exception: Failed to fetch all pages for
s1004_ks1_t80_1_5_3658451676704_4939483,
wn_address=<Write1004>:3930 status=14
message=failed to connect to all addresses
The subsequent client error is Error 1105 (HY000): other error for mpp stream ... Receiver state: ERROR ... Failed to fetch all pages ....
4. Actual versions
Upgrade version boundary (Compute and Write)
Both TiFlash Compute and TiFlash Write were upgraded from image tag v26.3.10-nextgen (logical v26.3.99) to image tag release-nextgen-202609 (requested logical v26.9.99).
| Version identity | Before upgrade | After restart during the failing upgrade |
|---|---|---|
| Image tag (Compute and Write) | v26.3.10-nextgen |
release-nextgen-202609 |
| Logical version | v26.3.99 |
requested v26.9.99 |
| Actual TiFlash executable self-reported version | 26.3.10 |
v27.0.0-pre |
| Actual TiFlash git commit | d89f347e5331ad47abe277764f67ff21f6a6976c |
4c437ce7f7b0cc8927266e04f1b3b194a15ab4ef |
Thus the tested image transition is v26.3.10-nextgen → release-nextgen-202609, and the observed executable transition is 26.3.10 / d89f347e → v27.0.0-pre / 4c437ce7. The target's self-reported version does not equal the requested logical version; both identities are retained rather than inferred from the image tag. This report does not claim every component completed the broader source-to-target upgrade: the case stopped after the TiFlash step's workload gate failed.
Please distinguish requested tags/logical versions from the executable's self-reported identity.
| Item | Observed value |
|---|---|
| Source TiFlash Compute and Write runtime | 26.3.10, git d89f347e5331ad47abe277764f67ff21f6a6976c |
| Requested target image tag, both Compute and Write | release-nextgen-202609 |
| Requested target logical version | v26.9.99 |
| Restarted Compute and Write self-reported release | v27.0.0-pre |
| Actual restarted TiFlash git | 4c437ce7f7b0cc8927266e04f1b3b194a15ab4ef |
| Reported build time | 2026-10-25 08:10:36 UTC |
| client-c gitlink at that TiFlash commit | 55aede24990ec5f92a1d74768d65a11eacc5f7d5 |
| gRPC gitlink | 591d56e1300b6d11948e1b821efac785a295989c |
The pre-release version string is an observation from the deployed executable; it does not by itself prove that an unrelated master image was selected.
5. Clinic logs
Full Clinic entry and complete run window
- Clinic cluster entry — search by shared pool ID
- Full-run logs — all components, no error/text filter
- Full-run TiFlash logs — Compute + Write, all log streams, no error/text filter
Full run window: 2026-10-29 15:13:00–15:48:00 UTC+08 (07:13:00–07:48:00 UTC). This covers preparation, the pre-upgrade workload, both upgrade steps reached, the first error and terminal failure/cleanup attempt. The process ran 15:13:35–15:46:12 UTC+08. The links below retain narrower incident windows for convenience; the full-run links above do not restrict logs to selected error keywords. Clinic login/internal access is required.
These links contain filters and absolute times, no credentials. They require internal Clinic access.
- Compute first error and handshake/connection errors — 15:44:50–15:45:20 UTC+08; focus on 15:45:07.706 (log body 07:45:07.706 UTC).
- Write rolling restart lifecycle — 15:43:20–15:46:20 UTC+08.
Routing: shared pool 643782309392196954, namespace tidb643782309392196954. C++ logs are in ti-serverlog / ti-errorlog, rather than only the tiflash proxy stdout stream.
6. Source analysis: confirmed recovery gap vs. open hypotheses
Confirmed from the exact running commit and matching failure location:
fetchPages()has up to three attempts, but only retriespingcap::Exception.finishPagesPacketStream()handles non-OK stream completion. StatusUNAVAILABLE(14)goes to the default branch and throwsDB::Exception(LOGICAL_ERROR).DB::Exceptionandpingcap::Exceptionare distinct exception types. Therefore this failure reachesfetchPages()'scatch (...), which logs at line 412 and immediately rethrows, bypassing its retry loop. This matches the observed server ERROR exactly.- client-c's connection array selects connections round-robin; each address has five clients. This selection does not check channel readiness.
dropConnIfNeeded()exists, but theFetchDisaggPagesstream-finish error path does not call it. The background cleanup checks whether addresses remain in PD, rather than whether individual channels are healthy.- gRPC does perform internal reconnect attempts; the logs show them. The finding is not that there is no reconnect at all, but that this query still fails without entering the page-fetch retry branch.
Not yet established:
- The original cause of connection/handshake timeout: channel/pool recovery, DNS/IP lifecycle, TLS/handshaking, or underlying network.
- A permanent stale-channel state or a specific failing channel mapped to every successful/failed RPC.
- Whether a bounded retry alone is sufficient. Retry correctness must consider partial packets, memtable data, snapshot release and ownership; switching a pinned segment snapshot to another replica is not simply replacing its address.
- Data loss, permanent unavailability, or behavior on a repeated query. No extra workload replay was performed.
7. Related issues and fixes
- #10513: similar WN rolling-restart symptom, but its documented cause was a long query exceeding the 600-second shutdown grace period; it was closed after scan optimizations #10522 / #10555. This report's short Q1 failure after the first Write was ready needs separate triage.
- #10994: related disaggregated fetch/establish recovery failures under WN network partition; different injected fault and timeout error.
- #10425 / #10430: existing WN graceful-shutdown work; not evidence that this post-recovery connection error is covered.
- tikv/client-c#233: connection-cleanup changes are already an ancestor of this run's client-c dependency; this is not simply evidence that #233 was absent.
Current default master inspected at f3b7883955fc7127476c913ead6ebbdd835c9caf has the same SegmentReadTask.cpp blob (188f88acc996854622e32f0f31f41ed405d6077a) as the running commit. No matching fix for the exception/retry gap was found in the scoped GitHub review. This is source-level verification, not a fixed-build regression test.
Current state
Subsequent upgrades are stopped and the existing environment was retained for diagnostics. No automatic rerun, downgrade, new environment or product-code change was performed. The full-case attempt failed; it must not be reported as a complete smoke pass.
- Lenguaje dominante
- C++
- Estrellas
- 1k
- Forks
- 428
- Merge medio
- 2 d 7 h
- PR fusionados (30 d)
- 100
Preparar el entorno
Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.
- Sin Dockerfile ni archivo de Docker Compose
- Tiene una plantilla de pull request
- Sin guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de pingcap/tiflash
-
component/compute type/bug
Dificultad 2/5 1-2 días Aptitud para principiantes 82/100
Los mantenedores suelen responder en 1 día
-
Out-of-bounds read in MyDateTime formatting from unvalidated packed valuesPosiblemente ocupada @ChangRui-Ryan la tomó hoy. Abiertoaffects-25.10 affects-26.3 affects-26.9 affects-7.5 affects-8.1 affects-8.5 severity/major type/bug
Dificultad 4/5 3-5 días Aptitud para principiantes 30/100
Los mantenedores suelen responder en 1 día
-
component/storage severity/moderate type/bug
Dificultad 4/5 3-5 días Aptitud para principiantes 48/100
pingcap/tiflash#11335 · 5 comentarios ·
Los mantenedores suelen responder en 1 día
-
Missing validation in JSON constructors, LEFT expressions, and CH-block decodingPosiblemente ocupada @windtalker la tomó hoy. Abiertotype/bug
Dificultad 4/5 3-5 días Aptitud para principiantes 25/100
Los mantenedores suelen responder en 1 día
-
[disagg] Query fails with Code: 11000 NoSuchKey when S3 GC (ScanThenDelete) removes a DMFile that a compute node still readsPosiblemente ocupada @JinheLin la tomó hace 6 días. Abiertomay-affects-25.10 may-affects-26.3 may-affects-26.9 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/moderate type/bug
pingcap/tiflash#11291 · 1 comentario · 1 asignado ·
Los mantenedores suelen responder en 1 día
Todos los issues de pingcap/tiflash
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
useless statementAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 86/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
grizzlypeak3d/DJV#962 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 1 día