Disaggregated TiFlash: FetchDisaggPages UNAVAILABLE aborts Q1 during rolling upgrade with two available replicas

Aperta
#11,169 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp, grpc

Direzione di ricerca

Start with SegmentReadTask.cpp around line 412, where FetchDisaggPages returns gRPC UNAVAILABLE, then read the source analysis in the issue for the retry and error-classification paths at the exact running commit. Identify the affected recovery coverage and add or update the relevant tests; done means transient page-fetch transport failures follow the intended retry or recovery semantics without aborting the query.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

contribution type/bug

Bug Report

Summary

A disaggregated TiFlash rolling upgrade caused a CH Q1 query to fail with MySQL Error 1105 / FetchDisaggPages gRPC UNAVAILABLE(14), despite two Write nodes, two Compute nodes, and test.order_line having two available TiFlash replicas before the upgrade.

The first failing page-fetch happened after the first Write node reported ready and before the second Write node received its termination signal. The same query successfully established snapshots on both Write stores and fetched other segments from the first Write. This is not evidence of both Write nodes being shut down simultaneously.

Source inspection identifies an error-classification/retry coverage gap matching the observed failure. The underlying reason for the initial connection/handshake failures remains unconfirmed; please do not treat a particular connection-pool, DNS, TLS, or network cause as established.

1. Reproduction / observed execution

Observed once in the following real environment; no additional reproduction or upgrade replay was performed.

  • AWS us-west-2, TiDB Cloud Premium, disaggregated LEGACY TiFlash topology.
  • 2 Write nodes + 2 Compute nodes; Write stores 1004 and 1006.
  • Source/downstream logical groups: cluster1 and cluster2.
  • Environment-only TCMS execution: 8214336.
  • The TCMS step itself is only testbed_ready and succeeded. The failure reported here belongs to a separately launched full local-binary case on that retained environment, not a failed TCMS testbed_ready step.
  • Case: premium_upgrade_all_with_ticdc_ch, focus #premium-upgrade-all-with-ticdc-ch#, testcase/upgrade/tidbx_upgrade_all.
  • Endless source: PR #4179, commit 4838f24eefd74487ff80fa1ca1e665ec4ca939cd.
  • Test binary SHA-256: dce51ede36249098a65e86b1a187abc1b7066f548f720163d1bec6cd98ca456b.
  • Run ID: smoke-8214336-pr4179-4838f24e; 2026-10-29 15:13:35–15:46:12 UTC+08.

Steps followed by that single complete-case attempt:

  1. Restore CH-150 into source and downstream.
  2. Set source test.order_line to TiFlash REPLICA 2 and wait for availability. Verify Q1 actually executes through TiFlash.
  3. Verify initial source/downstream CRC consistency.
  4. Create a case-owned MySQL TiCDC changefeed in FORCE_SYNC mode; align TiCDC to source image v26.3.4-nextgen.
  5. Run source CH TP concurrency 2 + AP Q1 concurrency 1, with the case's consistency checking and TiCDC checkpoint monitoring.
  6. Complete the original 10-minute pre-upgrade observation.
  7. Upgrade TiCDC to release-nextgen-202609; then execute the TiFlash Compute/Write upgrade using the same target tag and logical version v26.9.99.
  8. The AP query fails during the TiFlash rolling window. After the SDK operation completes, the runtime gate reports the workload failure and stops the case. PD and later component upgrades are not reached.

Original observation durations were not shortened. The remaining intended flow was later component upgrades, 10-minute post-observation, checkpoint catch-up, final CRC/MPP checks and cleanup. The complete case did not pass.

Replica evidence:

  • TiDB server DDL logs at 15:20:35.250 / 15:20:35.345 UTC+08 record:
    ALTER TABLE test.order_line SET TIFLASH REPLICA 2.
  • Pre-upgrade observation at 15:22:12, subsequent rolling-window observations, and the post-failure observation at 15:46:36 UTC+08 report:
    REPLICA_COUNT=2, AVAILABLE=1, PROGRESS=1.
  • These are table-level readiness observations, not proof that every physical replica or every RPC connection was healthy at every instant.
2. Expected behavior

The rolling upgrade should preserve successful AP reads under the supported two-Write/two-Compute configuration, or recover a transient page-fetch transport failure within the product's retry/recovery semantics. A healthy replica count should not be mistaken for proof that this transport failure is impossible.

3. Actual result and first-error timeline

All times below are 2026-10-29 UTC+08 (Asia/Shanghai). Log bodies use UTC, eight hours earlier.

Time UTC+08 Evidence
15:43:24.296 TiFlash SDK upgrade step starts.
15:43:43.509 Write 0glner / store 1004 receives a termination signal.
15:43:46.188 Write 1004 closes listening sockets.
15:44:25.631 Compute 6zypjl reports ready; Write 1004 is still starting.
15:44:56.307 Write 1004 reports Ready for connections.
15:45:04.780 etc. Compute-to-Write1004 gRPC logs still show connection timeout / FD Shutdown and internal reconnect attempts.
15:45:06.911 Previous AP Q1 completes successfully.
15:45:07.577 Failing Q1 starts; query TSO 470092325199806622.
15:45:07.592 / .597 Compute receives successful EstablishDisaggTask responses from stores 1004 and 1006.
15:45:07.663–.684 Write1004 serves other segments (14, 5, 23) for this same query.
15:45:07.706 First page-fetch ERROR: Compute 6zypjl, SegmentReadTask.cpp:412, segment s1004_ks1_t80_1_5_3658451676704_4939483, gRPC status 14.
15:45:07.708–.731 Compute ExchangeReceiver/Pipeline cancels the MPP query.
15:45:07.733 / .737 TiDB records the MPP error; client Q1 receives Error 1105.
15:45:08.399 AP workload exits; cancellation errors in other workload tasks follow.
15:45:10.079 Second Write 5rx4pc / store 1006 receives termination signal, after the first AP failure.
15:45:11.202 Compute logs a further handshake timeout to Write1004.
15:45:53.038 Second Write reports ready.
15:46:12.369 SDK marks TiFlash operation completed. The following runtime gate fails because the workload already exited; process ends with exit code 1.

Condensed first-error evidence (server log timestamp is UTC):

[2026/10/29 07:45:07.706 +00:00] [ERROR] [SegmentReadTask.cpp:412]
Code: 49, DB::Exception: Failed to fetch all pages for
s1004_ks1_t80_1_5_3658451676704_4939483,
wn_address=<Write1004>:3930 status=14
message=failed to connect to all addresses

The subsequent client error is Error 1105 (HY000): other error for mpp stream ... Receiver state: ERROR ... Failed to fetch all pages ....

4. Actual versions
Upgrade version boundary (Compute and Write)

Both TiFlash Compute and TiFlash Write were upgraded from image tag v26.3.10-nextgen (logical v26.3.99) to image tag release-nextgen-202609 (requested logical v26.9.99).

Version identity Before upgrade After restart during the failing upgrade
Image tag (Compute and Write) v26.3.10-nextgen release-nextgen-202609
Logical version v26.3.99 requested v26.9.99
Actual TiFlash executable self-reported version 26.3.10 v27.0.0-pre
Actual TiFlash git commit d89f347e5331ad47abe277764f67ff21f6a6976c 4c437ce7f7b0cc8927266e04f1b3b194a15ab4ef

Thus the tested image transition is v26.3.10-nextgen → release-nextgen-202609, and the observed executable transition is 26.3.10 / d89f347e → v27.0.0-pre / 4c437ce7. The target's self-reported version does not equal the requested logical version; both identities are retained rather than inferred from the image tag. This report does not claim every component completed the broader source-to-target upgrade: the case stopped after the TiFlash step's workload gate failed.

Please distinguish requested tags/logical versions from the executable's self-reported identity.

Item Observed value
Source TiFlash Compute and Write runtime 26.3.10, git d89f347e5331ad47abe277764f67ff21f6a6976c
Requested target image tag, both Compute and Write release-nextgen-202609
Requested target logical version v26.9.99
Restarted Compute and Write self-reported release v27.0.0-pre
Actual restarted TiFlash git 4c437ce7f7b0cc8927266e04f1b3b194a15ab4ef
Reported build time 2026-10-25 08:10:36 UTC
client-c gitlink at that TiFlash commit 55aede24990ec5f92a1d74768d65a11eacc5f7d5
gRPC gitlink 591d56e1300b6d11948e1b821efac785a295989c

The pre-release version string is an observation from the deployed executable; it does not by itself prove that an unrelated master image was selected.

5. Clinic logs
Full Clinic entry and complete run window

Full run window: 2026-10-29 15:13:00–15:48:00 UTC+08 (07:13:00–07:48:00 UTC). This covers preparation, the pre-upgrade workload, both upgrade steps reached, the first error and terminal failure/cleanup attempt. The process ran 15:13:35–15:46:12 UTC+08. The links below retain narrower incident windows for convenience; the full-run links above do not restrict logs to selected error keywords. Clinic login/internal access is required.

These links contain filters and absolute times, no credentials. They require internal Clinic access.

Routing: shared pool 643782309392196954, namespace tidb643782309392196954. C++ logs are in ti-serverlog / ti-errorlog, rather than only the tiflash proxy stdout stream.

6. Source analysis: confirmed recovery gap vs. open hypotheses

Confirmed from the exact running commit and matching failure location:

  1. fetchPages() has up to three attempts, but only retries pingcap::Exception.
  2. finishPagesPacketStream() handles non-OK stream completion. Status UNAVAILABLE(14) goes to the default branch and throws DB::Exception(LOGICAL_ERROR).
  3. DB::Exception and pingcap::Exception are distinct exception types. Therefore this failure reaches fetchPages()'s catch (...), which logs at line 412 and immediately rethrows, bypassing its retry loop. This matches the observed server ERROR exactly.
  4. client-c's connection array selects connections round-robin; each address has five clients. This selection does not check channel readiness.
  5. dropConnIfNeeded() exists, but the FetchDisaggPages stream-finish error path does not call it. The background cleanup checks whether addresses remain in PD, rather than whether individual channels are healthy.
  6. gRPC does perform internal reconnect attempts; the logs show them. The finding is not that there is no reconnect at all, but that this query still fails without entering the page-fetch retry branch.

Not yet established:

  • The original cause of connection/handshake timeout: channel/pool recovery, DNS/IP lifecycle, TLS/handshaking, or underlying network.
  • A permanent stale-channel state or a specific failing channel mapped to every successful/failed RPC.
  • Whether a bounded retry alone is sufficient. Retry correctness must consider partial packets, memtable data, snapshot release and ownership; switching a pinned segment snapshot to another replica is not simply replacing its address.
  • Data loss, permanent unavailability, or behavior on a repeated query. No extra workload replay was performed.
7. Related issues and fixes
  • #10513: similar WN rolling-restart symptom, but its documented cause was a long query exceeding the 600-second shutdown grace period; it was closed after scan optimizations #10522 / #10555. This report's short Q1 failure after the first Write was ready needs separate triage.
  • #10994: related disaggregated fetch/establish recovery failures under WN network partition; different injected fault and timeout error.
  • #10425 / #10430: existing WN graceful-shutdown work; not evidence that this post-recovery connection error is covered.
  • tikv/client-c#233: connection-cleanup changes are already an ancestor of this run's client-c dependency; this is not simply evidence that #233 was absent.

Current default master inspected at f3b7883955fc7127476c913ead6ebbdd835c9caf has the same SegmentReadTask.cpp blob (188f88acc996854622e32f0f31f41ed405d6077a) as the running commit. No matching fix for the exception/retry gap was found in the scoped GitHub review. This is source-level verification, not a fixed-build regression test.

Current state

Subsequent upgrades are stopped and the existing environment was retained for diagnostics. No automatic rerun, downgrade, new environment or product-code change was performed. The full-case attempt failed; it must not be reported as a complete smoke pass.

Lingua principale
C++
Stelle
1k
Fork
428
Merge medio
2g 7h
PR unite (30g)
100

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pingcap/tiflash

Tutte le issue di pingcap/tiflash

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.