Disaggregated TiFlash: FetchDisaggPages UNAVAILABLE aborts Q1 during rolling upgrade with two available replicas
Maintainer thường phản hồi trong vòng 1 ngày
Chưa có ai nhận issue này.
Đánh giá
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức phù hợp với người mới
- 35/100
- Loại issue
- Lỗi
- Độ rõ ràng
- Khá rõ ràng
- Mức độ hoạt động
- Sôi nổi
- Công nghệ
- cpp, grpc
- Lĩnh vực
- databases, distributed-systems
Hướng nghiên cứu
Start with SegmentReadTask.cpp around line 412, where FetchDisaggPages returns gRPC UNAVAILABLE, then read the source analysis in the issue for the retry and error-classification paths at the exact running commit. Identify the affected recovery coverage and add or update the relevant tests; done means transient page-fetch transport failures follow the intended retry or recovery semantics without aborting the query.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Mô tả
Bug Report
Summary
A disaggregated TiFlash rolling upgrade caused a CH Q1 query to fail with MySQL Error 1105 / FetchDisaggPages gRPC UNAVAILABLE(14), despite two Write nodes, two Compute nodes, and test.order_line having two available TiFlash replicas before the upgrade.
The first failing page-fetch happened after the first Write node reported ready and before the second Write node received its termination signal. The same query successfully established snapshots on both Write stores and fetched other segments from the first Write. This is not evidence of both Write nodes being shut down simultaneously.
Source inspection identifies an error-classification/retry coverage gap matching the observed failure. The underlying reason for the initial connection/handshake failures remains unconfirmed; please do not treat a particular connection-pool, DNS, TLS, or network cause as established.
1. Reproduction / observed execution
Observed once in the following real environment; no additional reproduction or upgrade replay was performed.
- AWS us-west-2, TiDB Cloud Premium, disaggregated LEGACY TiFlash topology.
- 2 Write nodes + 2 Compute nodes; Write stores 1004 and 1006.
- Source/downstream logical groups:
cluster1andcluster2. - Environment-only TCMS execution: 8214336.
- The TCMS step itself is only
testbed_readyand succeeded. The failure reported here belongs to a separately launched full local-binary case on that retained environment, not a failed TCMStestbed_readystep. - Case:
premium_upgrade_all_with_ticdc_ch, focus#premium-upgrade-all-with-ticdc-ch#,testcase/upgrade/tidbx_upgrade_all. - Endless source: PR #4179, commit
4838f24eefd74487ff80fa1ca1e665ec4ca939cd. - Test binary SHA-256:
dce51ede36249098a65e86b1a187abc1b7066f548f720163d1bec6cd98ca456b. - Run ID:
smoke-8214336-pr4179-4838f24e; 2026-10-29 15:13:35–15:46:12 UTC+08.
Steps followed by that single complete-case attempt:
- Restore CH-150 into source and downstream.
- Set source
test.order_lineto TiFlash REPLICA 2 and wait for availability. Verify Q1 actually executes through TiFlash. - Verify initial source/downstream CRC consistency.
- Create a case-owned MySQL TiCDC changefeed in
FORCE_SYNCmode; align TiCDC to source imagev26.3.4-nextgen. - Run source CH TP concurrency 2 + AP Q1 concurrency 1, with the case's consistency checking and TiCDC checkpoint monitoring.
- Complete the original 10-minute pre-upgrade observation.
- Upgrade TiCDC to
release-nextgen-202609; then execute the TiFlash Compute/Write upgrade using the same target tag and logical versionv26.9.99. - The AP query fails during the TiFlash rolling window. After the SDK operation completes, the runtime gate reports the workload failure and stops the case. PD and later component upgrades are not reached.
Original observation durations were not shortened. The remaining intended flow was later component upgrades, 10-minute post-observation, checkpoint catch-up, final CRC/MPP checks and cleanup. The complete case did not pass.
Replica evidence:
- TiDB server DDL logs at 15:20:35.250 / 15:20:35.345 UTC+08 record:
ALTER TABLE test.order_line SET TIFLASH REPLICA 2. - Pre-upgrade observation at 15:22:12, subsequent rolling-window observations, and the post-failure observation at 15:46:36 UTC+08 report:
REPLICA_COUNT=2, AVAILABLE=1, PROGRESS=1. - These are table-level readiness observations, not proof that every physical replica or every RPC connection was healthy at every instant.
2. Expected behavior
The rolling upgrade should preserve successful AP reads under the supported two-Write/two-Compute configuration, or recover a transient page-fetch transport failure within the product's retry/recovery semantics. A healthy replica count should not be mistaken for proof that this transport failure is impossible.
3. Actual result and first-error timeline
All times below are 2026-10-29 UTC+08 (Asia/Shanghai). Log bodies use UTC, eight hours earlier.
| Time UTC+08 | Evidence |
|---|---|
| 15:43:24.296 | TiFlash SDK upgrade step starts. |
| 15:43:43.509 | Write 0glner / store 1004 receives a termination signal. |
| 15:43:46.188 | Write 1004 closes listening sockets. |
| 15:44:25.631 | Compute 6zypjl reports ready; Write 1004 is still starting. |
| 15:44:56.307 | Write 1004 reports Ready for connections. |
| 15:45:04.780 etc. | Compute-to-Write1004 gRPC logs still show connection timeout / FD Shutdown and internal reconnect attempts. |
| 15:45:06.911 | Previous AP Q1 completes successfully. |
| 15:45:07.577 | Failing Q1 starts; query TSO 470092325199806622. |
| 15:45:07.592 / .597 | Compute receives successful EstablishDisaggTask responses from stores 1004 and 1006. |
| 15:45:07.663–.684 | Write1004 serves other segments (14, 5, 23) for this same query. |
| 15:45:07.706 | First page-fetch ERROR: Compute 6zypjl, SegmentReadTask.cpp:412, segment s1004_ks1_t80_1_5_3658451676704_4939483, gRPC status 14. |
| 15:45:07.708–.731 | Compute ExchangeReceiver/Pipeline cancels the MPP query. |
| 15:45:07.733 / .737 | TiDB records the MPP error; client Q1 receives Error 1105. |
| 15:45:08.399 | AP workload exits; cancellation errors in other workload tasks follow. |
| 15:45:10.079 | Second Write 5rx4pc / store 1006 receives termination signal, after the first AP failure. |
| 15:45:11.202 | Compute logs a further handshake timeout to Write1004. |
| 15:45:53.038 | Second Write reports ready. |
| 15:46:12.369 | SDK marks TiFlash operation completed. The following runtime gate fails because the workload already exited; process ends with exit code 1. |
Condensed first-error evidence (server log timestamp is UTC):
[2026/10/29 07:45:07.706 +00:00] [ERROR] [SegmentReadTask.cpp:412]
Code: 49, DB::Exception: Failed to fetch all pages for
s1004_ks1_t80_1_5_3658451676704_4939483,
wn_address=<Write1004>:3930 status=14
message=failed to connect to all addresses
The subsequent client error is Error 1105 (HY000): other error for mpp stream ... Receiver state: ERROR ... Failed to fetch all pages ....
4. Actual versions
Upgrade version boundary (Compute and Write)
Both TiFlash Compute and TiFlash Write were upgraded from image tag v26.3.10-nextgen (logical v26.3.99) to image tag release-nextgen-202609 (requested logical v26.9.99).
| Version identity | Before upgrade | After restart during the failing upgrade |
|---|---|---|
| Image tag (Compute and Write) | v26.3.10-nextgen |
release-nextgen-202609 |
| Logical version | v26.3.99 |
requested v26.9.99 |
| Actual TiFlash executable self-reported version | 26.3.10 |
v27.0.0-pre |
| Actual TiFlash git commit | d89f347e5331ad47abe277764f67ff21f6a6976c |
4c437ce7f7b0cc8927266e04f1b3b194a15ab4ef |
Thus the tested image transition is v26.3.10-nextgen → release-nextgen-202609, and the observed executable transition is 26.3.10 / d89f347e → v27.0.0-pre / 4c437ce7. The target's self-reported version does not equal the requested logical version; both identities are retained rather than inferred from the image tag. This report does not claim every component completed the broader source-to-target upgrade: the case stopped after the TiFlash step's workload gate failed.
Please distinguish requested tags/logical versions from the executable's self-reported identity.
| Item | Observed value |
|---|---|
| Source TiFlash Compute and Write runtime | 26.3.10, git d89f347e5331ad47abe277764f67ff21f6a6976c |
| Requested target image tag, both Compute and Write | release-nextgen-202609 |
| Requested target logical version | v26.9.99 |
| Restarted Compute and Write self-reported release | v27.0.0-pre |
| Actual restarted TiFlash git | 4c437ce7f7b0cc8927266e04f1b3b194a15ab4ef |
| Reported build time | 2026-10-25 08:10:36 UTC |
| client-c gitlink at that TiFlash commit | 55aede24990ec5f92a1d74768d65a11eacc5f7d5 |
| gRPC gitlink | 591d56e1300b6d11948e1b821efac785a295989c |
The pre-release version string is an observation from the deployed executable; it does not by itself prove that an unrelated master image was selected.
5. Clinic logs
Full Clinic entry and complete run window
- Clinic cluster entry — search by shared pool ID
- Full-run logs — all components, no error/text filter
- Full-run TiFlash logs — Compute + Write, all log streams, no error/text filter
Full run window: 2026-10-29 15:13:00–15:48:00 UTC+08 (07:13:00–07:48:00 UTC). This covers preparation, the pre-upgrade workload, both upgrade steps reached, the first error and terminal failure/cleanup attempt. The process ran 15:13:35–15:46:12 UTC+08. The links below retain narrower incident windows for convenience; the full-run links above do not restrict logs to selected error keywords. Clinic login/internal access is required.
These links contain filters and absolute times, no credentials. They require internal Clinic access.
- Compute first error and handshake/connection errors — 15:44:50–15:45:20 UTC+08; focus on 15:45:07.706 (log body 07:45:07.706 UTC).
- Write rolling restart lifecycle — 15:43:20–15:46:20 UTC+08.
Routing: shared pool 643782309392196954, namespace tidb643782309392196954. C++ logs are in ti-serverlog / ti-errorlog, rather than only the tiflash proxy stdout stream.
6. Source analysis: confirmed recovery gap vs. open hypotheses
Confirmed from the exact running commit and matching failure location:
fetchPages()has up to three attempts, but only retriespingcap::Exception.finishPagesPacketStream()handles non-OK stream completion. StatusUNAVAILABLE(14)goes to the default branch and throwsDB::Exception(LOGICAL_ERROR).DB::Exceptionandpingcap::Exceptionare distinct exception types. Therefore this failure reachesfetchPages()'scatch (...), which logs at line 412 and immediately rethrows, bypassing its retry loop. This matches the observed server ERROR exactly.- client-c's connection array selects connections round-robin; each address has five clients. This selection does not check channel readiness.
dropConnIfNeeded()exists, but theFetchDisaggPagesstream-finish error path does not call it. The background cleanup checks whether addresses remain in PD, rather than whether individual channels are healthy.- gRPC does perform internal reconnect attempts; the logs show them. The finding is not that there is no reconnect at all, but that this query still fails without entering the page-fetch retry branch.
Not yet established:
- The original cause of connection/handshake timeout: channel/pool recovery, DNS/IP lifecycle, TLS/handshaking, or underlying network.
- A permanent stale-channel state or a specific failing channel mapped to every successful/failed RPC.
- Whether a bounded retry alone is sufficient. Retry correctness must consider partial packets, memtable data, snapshot release and ownership; switching a pinned segment snapshot to another replica is not simply replacing its address.
- Data loss, permanent unavailability, or behavior on a repeated query. No extra workload replay was performed.
7. Related issues and fixes
- #10513: similar WN rolling-restart symptom, but its documented cause was a long query exceeding the 600-second shutdown grace period; it was closed after scan optimizations #10522 / #10555. This report's short Q1 failure after the first Write was ready needs separate triage.
- #10994: related disaggregated fetch/establish recovery failures under WN network partition; different injected fault and timeout error.
- #10425 / #10430: existing WN graceful-shutdown work; not evidence that this post-recovery connection error is covered.
- tikv/client-c#233: connection-cleanup changes are already an ancestor of this run's client-c dependency; this is not simply evidence that #233 was absent.
Current default master inspected at f3b7883955fc7127476c913ead6ebbdd835c9caf has the same SegmentReadTask.cpp blob (188f88acc996854622e32f0f31f41ed405d6077a) as the running commit. No matching fix for the exception/retry gap was found in the scoped GitHub review. This is source-level verification, not a fixed-build regression test.
Current state
Subsequent upgrades are stopped and the existing environment was retained for diagnostics. No automatic rerun, downgrade, new environment or product-code change was performed. The full-case attempt failed; it must not be reported as a complete smoke pass.
- Ngôn ngữ chính
- C++
- Star
- 1k
- Fork
- 428
- Merge trung bình
- 2 ngày 7 giờ
- Pull request đã merge (30 ngày)
- 100
Chuẩn bị môi trường
Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.
- Không có Dockerfile hay tệp Docker Compose
- Có mẫu pull request
- Không có hướng dẫn đóng góp
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Issue khác của pingcap/tiflash
-
component/compute type/bug
Độ khó 2/5 1-2 ngày Mức phù hợp với người mới 82/100
Maintainer thường phản hồi trong vòng 1 ngày
-
Out-of-bounds read in MyDateTime formatting from unvalidated packed valuesCó thể đã có người làm @ChangRui-Ryan đã nhận hôm nay. Đang mởaffects-25.10 affects-26.3 affects-26.9 affects-7.5 affects-8.1 affects-8.5 severity/major type/bug
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 30/100
Maintainer thường phản hồi trong vòng 1 ngày
-
component/storage severity/moderate type/bug
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 48/100
pingcap/tiflash#11335 · 5 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Missing validation in JSON constructors, LEFT expressions, and CH-block decodingCó thể đã có người làm @windtalker đã nhận hôm nay. Đang mởtype/bug
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 25/100
Maintainer thường phản hồi trong vòng 1 ngày
-
[disagg] Query fails with Code: 11000 NoSuchKey when S3 GC (ScanThenDelete) removes a DMFile that a compute node still readsCó thể đã có người làm @JinheLin đã nhận 6 ngày trước. Đang mởmay-affects-25.10 may-affects-26.3 may-affects-26.9 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/moderate type/bug
pingcap/tiflash#11291 · 1 bình luận · 1 người được giao ·
Maintainer thường phản hồi trong vòng 1 ngày
Tất cả issue của pingcap/tiflash
Issue tương tự
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 72/100
ROCm/FastFlowLM#1057 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
documentation GPU low priority
Độ khó 1/5 1-3 giờ Mức phù hợp với người mới 87/100
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 72/100
Simba-Avionic/srp#510 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 78/100
Maintainer thường phản hồi trong vòng 1 ngày
-
useless statementĐang mở
Độ khó 1/5 Dưới một giờ Mức phù hợp với người mới 86/100
Maintainer thường phản hồi trong vòng 1 ngày