[disagg] Query fails with Code: 11000 NoSuchKey when S3 GC (ScanThenDelete) removes a DMFile that a compute node still reads

Đang mở
#11,291 1 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@JinheLin đang làm issue này rồi.

Từ ngày 10/11/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

may-affects-25.10 may-affects-26.3 may-affects-26.9 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/moderate type/bug

Bug Report

1. Minimal reproduce step (Required)

Environment: disaggregated TiFlash (write nodes + compute nodes), S3-compatible object store,
remote_gc_method = 2 (ScanThenDelete), remote_gc_interval_seconds = 60 (default is 3600).
Topology: 2 write nodes (tc-tiflash-0, tc-tiflash-1) + 2 compute nodes
(secondary-tc-tiflash-0, secondary-tc-tiflash-1).

  1. Import a TPC-C dataset into a table (order_line, ~48 GB / 2346 regions) so that TiFlash
    writes DMFiles to the object store.
  2. Run a 3-hour AP stability workload on top of it (CH workload: 1000 warehouses, 32 threads,
    TPC-H style queries q1..q23 plus TPC-C transactions). No chaos / fault injection in this step.
  3. After ~73 minutes, the TiFlash S3 GC deletes a set of DMFiles of order_line
    (dmf_10, dmf_12, dmf_15, dmf_17) whose delmark has just expired (1 hour).
  4. ~8 minutes later a compute node opens one of those DMFiles for a query.

The run is not deterministic; the same workload/topology completed successfully on other runs.

2. What did you expect to see? (Required)

The query should keep succeeding, i.e. a compute node should never read a DMFile object that
TiFlash's own GC has already removed. If a DMFile turns out to be already deleted, the query
should recover (refresh the metadata and retry) instead of returning an error to the client.

3. What did you see instead (Required)

A compute node got NoSuchKey (HTTP 404) for the DMFile meta objects, retried 3 times,
then failed the MPP task; the query returned Error 1105 (HY000) and the case failed.
The TiFlash process itself did not crash or restart; only the query failed.

Compute node (secondary-tc-tiflash-1, errorlog):

[2026/11/09 11:40:12.660 +08:00] [ERROR] [Exception.cpp:88] ["Code: 11000, e.displayText() = DB::Exception:
  Open S3 file for read fail after retries when init file, key=s1137/data/ks_1_t_2684/dmf_12/meta,
  e.what() = DB::Exception, Stack trace:
  ...
  DB::S3::S3RandomAccessFile::initialize(...)        dbms/src/Storages/S3/S3RandomAccessFile.cpp:564
  DB::S3::S3RandomAccessFile::S3RandomAccessFile(...) dbms/src/Storages/S3/S3RandomAccessFile.cpp:90
  DB::S3::createFromNormalFile(...)
  DB::S3::S3RandomAccessFile::create(...)            dbms/src/Storages/S3/S3RandomAccessFile.cpp:623
  DB::FileProvider::newRandomAccessFile(...)          dbms/src/IO/FileProvider/FileProvider.cpp:43
  DB::ReadBufferFromRandomAccessFileBuilder::build(...)
  DB::DM::DMFileMetaV2::read(...)                     dbms/src/Storages/DeltaMerge/File/DMFileMeta.h:295
  ...
  "]

The underlying errors are plain 404s for the 4 DMFile meta objects (and their sibling files):

[WARN] [S3Common.cpp:147] ["tag=AWSXmlClient message=HTTP response code: 404 ... Exception name: NoSuchKey"]
[WARN] [S3RandomAccessFile.cpp:549] ["S3 GetObject failed: s3exception_name=NoSuchKey
   s3msg=The specified key does not exist. request_id=..., retry=1/3,
   key=tiflash/s1137/data/ks_1_t_2684/dmf_12/meta"]

Client side:

[ERROR] [ch.go:506] ["end query"] [qno=q7] ["consume time"=24s] [result count"=0]
  [error="Error 1105 (HY000): other error for mpp stream: Code: 11000, e.displayText() = DB::Exception:
   Open S3 file for read fail after retries when init file, key=s1137/data/ks_1_t_2684/dmf_12/meta ..."]
4. What is your TiFlash version? (Required)
TiFlash build: v9.0.0-beta.2.pre-245-g7fd0bf579e
Git Commit Hash: 7fd0bf579ef8e41ae233adacfe455792fa9e718f
Branch: release-nextgen-202609
UTC Build Time: 2026-11-06
Edition: Enterprise (disaggregated / next-gen)

Cluster components in the same environment: TiDB v26.11.0-70128b3734 (CLOUD.202611.0),
CSE TiKV 26.9.0. Object storage: self-hosted MinIO (S3-compatible), bucket tiflash,
root tiflash/.


Timeline (UTC+08:00, 2026-11-09)

time event
10:23:56 compute node secondary-tc-tiflash-1 starts (disaggregated_mode=Compute), no restart afterwards
10:24:30 Using remote_gc_method=ScanThenDelete
10:27:00 workload starts (BR import of the dataset had just finished)
10:31:15 BR restore finishes (total-take≈4m12s, 48.08 GB)
10:31:37 S3 GC (S3GCManager.cpp:468) creates delmarks for dmf_10/12/15/17 of s1137/data/ks_1_t_2684
11:31:03 GC round: delmark exist ... diff_sec=3563.532 expired=false (not yet 1 hour)
11:32:12 GC round: diff_sec=3633.205 expired=true → lists 8 sub-objects of each DMFile and deletes them one by one, including meta; then deletes the delmark
11:40:08 query q7 starts (start_ts=470337612754714633)
11:40:10–12 compute node GetObject → NoSuchKey for dmf_10/12/15/17/meta, 3 retries each
11:40:31 Code: 11000 → Error 1105 → query fails, case fails

GC side logs of the physical deletion (write node tc-tiflash-0, store id 1137):

11:31:03  [INFO] [S3GCManager.cpp:571] ["delmark exist, datafile=s1137/data/ks_1_t_2684/dmf_12
            mark_mtime=2026-11-09T02:31:37Z now=2026-11-09T03:31:00Z diff_sec=3563.532 expired=false"]
11:32:12  [INFO] [S3GCManager.cpp:571] ["delmark exist, datafile=s1137/data/ks_1_t_2684/dmf_12
            mark_mtime=2026-11-09T02:31:37Z now=2026-11-09T03:32:10Z diff_sec=3633.205 expired=true"]
11:32:12  [DEBUG] [S3Common.cpp:895] ["listPrefix page result, prefix=s1137/data/ks_1_t_2684/dmf_12/ keys=8 total_keys=8"]
11:32:12  [INFO]  [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_10/meta"]
11:32:12  [INFO]  [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_12/meta"]
11:32:12  [INFO]  [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_15/meta"]
11:32:12  [INFO]  [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_17/meta"]
11:32:12  [INFO]  [S3GCManager.cpp:692] ["datafile deleted, all sub keys are deleted"]
11:32:12  [INFO]  [S3GCManager.cpp:585] ["datafile delmark deleted, key=s1137/data/ks_1_t_2684/dmf_12.del"]

What we have verified

  • The 404 is not a transient object-store visibility problem and not caused by an external
    cleaner: every failing key was deleted minutes earlier by TiFlash's own GC
    (S3GCManager::physicalRemoveDataFile, which asserts method == ScanThenDelete and logs
    datafile deleted, sub_key=... right after a successful DeleteObject). The listPrefix
    immediately before the deletes returned keys=8, i.e. the objects were still present then.
  • No 5xx / SlowDown / timeout / connection reset from the object store in the whole window;
    the deletion is entirely internal to TiFlash.
  • No TiFlash process crash, OOM or restart during the run (all four TiFlash pods started once at
    10:22–10:24 and kept serving); the failure is a query-level failure only.
  • The 1-hour grace comes from TiFlash itself: S3GCConfig::delmark_expired_hour = 1
    (S3GCManager.h), used only by removeDataFileIfDelmarkExpired()
    (S3GCManager.cpp:560). It is a compiled-in default and does not depend on the object store
    vendor; only the GC method (remote_gc_method) selects the lifecycle path vs this scan path.
  • S3 lock files (tryAddLock) are only registered by the write path (S3LockLocalManager);
    a compute node reading a DMFile does not register any reference, so the 1-hour delmark expiry
    is the only protection for readers.
Ngôn ngữ chính
C++
Star
1k
Fork
428
Merge trung bình
2 ngày 7 giờ
Pull request đã merge (30 ngày)
100

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của pingcap/tiflash

Tất cả issue của pingcap/tiflash

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.