[disagg] Query fails with Code: 11000 NoSuchKey when S3 GC (ScanThenDelete) removes a DMFile that a compute node still reads

Aperta
#11,291 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@JinheLin ci sta già lavorando.

Dal 10/11/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

may-affects-25.10 may-affects-26.3 may-affects-26.9 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/moderate type/bug

Bug Report

1. Minimal reproduce step (Required)

Environment: disaggregated TiFlash (write nodes + compute nodes), S3-compatible object store,
remote_gc_method = 2 (ScanThenDelete), remote_gc_interval_seconds = 60 (default is 3600).
Topology: 2 write nodes (tc-tiflash-0, tc-tiflash-1) + 2 compute nodes
(secondary-tc-tiflash-0, secondary-tc-tiflash-1).

  1. Import a TPC-C dataset into a table (order_line, ~48 GB / 2346 regions) so that TiFlash
    writes DMFiles to the object store.
  2. Run a 3-hour AP stability workload on top of it (CH workload: 1000 warehouses, 32 threads,
    TPC-H style queries q1..q23 plus TPC-C transactions). No chaos / fault injection in this step.
  3. After ~73 minutes, the TiFlash S3 GC deletes a set of DMFiles of order_line
    (dmf_10, dmf_12, dmf_15, dmf_17) whose delmark has just expired (1 hour).
  4. ~8 minutes later a compute node opens one of those DMFiles for a query.

The run is not deterministic; the same workload/topology completed successfully on other runs.

2. What did you expect to see? (Required)

The query should keep succeeding, i.e. a compute node should never read a DMFile object that
TiFlash's own GC has already removed. If a DMFile turns out to be already deleted, the query
should recover (refresh the metadata and retry) instead of returning an error to the client.

3. What did you see instead (Required)

A compute node got NoSuchKey (HTTP 404) for the DMFile meta objects, retried 3 times,
then failed the MPP task; the query returned Error 1105 (HY000) and the case failed.
The TiFlash process itself did not crash or restart; only the query failed.

Compute node (secondary-tc-tiflash-1, errorlog):

[2026/11/09 11:40:12.660 +08:00] [ERROR] [Exception.cpp:88] ["Code: 11000, e.displayText() = DB::Exception:
  Open S3 file for read fail after retries when init file, key=s1137/data/ks_1_t_2684/dmf_12/meta,
  e.what() = DB::Exception, Stack trace:
  ...
  DB::S3::S3RandomAccessFile::initialize(...)        dbms/src/Storages/S3/S3RandomAccessFile.cpp:564
  DB::S3::S3RandomAccessFile::S3RandomAccessFile(...) dbms/src/Storages/S3/S3RandomAccessFile.cpp:90
  DB::S3::createFromNormalFile(...)
  DB::S3::S3RandomAccessFile::create(...)            dbms/src/Storages/S3/S3RandomAccessFile.cpp:623
  DB::FileProvider::newRandomAccessFile(...)          dbms/src/IO/FileProvider/FileProvider.cpp:43
  DB::ReadBufferFromRandomAccessFileBuilder::build(...)
  DB::DM::DMFileMetaV2::read(...)                     dbms/src/Storages/DeltaMerge/File/DMFileMeta.h:295
  ...
  "]

The underlying errors are plain 404s for the 4 DMFile meta objects (and their sibling files):

[WARN] [S3Common.cpp:147] ["tag=AWSXmlClient message=HTTP response code: 404 ... Exception name: NoSuchKey"]
[WARN] [S3RandomAccessFile.cpp:549] ["S3 GetObject failed: s3exception_name=NoSuchKey
   s3msg=The specified key does not exist. request_id=..., retry=1/3,
   key=tiflash/s1137/data/ks_1_t_2684/dmf_12/meta"]

Client side:

[ERROR] [ch.go:506] ["end query"] [qno=q7] ["consume time"=24s] [result count"=0]
  [error="Error 1105 (HY000): other error for mpp stream: Code: 11000, e.displayText() = DB::Exception:
   Open S3 file for read fail after retries when init file, key=s1137/data/ks_1_t_2684/dmf_12/meta ..."]
4. What is your TiFlash version? (Required)
TiFlash build: v9.0.0-beta.2.pre-245-g7fd0bf579e
Git Commit Hash: 7fd0bf579ef8e41ae233adacfe455792fa9e718f
Branch: release-nextgen-202609
UTC Build Time: 2026-11-06
Edition: Enterprise (disaggregated / next-gen)

Cluster components in the same environment: TiDB v26.11.0-70128b3734 (CLOUD.202611.0),
CSE TiKV 26.9.0. Object storage: self-hosted MinIO (S3-compatible), bucket tiflash,
root tiflash/.


Timeline (UTC+08:00, 2026-11-09)

time event
10:23:56 compute node secondary-tc-tiflash-1 starts (disaggregated_mode=Compute), no restart afterwards
10:24:30 Using remote_gc_method=ScanThenDelete
10:27:00 workload starts (BR import of the dataset had just finished)
10:31:15 BR restore finishes (total-take≈4m12s, 48.08 GB)
10:31:37 S3 GC (S3GCManager.cpp:468) creates delmarks for dmf_10/12/15/17 of s1137/data/ks_1_t_2684
11:31:03 GC round: delmark exist ... diff_sec=3563.532 expired=false (not yet 1 hour)
11:32:12 GC round: diff_sec=3633.205 expired=true → lists 8 sub-objects of each DMFile and deletes them one by one, including meta; then deletes the delmark
11:40:08 query q7 starts (start_ts=470337612754714633)
11:40:10–12 compute node GetObject → NoSuchKey for dmf_10/12/15/17/meta, 3 retries each
11:40:31 Code: 11000 → Error 1105 → query fails, case fails

GC side logs of the physical deletion (write node tc-tiflash-0, store id 1137):

11:31:03  [INFO] [S3GCManager.cpp:571] ["delmark exist, datafile=s1137/data/ks_1_t_2684/dmf_12
            mark_mtime=2026-11-09T02:31:37Z now=2026-11-09T03:31:00Z diff_sec=3563.532 expired=false"]
11:32:12  [INFO] [S3GCManager.cpp:571] ["delmark exist, datafile=s1137/data/ks_1_t_2684/dmf_12
            mark_mtime=2026-11-09T02:31:37Z now=2026-11-09T03:32:10Z diff_sec=3633.205 expired=true"]
11:32:12  [DEBUG] [S3Common.cpp:895] ["listPrefix page result, prefix=s1137/data/ks_1_t_2684/dmf_12/ keys=8 total_keys=8"]
11:32:12  [INFO]  [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_10/meta"]
11:32:12  [INFO]  [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_12/meta"]
11:32:12  [INFO]  [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_15/meta"]
11:32:12  [INFO]  [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_17/meta"]
11:32:12  [INFO]  [S3GCManager.cpp:692] ["datafile deleted, all sub keys are deleted"]
11:32:12  [INFO]  [S3GCManager.cpp:585] ["datafile delmark deleted, key=s1137/data/ks_1_t_2684/dmf_12.del"]

What we have verified

  • The 404 is not a transient object-store visibility problem and not caused by an external
    cleaner: every failing key was deleted minutes earlier by TiFlash's own GC
    (S3GCManager::physicalRemoveDataFile, which asserts method == ScanThenDelete and logs
    datafile deleted, sub_key=... right after a successful DeleteObject). The listPrefix
    immediately before the deletes returned keys=8, i.e. the objects were still present then.
  • No 5xx / SlowDown / timeout / connection reset from the object store in the whole window;
    the deletion is entirely internal to TiFlash.
  • No TiFlash process crash, OOM or restart during the run (all four TiFlash pods started once at
    10:22–10:24 and kept serving); the failure is a query-level failure only.
  • The 1-hour grace comes from TiFlash itself: S3GCConfig::delmark_expired_hour = 1
    (S3GCManager.h), used only by removeDataFileIfDelmarkExpired()
    (S3GCManager.cpp:560). It is a compiled-in default and does not depend on the object store
    vendor; only the GC method (remote_gc_method) selects the lifecycle path vs this scan path.
  • S3 lock files (tryAddLock) are only registered by the write path (S3LockLocalManager);
    a compute node reading a DMFile does not register any reference, so the 1-hour delmark expiry
    is the only protection for readers.
Lingua principale
C++
Stelle
1k
Fork
428
Merge medio
2g 7h
PR unite (30g)
100

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pingcap/tiflash

Tutte le issue di pingcap/tiflash

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.