[disagg] Query fails with Code: 11000 NoSuchKey when S3 GC (ScanThenDelete) removes a DMFile that a compute node still reads
I maintainer di solito rispondono entro 1 giorno
@JinheLin ci sta già lavorando.
Dal 10/11/2026.
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
Bug Report
1. Minimal reproduce step (Required)
Environment: disaggregated TiFlash (write nodes + compute nodes), S3-compatible object store,
remote_gc_method = 2 (ScanThenDelete), remote_gc_interval_seconds = 60 (default is 3600).
Topology: 2 write nodes (tc-tiflash-0, tc-tiflash-1) + 2 compute nodes
(secondary-tc-tiflash-0, secondary-tc-tiflash-1).
- Import a TPC-C dataset into a table (
order_line, ~48 GB / 2346 regions) so that TiFlash
writes DMFiles to the object store. - Run a 3-hour AP stability workload on top of it (CH workload: 1000 warehouses, 32 threads,
TPC-H style queries q1..q23 plus TPC-C transactions). No chaos / fault injection in this step. - After ~73 minutes, the TiFlash S3 GC deletes a set of DMFiles of
order_line
(dmf_10,dmf_12,dmf_15,dmf_17) whose delmark has just expired (1 hour). - ~8 minutes later a compute node opens one of those DMFiles for a query.
The run is not deterministic; the same workload/topology completed successfully on other runs.
2. What did you expect to see? (Required)
The query should keep succeeding, i.e. a compute node should never read a DMFile object that
TiFlash's own GC has already removed. If a DMFile turns out to be already deleted, the query
should recover (refresh the metadata and retry) instead of returning an error to the client.
3. What did you see instead (Required)
A compute node got NoSuchKey (HTTP 404) for the DMFile meta objects, retried 3 times,
then failed the MPP task; the query returned Error 1105 (HY000) and the case failed.
The TiFlash process itself did not crash or restart; only the query failed.
Compute node (secondary-tc-tiflash-1, errorlog):
[2026/11/09 11:40:12.660 +08:00] [ERROR] [Exception.cpp:88] ["Code: 11000, e.displayText() = DB::Exception:
Open S3 file for read fail after retries when init file, key=s1137/data/ks_1_t_2684/dmf_12/meta,
e.what() = DB::Exception, Stack trace:
...
DB::S3::S3RandomAccessFile::initialize(...) dbms/src/Storages/S3/S3RandomAccessFile.cpp:564
DB::S3::S3RandomAccessFile::S3RandomAccessFile(...) dbms/src/Storages/S3/S3RandomAccessFile.cpp:90
DB::S3::createFromNormalFile(...)
DB::S3::S3RandomAccessFile::create(...) dbms/src/Storages/S3/S3RandomAccessFile.cpp:623
DB::FileProvider::newRandomAccessFile(...) dbms/src/IO/FileProvider/FileProvider.cpp:43
DB::ReadBufferFromRandomAccessFileBuilder::build(...)
DB::DM::DMFileMetaV2::read(...) dbms/src/Storages/DeltaMerge/File/DMFileMeta.h:295
...
"]
The underlying errors are plain 404s for the 4 DMFile meta objects (and their sibling files):
[WARN] [S3Common.cpp:147] ["tag=AWSXmlClient message=HTTP response code: 404 ... Exception name: NoSuchKey"]
[WARN] [S3RandomAccessFile.cpp:549] ["S3 GetObject failed: s3exception_name=NoSuchKey
s3msg=The specified key does not exist. request_id=..., retry=1/3,
key=tiflash/s1137/data/ks_1_t_2684/dmf_12/meta"]
Client side:
[ERROR] [ch.go:506] ["end query"] [qno=q7] ["consume time"=24s] [result count"=0]
[error="Error 1105 (HY000): other error for mpp stream: Code: 11000, e.displayText() = DB::Exception:
Open S3 file for read fail after retries when init file, key=s1137/data/ks_1_t_2684/dmf_12/meta ..."]
4. What is your TiFlash version? (Required)
TiFlash build: v9.0.0-beta.2.pre-245-g7fd0bf579e
Git Commit Hash: 7fd0bf579ef8e41ae233adacfe455792fa9e718f
Branch: release-nextgen-202609
UTC Build Time: 2026-11-06
Edition: Enterprise (disaggregated / next-gen)
Cluster components in the same environment: TiDB v26.11.0-70128b3734 (CLOUD.202611.0),
CSE TiKV 26.9.0. Object storage: self-hosted MinIO (S3-compatible), bucket tiflash,
root tiflash/.
Timeline (UTC+08:00, 2026-11-09)
| time | event |
|---|---|
| 10:23:56 | compute node secondary-tc-tiflash-1 starts (disaggregated_mode=Compute), no restart afterwards |
| 10:24:30 | Using remote_gc_method=ScanThenDelete |
| 10:27:00 | workload starts (BR import of the dataset had just finished) |
| 10:31:15 | BR restore finishes (total-take≈4m12s, 48.08 GB) |
| 10:31:37 | S3 GC (S3GCManager.cpp:468) creates delmarks for dmf_10/12/15/17 of s1137/data/ks_1_t_2684 |
| 11:31:03 | GC round: delmark exist ... diff_sec=3563.532 expired=false (not yet 1 hour) |
| 11:32:12 | GC round: diff_sec=3633.205 expired=true → lists 8 sub-objects of each DMFile and deletes them one by one, including meta; then deletes the delmark |
| 11:40:08 | query q7 starts (start_ts=470337612754714633) |
| 11:40:10–12 | compute node GetObject → NoSuchKey for dmf_10/12/15/17/meta, 3 retries each |
| 11:40:31 | Code: 11000 → Error 1105 → query fails, case fails |
GC side logs of the physical deletion (write node tc-tiflash-0, store id 1137):
11:31:03 [INFO] [S3GCManager.cpp:571] ["delmark exist, datafile=s1137/data/ks_1_t_2684/dmf_12
mark_mtime=2026-11-09T02:31:37Z now=2026-11-09T03:31:00Z diff_sec=3563.532 expired=false"]
11:32:12 [INFO] [S3GCManager.cpp:571] ["delmark exist, datafile=s1137/data/ks_1_t_2684/dmf_12
mark_mtime=2026-11-09T02:31:37Z now=2026-11-09T03:32:10Z diff_sec=3633.205 expired=true"]
11:32:12 [DEBUG] [S3Common.cpp:895] ["listPrefix page result, prefix=s1137/data/ks_1_t_2684/dmf_12/ keys=8 total_keys=8"]
11:32:12 [INFO] [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_10/meta"]
11:32:12 [INFO] [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_12/meta"]
11:32:12 [INFO] [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_15/meta"]
11:32:12 [INFO] [S3GCManager.cpp:689] ["datafile deleted, sub_key=s1137/data/ks_1_t_2684/dmf_17/meta"]
11:32:12 [INFO] [S3GCManager.cpp:692] ["datafile deleted, all sub keys are deleted"]
11:32:12 [INFO] [S3GCManager.cpp:585] ["datafile delmark deleted, key=s1137/data/ks_1_t_2684/dmf_12.del"]
What we have verified
- The 404 is not a transient object-store visibility problem and not caused by an external
cleaner: every failing key was deleted minutes earlier by TiFlash's own GC
(S3GCManager::physicalRemoveDataFile, which assertsmethod == ScanThenDeleteand logs
datafile deleted, sub_key=...right after a successfulDeleteObject). ThelistPrefix
immediately before the deletes returnedkeys=8, i.e. the objects were still present then. - No 5xx /
SlowDown/ timeout / connection reset from the object store in the whole window;
the deletion is entirely internal to TiFlash. - No TiFlash process crash, OOM or restart during the run (all four TiFlash pods started once at
10:22–10:24 and kept serving); the failure is a query-level failure only. - The 1-hour grace comes from TiFlash itself:
S3GCConfig::delmark_expired_hour = 1
(S3GCManager.h), used only byremoveDataFileIfDelmarkExpired()
(S3GCManager.cpp:560). It is a compiled-in default and does not depend on the object store
vendor; only the GC method (remote_gc_method) selects the lifecycle path vs this scan path. - S3 lock files (
tryAddLock) are only registered by the write path (S3LockLocalManager);
a compute node reading a DMFile does not register any reference, so the 1-hour delmark expiry
is the only protection for readers.
- Lingua principale
- C++
- Stelle
- 1k
- Fork
- 428
- Merge medio
- 2g 7h
- PR unite (30g)
- 100
Preparare l'ambiente
Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Nessuna guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di pingcap/tiflash
-
component/compute type/bug
Difficoltà 2/5 1-2 giorni Idoneità per principianti 82/100
I maintainer di solito rispondono entro 1 giorno
-
Out-of-bounds read in MyDateTime formatting from unvalidated packed valuesForse già presa @ChangRui-Ryan l’ha presa oggi. Apertaaffects-25.10 affects-26.3 affects-26.9 affects-7.5 affects-8.1 affects-8.5 severity/major type/bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
I maintainer di solito rispondono entro 1 giorno
-
component/storage severity/moderate type/bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
pingcap/tiflash#11335 · 5 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Missing validation in JSON constructors, LEFT expressions, and CH-block decodingForse già presa @windtalker l’ha presa oggi. Apertatype/bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 25/100
I maintainer di solito rispondono entro 1 giorno
-
Support binding a TiFlash node to a specific keyspaceForse già presa @ChangRui-Ryan l’ha presa 7 giorni fa. Apertatype/enhancement
Difficoltà 4/5 3-5 giorni Idoneità per principianti 42/100
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di pingcap/tiflash
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno
-
useless statementAperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 86/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
grizzlypeak3d/DJV#962 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 1 giorno