storage: concurrent table creation can lose schema updates and stall vector index DDL
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 48/100
- issue の種類
- バグ
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- cpp
調査の方向性
Start with SchemaBuilder::applyTable in dbms/src/TiDB/Schema/SchemaBuilder.cpp and trace its missing-storage branch alongside tryGetDDLGuard and createTable in dbms/src/Interpreters/InterpreterCreateQuery.cpp. Reproduce the reported concurrent schema sync and inspect the persisted table metadata and index visibility before and after sync. Done means the newer schema is applied when concurrent storage creation returns, and the reported vector index DDL completes.
索引モデルが issue の本文から書いたものです。
説明
Bug Report
1. Minimal reproduce step (Required)
On a Classic cluster with one PD, one TiKV, one TiDB, and one TiFlash, repeatedly create a fresh table and immediately add a vector index after enabling its TiFlash replica:
CREATE DATABASE br_vector_race;
CREATE TABLE br_vector_race.t0 (id INT, v VECTOR(3));
ALTER TABLE br_vector_race.t0 SET TIFLASH REPLICA 1;
INSERT INTO br_vector_race.t0 VALUES (1, '[1,2.1,3.3]');
ALTER TABLE br_vector_race.t0
ADD VECTOR INDEX idx((VEC_COSINE_DISTANCE(v))) USING HNSW;
Repeat the four table statements sequentially with fresh names (t1, t2, ...), without waiting for TiFlash replica availability before adding the index. No concurrent client DDL, failpoints, source modifications, or process suspension are needed. The race is timing-dependent: in the recorded run, the first 11 iterations completed in 0.114–0.232 seconds each; the twelfth table (t11) stalled.
The local environment was started with:
tiup playground v9.0.0-beta.2.pre-nightly --tag br-vector-race-20261116 \
--db 1 --kv 1 --pd 1 --tiflash 1 --without-monitor --port-offset 10000
The exact cached binary versions are listed in section 4; the mutable nightly name alone does not identify these binaries.
While the index DDL is still running, inspect from a separate SQL session:
ADMIN SHOW DDL JOBS 20;
SELECT * FROM information_schema.tiflash_replica
WHERE TABLE_SCHEMA='br_vector_race' AND TABLE_NAME='t11';
SELECT * FROM information_schema.tiflash_indexes
WHERE TIDB_DATABASE='br_vector_race' AND TIDB_TABLE='t11';
2. What did you expect to see? (Required)
ADD VECTOR INDEX should complete after TiFlash applies the index's write-reorganization state and builds the one-row index. Concurrent initialization of the same local storage must preserve the newer schema requested by the DDL synchronization caller.
3. What did you see instead (Required)
The index DDL remained in write reorganization with no reported error. For the affected table (ID 177):
information_schema.tiflash_indexesreturned zero rows.- TiFlash's persisted table metadata still had index state 2 / write only, although TiDB had advanced it to 3 / write reorganization.
- Replica status showed
AVAILABLE=0, PROGRESS=1; the availability-update DDL was queued behind the index DDL.
After approximately 67 seconds, sending one schema-sync request to this isolated local test table made the original DDL complete about 60ms later:
curl http://127.0.0.1:30292/tiflash/sync-schema/keyspace/4294967295/table/177
The request applied state 3, TiFlash built the one-row index in about 2ms, and tiflash_indexes then contained one row with ROWS_STABLE_INDEXED=1. This is diagnostic evidence of a missing schema transition, rather than a slow index build.
Observed race
Two independent paths can initialize the same TiFlash storage: incoming snapshot decoding needs the table schema, while the index DDL explicitly requests schema synchronization. Each sync reads table information at its own timestamp.
- The snapshot-processing thread reads the older index state,
write only, and begins storage creation. - The second sync reads
write reorganizationwhile the storage is still absent, and also enters the create path. - The second creator encounters the DDL guard, waits until the first creator has finished, and returns success without applying its newer table information.
- TiDB considers the explicit schema sync successful and proceeds to poll index progress. The persisted write-only index does not create a local index, so the poll cannot observe completion.
Captured local timeline and before/after evidence
Timestamps are from the same local run, timezone -05:00. The following is a condensed extraction of the TiFlash logs; schema states were extracted from the logged TableInfo/Create statements.
| Time | Thread | Event |
|---|---|---|
| 01:04:19.887 | 115 | AtomicGetStorageSchema, region 224, table 177; schema sync begins |
| 01:04:19.906 | 116 | Second schema sync begins for table 177 |
| 01:04:19.968 | 115 | Create with index state 2, revision 3, update_timestamp 470498426172735491 |
| 01:04:19.969 | 116 | Create with index state 3, revision 4, update_timestamp 470498426185580547 |
| 01:04:19.996 | 116 | Concurrent create table happens ... Table db_127.t_177 is creating or attaching right now |
| 01:04:19.996 | 115 | Create and schema sync finish successfully |
| 01:04:20.046 | 116 | Create and schema sync also report success |
The saved t_177.sql metadata still contains index state 2. SQL sampling confirms that the index-system-table result is empty; this conclusion is not inferred from the DDL's zero progress counters.
After the single diagnostic resync request at 01:05:26.751:
01:05:26.778 Alter table db_127.t11 begin
01:05:26.778 Local index info generated, keep=[] added=[index_id=1]
01:05:26.778 EnsureStableLocalIndex - Begin building index
01:05:26.780 EnsureStableLocalIndex - Finish building index
01:05:26.780 Alter table db_127.t11 end; Sync table schema end
01:05:26.804 TiDB: finish DDL job (job 180, synced/public, RowCount=1)
01:05:26.811 Original submitting session: DDL job is finished
The post-resync metadata contains state 3, and replica availability subsequently becomes 1. Full local logs, the reproduction harness, SQL samples, and before/after metadata were retained; the decisive excerpts are included here so the report does not depend on private filesystem links.
Relevant code
SchemaBuilder::applyTable: the missing-storage branch callsapplyCreateStorageInstanceand returns success without subsequently applying the requested schema to the resulting instance.tryGetDDLGuardandcreateTable: a concurrent creator waits for table existence and returns a null guard;createTablethen returns without applying that caller's schema.generateLocalIndexInfos: a new local index is created only in Public or WriteReorganization state, explaining why the stale WriteOnly state produces no index.
The concurrency defect is in shared schema/storage initialization, not specific to full-text indexing. This report directly reproduces HNSW vector indexing; full-text and other index types were not tested.
A related BR CI run, pull_br_integration_test/1269, stalls in the same vector DDL in br_pitr_table_filter and contains the same affected-table concurrent-create warning before the overall 90-minute timeout. The local experiment establishes the race above; that CI log lacks both creators' full metadata, so its precise interleaving cannot be proven retrospectively.
4. What is your TiFlash version? (Required)
Locally reproduced:
| Component | Version / commit |
|---|---|
| TiFlash | v9.0.0-beta.2.pre-234-gad839fd46, ad839fd46ab9f86b5ec742d20e32336371e2177e |
| TiDB | Classic v9.0.0-beta.2.pre-2275-g51263506a5, 51263506a5005ca119f3a191a21c3ea91a419b16 |
| TiKV | 67ffce0c137b7c171405a76df8bc8e9f04a19ce6 |
| PD | 20a18f227345b20f4e0f97305ee4b7300579b7e6 |
Source comparison only: the faulty concurrent-create path is unchanged in CI TiFlash bcc2b5b567162ef527b829f6ccd3a7809139feb9 and master fd9e72e694e85240d74c9b2113322b2d4b9b210f checked on 2026-11-16. The latter's SchemaBuilder.cpp, TiDBSchemaSyncer.cpp, and InterpreterCreateQuery.cpp are identical to the CI revision. Neither newer binary was run locally. No stable-release affected range is established by this report.
- 主要言語
- C++
- スター
- 1k
- フォーク
- 428
- 平均マージ
- 2日 7時間
- マージ済み PR(30日)
- 100
環境構築
このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドなし
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
pingcap/tiflash のほかの issue
-
component/compute type/bug
難易度 2/5 1〜2日 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
Out-of-bounds read in MyDateTime formatting from unvalidated packed values対応中かも @ChangRui-Ryan が今日担当しました。 オープンaffects-25.10 affects-26.3 affects-26.9 affects-7.5 affects-8.1 affects-8.5 severity/major type/bug
難易度 4/5 3〜5日 初心者へのやさしさ 30/100
メンテナーはふだん 1 日以内に返信
-
Missing validation in JSON constructors, LEFT expressions, and CH-block decoding対応中かも @windtalker が今日担当しました。 オープンtype/bug
難易度 4/5 3〜5日 初心者へのやさしさ 25/100
メンテナーはふだん 1 日以内に返信
-
[disagg] Query fails with Code: 11000 NoSuchKey when S3 GC (ScanThenDelete) removes a DMFile that a compute node still reads対応中かも @JinheLin が 6 日前に担当しました。 オープンmay-affects-25.10 may-affects-26.3 may-affects-26.9 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/moderate type/bug
pingcap/tiflash#11291 · コメント 1 件 · 担当者 1 名 ·
メンテナーはふだん 1 日以内に返信
-
Support binding a TiFlash node to a specific keyspace対応中かも @ChangRui-Ryan が 7 日前に担当しました。 オープンtype/enhancement
難易度 4/5 3〜5日 初心者へのやさしさ 42/100
メンテナーはふだん 1 日以内に返信
pingcap/tiflash の issue をすべて見る
似ている issue
-
useless statementオープン
難易度 1/5 1時間未満 初心者へのやさしさ 86/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 65/100
grizzlypeak3d/DJV#962 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
cognizant-backlog
難易度 2/5 1〜3時間 初心者へのやさしさ 61/100
KhronosGroup/OpenCL-CTS#2882 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
Documentation: Fix parameter markup, legacy branch link, and phrasing in Detachable Joints and Model Command tutorials対応中かも @patelshivani2283-lab が今日担当しました。 オープン
難易度 1/5 1〜3時間 初心者へのやさしさ 88/100
メンテナーはふだん 1 日以内に返信