storage: concurrent table creation can lose schema updates and stall vector index DDL

オープン
#11,335 コメント 5 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
cpp

調査の方向性

Start with SchemaBuilder::applyTable in dbms/src/TiDB/Schema/SchemaBuilder.cpp and trace its missing-storage branch alongside tryGetDDLGuard and createTable in dbms/src/Interpreters/InterpreterCreateQuery.cpp. Reproduce the reported concurrent schema sync and inspect the persisted table metadata and index visibility before and after sync. Done means the newer schema is applied when concurrent storage creation returns, and the reported vector index DDL completes.

索引モデルが issue の本文から書いたものです。

説明

component/storage severity/moderate type/bug

Bug Report

1. Minimal reproduce step (Required)

On a Classic cluster with one PD, one TiKV, one TiDB, and one TiFlash, repeatedly create a fresh table and immediately add a vector index after enabling its TiFlash replica:

CREATE DATABASE br_vector_race;

CREATE TABLE br_vector_race.t0 (id INT, v VECTOR(3));
ALTER TABLE br_vector_race.t0 SET TIFLASH REPLICA 1;
INSERT INTO br_vector_race.t0 VALUES (1, '[1,2.1,3.3]');
ALTER TABLE br_vector_race.t0
  ADD VECTOR INDEX idx((VEC_COSINE_DISTANCE(v))) USING HNSW;

Repeat the four table statements sequentially with fresh names (t1, t2, ...), without waiting for TiFlash replica availability before adding the index. No concurrent client DDL, failpoints, source modifications, or process suspension are needed. The race is timing-dependent: in the recorded run, the first 11 iterations completed in 0.114–0.232 seconds each; the twelfth table (t11) stalled.

The local environment was started with:

tiup playground v9.0.0-beta.2.pre-nightly --tag br-vector-race-20261116 \
  --db 1 --kv 1 --pd 1 --tiflash 1 --without-monitor --port-offset 10000

The exact cached binary versions are listed in section 4; the mutable nightly name alone does not identify these binaries.

While the index DDL is still running, inspect from a separate SQL session:

ADMIN SHOW DDL JOBS 20;
SELECT * FROM information_schema.tiflash_replica
WHERE TABLE_SCHEMA='br_vector_race' AND TABLE_NAME='t11';
SELECT * FROM information_schema.tiflash_indexes
WHERE TIDB_DATABASE='br_vector_race' AND TIDB_TABLE='t11';
2. What did you expect to see? (Required)

ADD VECTOR INDEX should complete after TiFlash applies the index's write-reorganization state and builds the one-row index. Concurrent initialization of the same local storage must preserve the newer schema requested by the DDL synchronization caller.

3. What did you see instead (Required)

The index DDL remained in write reorganization with no reported error. For the affected table (ID 177):

  • information_schema.tiflash_indexes returned zero rows.
  • TiFlash's persisted table metadata still had index state 2 / write only, although TiDB had advanced it to 3 / write reorganization.
  • Replica status showed AVAILABLE=0, PROGRESS=1; the availability-update DDL was queued behind the index DDL.

After approximately 67 seconds, sending one schema-sync request to this isolated local test table made the original DDL complete about 60ms later:

curl http://127.0.0.1:30292/tiflash/sync-schema/keyspace/4294967295/table/177

The request applied state 3, TiFlash built the one-row index in about 2ms, and tiflash_indexes then contained one row with ROWS_STABLE_INDEXED=1. This is diagnostic evidence of a missing schema transition, rather than a slow index build.

Observed race

Two independent paths can initialize the same TiFlash storage: incoming snapshot decoding needs the table schema, while the index DDL explicitly requests schema synchronization. Each sync reads table information at its own timestamp.

  1. The snapshot-processing thread reads the older index state, write only, and begins storage creation.
  2. The second sync reads write reorganization while the storage is still absent, and also enters the create path.
  3. The second creator encounters the DDL guard, waits until the first creator has finished, and returns success without applying its newer table information.
  4. TiDB considers the explicit schema sync successful and proceeds to poll index progress. The persisted write-only index does not create a local index, so the poll cannot observe completion.
Captured local timeline and before/after evidence

Timestamps are from the same local run, timezone -05:00. The following is a condensed extraction of the TiFlash logs; schema states were extracted from the logged TableInfo/Create statements.

Time Thread Event
01:04:19.887 115 AtomicGetStorageSchema, region 224, table 177; schema sync begins
01:04:19.906 116 Second schema sync begins for table 177
01:04:19.968 115 Create with index state 2, revision 3, update_timestamp 470498426172735491
01:04:19.969 116 Create with index state 3, revision 4, update_timestamp 470498426185580547
01:04:19.996 116 Concurrent create table happens ... Table db_127.t_177 is creating or attaching right now
01:04:19.996 115 Create and schema sync finish successfully
01:04:20.046 116 Create and schema sync also report success

The saved t_177.sql metadata still contains index state 2. SQL sampling confirms that the index-system-table result is empty; this conclusion is not inferred from the DDL's zero progress counters.

After the single diagnostic resync request at 01:05:26.751:

01:05:26.778  Alter table db_127.t11 begin
01:05:26.778  Local index info generated, keep=[] added=[index_id=1]
01:05:26.778  EnsureStableLocalIndex - Begin building index
01:05:26.780  EnsureStableLocalIndex - Finish building index
01:05:26.780  Alter table db_127.t11 end; Sync table schema end
01:05:26.804  TiDB: finish DDL job (job 180, synced/public, RowCount=1)
01:05:26.811  Original submitting session: DDL job is finished

The post-resync metadata contains state 3, and replica availability subsequently becomes 1. Full local logs, the reproduction harness, SQL samples, and before/after metadata were retained; the decisive excerpts are included here so the report does not depend on private filesystem links.

Relevant code
  • SchemaBuilder::applyTable: the missing-storage branch calls applyCreateStorageInstance and returns success without subsequently applying the requested schema to the resulting instance.
  • tryGetDDLGuard and createTable: a concurrent creator waits for table existence and returns a null guard; createTable then returns without applying that caller's schema.
  • generateLocalIndexInfos: a new local index is created only in Public or WriteReorganization state, explaining why the stale WriteOnly state produces no index.

The concurrency defect is in shared schema/storage initialization, not specific to full-text indexing. This report directly reproduces HNSW vector indexing; full-text and other index types were not tested.

A related BR CI run, pull_br_integration_test/1269, stalls in the same vector DDL in br_pitr_table_filter and contains the same affected-table concurrent-create warning before the overall 90-minute timeout. The local experiment establishes the race above; that CI log lacks both creators' full metadata, so its precise interleaving cannot be proven retrospectively.

4. What is your TiFlash version? (Required)

Locally reproduced:

Component Version / commit
TiFlash v9.0.0-beta.2.pre-234-gad839fd46, ad839fd46ab9f86b5ec742d20e32336371e2177e
TiDB Classic v9.0.0-beta.2.pre-2275-g51263506a5, 51263506a5005ca119f3a191a21c3ea91a419b16
TiKV 67ffce0c137b7c171405a76df8bc8e9f04a19ce6
PD 20a18f227345b20f4e0f97305ee4b7300579b7e6

Source comparison only: the faulty concurrent-create path is unchanged in CI TiFlash bcc2b5b567162ef527b829f6ccd3a7809139feb9 and master fd9e72e694e85240d74c9b2113322b2d4b9b210f checked on 2026-11-16. The latter's SchemaBuilder.cpp, TiDBSchemaSyncer.cpp, and InterpreterCreateQuery.cpp are identical to the CI revision. Neither newer binary was run locally. No stable-release affected range is established by this report.

主要言語
C++
スター
1k
フォーク
428
平均マージ
2日 7時間
マージ済み PR(30日)
100

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

  • Dockerfile・Docker Compose ファイルなし
  • プルリクエストのテンプレートあり
  • コントリビューションガイドなし

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

pingcap/tiflash のほかの issue

pingcap/tiflash の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。