[Feature] Split the table hierarchy and materialize the store a table is served by
メンテナーはふだん 3 日以内に返信
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 20/100
調査の方向性
The issue describes changes across Table, FileStoreTable, FormatTable, SystemTable, RestCatalog::GetTable, and the table scan/read paths, but names no source files or tests. Start by locating those definitions and the existing table hierarchy, then read the open pull request #447 to understand the work already underway. Done means the hierarchy and file system behavior match the described solution and the affected tests pass.
索引モデルが issue の本文から書いたものです。
説明
Search before asking
- I searched in the issues and found nothing similar.
Motivation
A single Table class dispatched everything: the differences between managed, format, and system tables lived in if branches inside one 1400-line table.cpp, every operator call (NewScan/NewRead/NewWrite/NewCommit) re-merged a fresh options map and re-parsed CoreOptions — the state Java Paimon's AbstractFileStore parses exactly once — and the file system a table is served by was resolved before the schema's access configuration was even known.
The ambiguity produced real defects:
RestCatalog::GetTable(Identifier("sys", "tables"))returned the read-only schema of a global system table but still passed it to the managed-table load, which statically cast it toTableSchemaand calledPrimaryKeys()on it — undefined behavior.- The operator factory path built the file system from the context options alone; a schema carrying remote backend and credentials (delivered through
SetTableSchema()) was therefore served by the default local backend, and reading its data failed. - The
$roand$audit_logviews built their underlying scans and reads from the context's file system, ignoring the one their table was opened with; a context that also carried a file-system scheme map was then refused outright (WithFileSystem()andWithFileSystemSchemeToIdentifierMap() cannot be used together). FileStoreTable::Copyhalf-applied abranchoption: it was written into the copy's schema options but never moved the copy's branch, so the consistency check reported the branch actually served as a conflict and let a mismatched one through.- An option conflict between a context and a held table was reported with both raw values in the error message, leaking credentials into logs.
- With REST data tokens enabled, the system tables a
GetTable()call returned were bound to the catalog-wide file system, so$snapshotsand friends read the data table's files without the table-level credentials and failed.
Solution
- Split the hierarchy:
Tablebecomes an abstract base (NewXxxpure virtual, a new virtualCopy(dynamic_options)), implemented byFileStoreTable— withAppendOnlyFileStoreTable/PrimaryKeyFileStoreTablesubclasses — byFormatTable(which now derives fromTableand owns its own assembly, absorbingformat_table_loader), and by theSystemTablefamily. - Introduce a
FileStore(the counterpart of Java'sAbstractFileStore) that materializesCoreOptionsonce at construction, resolves the served file system after the options are known (schema-carried access configuration names its own backend; the caller's explicit file system wins over it), carries the file-system scheme map, and hosts the operator factories (NewPathFactory,NewSnapshotManager,NewManifestList, ...).FileStoreTableholds its schema/snapshot managers and its branch;SwitchToBranchderives a table of another branch. - Give context options one clear place to speak: the load. A factory-path load (
TableScan::Create(ctx)etc.) merges the context options over the schema's throughTable::Copy()— a table is the product of its load — whileNewXxxon a held table refuses a contradicting option (option 'X' is 'A' in the context but 'B' on the table, values redacted).Copynever half-appliestypeorbranch; managers are handed over unless a file-system option among the copy's changes the resolution, in which case fresh ones are built. - Fix the surrounding defects:
RestCatalog::GetTableservessys-database global system tables locally and binds per-table system tables toGetTableFileSystem();$ro/$audit_log/$binlogprefer the context's file system and fall back to the table's only when the context names neither a file system nor a scheme map; the scan-fallback read context merges the load's dynamic options so the fallback reader recognises what the main one does.
Anything else?
- Options a schema carries that do not parse now fail at table-build time rather than at the first operator; a table opened with neither a schema-carried backend nor a caller file system is served by the default local one instead of holding a null one.
- No storage format or protocol change. Public API changes:
Tableis abstract (constructor protected,Name/FullName/NewXxx/Copyvirtual,Table::CurrentBranch()replaced byFileStoreTable::CurrentBranch()),FormatTablederives fromTable(staticFormatTable::Copy(table, options)becomes the virtualTable::Copy(options),GetLocation()becomesGetRootPath()), andFileStoreTable::SwitchToBranchis new.
Are you willing to submit a PR?
- I'm willing to submit a PR!
- 主要言語
- C++
- スター
- 70
- フォーク
- 36
- 平均マージ
- 3日 14時間
- マージ済み PR(30日)
- 38
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
apache/paimon-cpp のほかの issue
-
[Bug] Documentation contradicts the implementation: a nonexistent option, a stale version and outdated feature lists対応中かも @lucasfang が今日担当しました。 オープンbug
難易度 4/5 3〜5日 初心者へのやさしさ 35/100
apache/paimon-cpp#466 ·
メンテナーはふだん 3 日以内に返信
-
bug
難易度 4/5 3〜5日 初心者へのやさしさ 57/100
apache/paimon-cpp#454 ·
メンテナーはふだん 3 日以内に返信
-
[Feature] Support hybrid file-index queries with full-text search, vector search, and predicateオープンenhancement
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
apache/paimon-cpp#452 ·
メンテナーはふだん 3 日以内に返信
-
難易度 5/5 1週間以上 初心者へのやさしさ 8/100
apache/paimon-cpp#435 · コメント 2 件 ·
メンテナーはふだん 3 日以内に返信
-
enhancement
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
apache/paimon-cpp#410 ·
メンテナーはふだん 3 日以内に返信
apache/paimon-cpp の issue をすべて見る
似ている issue
-
crash lld:COFF
難易度 2/5 1〜3時間 初心者へのやさしさ 77/100
llvm/llvm-project#235970 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
[Bug]: iOS textarea wraps text mid-word because LynxUIBaseInput hardcodes NSLineBreakByCharWrappingオープン
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
lynx-family/lynx#10455 ·
メンテナーはふだん 1 日以内に返信
-
A-tools P-low S-blocked task
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
cv::checkRange fails with "Unsupported matrix depth" for fp8 depths対応中かも @sshivensolanki が今日担当しました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信
-
Rifle-variant weapon offsets are saved under the pistol's name [FRIK-093]対応中かも @rollingrock が今日担当しました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
rollingrock/Fallout-4-VR-Body#182 ·