[Feature] Replace Bluge with a native ICE-compatible inverted index

Đây là một tracker tổng thể, không phải một ứng viên triển khai. Bắt đầu với leaf issue theo thứ tự #14008, sau đó tiếp tục với #14009 đến #14012 khi các blocker được giải quyết. Toàn bộ công việc hoàn tất khi tất cả caller được nêu đều sử dụng native index, các gate về tính tương thích và khả năng khôi phục đều được thông qua, và các tiêu chí loại bỏ dependency được đáp ứng.

Summary

Replace runtime Bluge/ICE with a minimal BanyanDB-owned inverted index while preserving ICE v3 and snapshot v3. BanyanDB owns index behavior; this is not a new-format migration.

The behavioral and compatibility authority remains BDB-NIDX-SPEC-001 revision 0.2, introduced by skywalking-banyandb#1306 and refined by skywalking-banyandb#1307. The tracker hierarchy below refines implementation granularity so each automated TDD run owns one observable seam and one mergeable live-caller replacement.

Boundary

Public pkg/index/ contracts remain stable for callers. Native implementation types are internal; no Bluge, ICE, or bluge_segment_api type may escape. The five top-level NIDX issues are tracking workstreams, not implementation candidates:

1. bounded direct read-only operations;
2. Property shard index;
3. per-segment series sidx;
4. Stream element idx; and
5. remaining administration plus dependency removal.

TDD execution model

The TDD workflow selects the lowest-number open issue carrying both database and Backlog. Therefore:

• never apply Backlog to this umbrella or a workstream parent;
• only the oldest unblocked leaf may carry Backlog;
• a successor is not unblocked until its predecessor PR merges;
• create downstream leaves just in time, after their blocker has taught us the real seam; and
• every leaf must switch a named production/CLI caller in its merge—no fixture-only, parser-only, codec-only, shadow-only, or test-only leaf.

graph LR
  A["NIDX-01A<br/>baseline count"] --> B["NIDX-01B<br/>generations + deletions"]
  B --> C["NIDX-01C<br/>stored migration walks"]
  C --> D["NIDX-01D<br/>exact schema walks"]
  D --> E["NIDX-01E<br/>repair search-after"]
  E --> P["decompose NIDX-02<br/>from merged seams"]
  P --> S["decompose NIDX-03"] --> T["decompose NIDX-04"] --> R["decompose NIDX-05"]

Pre-filing all downstream leaves would be horizontal slicing: their contracts would describe interfaces that do not exist yet and would be stale when selected. File them only when the preceding workstream closes; their issue numbers will then naturally follow dependency order.

Current executable leaves: NIDX-01

1. #14008 — baseline single-segment ReadOnlyDocCount
2. #14009 — multi-segment generations and deletion masks, blocked by #14008
3. #14010 — match-all stored walks for migration sources, blocked by #14009
4. #14011 — exact-term/OR schema walks, blocked by #14010
5. #14012 — Property repair tuple search-after, blocked by #14011

Each leaf owns checked-in compatibility-writer fixture bytes, literal expected values, one pre-agreed behavioral seam, RED and e2e commands, focused race/package suites, immutable-file proof, typed bounded corruption behavior, and explicit rollback.

Compatibility and durability contract

• Native readers open supported historical ICE v3/snapshot directories and return canonical logical results.
• During the retirement window, the pinned legacy oracle opens native output and can query, merge, restart, and externally receive it where applicable.
• No new disk version, opening-time migration, replay, or rewrite is allowed merely to remove dependencies.
• Required CRC32 fields remain layout bytes only; native code never calculates, validates, caches, or uses them.
• Publication is atomic; restart selects a complete preceding or new generation, never a partial snapshot.
• Malformed input returns bounded typed errors without panic, hang, leak, or unbounded allocation.
• Native index code uses BanyanDB database ownership for writer exclusion; it creates and requires no index-local PID or lock file.
• Read-only operations create no runtime file and do not change bytes, mtimes, or directory entries.

Required final product surface

• Bounded reads: committed generation, visible count, match-all stored walk, exact term/OR, deletion masks, and Property repair search-after.
• Property: upsert/delete, stored fields, required filters/sort, callbacks, snapshots, recovery, expiry merge, GC, repair, and backup.
• Series: field-aware mutation/projection, identity matching/dictionaries, filters/sort, index-mode Measure, snapshots, cache reset, and mixed-version receive.
• Stream elements: batch/NRT, required filters/analyzers, paired postings, doc-value sort, snapshot/restart/receive.
• Administration: verify, rebuild, union/dedup, reconstruction, migration output, dump, and final dependency removal.

BM25, boosts, phrase/fuzzy/geo, highlighting, facets, explanations, generic collectors, and plugin compatibility remain out of scope.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Completion criteria

• Every leaf is merged and its named caller is native; no dormant native implementation lands.
• Product, historical-fixture, two-binary, crash, corruption, race, resource, and performance gates pass for the applicable workstream.
• Operational metrics, mismatch artifacts, recovery guidance, and rollback instructions are exercised.
• NIDX-01 through NIDX-04 add no new case-insensitive bluge token or matching tracked path in active repository surfaces.
• The last NIDX-05 removal leaf deletes every pre-existing case-insensitive bluge token and matching tracked path outside archived design/research evidence and Git history; no alias, filename, runtime name, fixture metadata, or generated license remains.
• Final dependency scans find no retired runtime or CLI edge; compatibility fixtures use neutral oracle identity and immutable hashes.

Top-level workstreams

1. NIDX-01 — bounded read-only access, with five ordered TDD leaves above
2. NIDX-02 — Property shard index, decompose after NIDX-01 closes
3. NIDX-03 — per-segment series sidx, decompose after NIDX-02 closes
4. NIDX-04 — Stream element idx, decompose after NIDX-03 closes
5. NIDX-05 — administration and dependency removal, decompose after NIDX-01 through NIDX-04 close and retirement is approved