[Feature] Replace Bluge with a native ICE-compatible inverted index

Chiusa
  • Qualcosa non va?
#13,990 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Attiva
Ambito
backend, database

Direzione di ricerca

Questo è un tracker generale, non un candidato all’implementazione. Inizia con la leaf issue ordinata #14008, quindi procedi con #14009 fino a #14012 man mano che i blocker vengono risolti. Il lavoro complessivo è completato quando tutti i caller indicati usano l’indice nativo, i gate di compatibilità e recupero vengono superati e i criteri per la rimozione della dipendenza sono soddisfatti.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bydb file compatible change database feature

Summary

Replace runtime Bluge/ICE with a minimal BanyanDB-owned inverted index while preserving ICE v3 and snapshot v3. BanyanDB owns index behavior; this is not a new-format migration.

The behavioral and compatibility authority remains BDB-NIDX-SPEC-001 revision 0.2, introduced by skywalking-banyandb#1306 and refined by skywalking-banyandb#1307. The tracker hierarchy below refines implementation granularity so each automated TDD run owns one observable seam and one mergeable live-caller replacement.

Boundary

Public pkg/index/ contracts remain stable for callers. Native implementation types are internal; no Bluge, ICE, or bluge_segment_api type may escape. The five top-level NIDX issues are tracking workstreams, not implementation candidates:

  1. bounded direct read-only operations;
  2. Property shard index;
  3. per-segment series sidx;
  4. Stream element idx; and
  5. remaining administration plus dependency removal.

TDD execution model

The TDD workflow selects the lowest-number open issue carrying both database and Backlog. Therefore:

  • never apply Backlog to this umbrella or a workstream parent;
  • only the oldest unblocked leaf may carry Backlog;
  • a successor is not unblocked until its predecessor PR merges;
  • create downstream leaves just in time, after their blocker has taught us the real seam; and
  • every leaf must switch a named production/CLI caller in its merge—no fixture-only, parser-only, codec-only, shadow-only, or test-only leaf.
graph LR
  A["NIDX-01A<br/>baseline count"] --> B["NIDX-01B<br/>generations + deletions"]
  B --> C["NIDX-01C<br/>stored migration walks"]
  C --> D["NIDX-01D<br/>exact schema walks"]
  D --> E["NIDX-01E<br/>repair search-after"]
  E --> P["decompose NIDX-02<br/>from merged seams"]
  P --> S["decompose NIDX-03"] --> T["decompose NIDX-04"] --> R["decompose NIDX-05"]

Pre-filing all downstream leaves would be horizontal slicing: their contracts would describe interfaces that do not exist yet and would be stale when selected. File them only when the preceding workstream closes; their issue numbers will then naturally follow dependency order.

Current executable leaves: NIDX-01

  1. #14008 — baseline single-segment ReadOnlyDocCount
  2. #14009 — multi-segment generations and deletion masks, blocked by #14008
  3. #14010 — match-all stored walks for migration sources, blocked by #14009
  4. #14011 — exact-term/OR schema walks, blocked by #14010
  5. #14012 — Property repair tuple search-after, blocked by #14011

Each leaf owns checked-in compatibility-writer fixture bytes, literal expected values, one pre-agreed behavioral seam, RED and e2e commands, focused race/package suites, immutable-file proof, typed bounded corruption behavior, and explicit rollback.

Compatibility and durability contract

  • Native readers open supported historical ICE v3/snapshot directories and return canonical logical results.
  • During the retirement window, the pinned legacy oracle opens native output and can query, merge, restart, and externally receive it where applicable.
  • No new disk version, opening-time migration, replay, or rewrite is allowed merely to remove dependencies.
  • Required CRC32 fields remain layout bytes only; native code never calculates, validates, caches, or uses them.
  • Publication is atomic; restart selects a complete preceding or new generation, never a partial snapshot.
  • Malformed input returns bounded typed errors without panic, hang, leak, or unbounded allocation.
  • Native index code uses BanyanDB database ownership for writer exclusion; it creates and requires no index-local PID or lock file.
  • Read-only operations create no runtime file and do not change bytes, mtimes, or directory entries.

Required final product surface

  • Bounded reads: committed generation, visible count, match-all stored walk, exact term/OR, deletion masks, and Property repair search-after.
  • Property: upsert/delete, stored fields, required filters/sort, callbacks, snapshots, recovery, expiry merge, GC, repair, and backup.
  • Series: field-aware mutation/projection, identity matching/dictionaries, filters/sort, index-mode Measure, snapshots, cache reset, and mixed-version receive.
  • Stream elements: batch/NRT, required filters/analyzers, paired postings, doc-value sort, snapshot/restart/receive.
  • Administration: verify, rebuild, union/dedup, reconstruction, migration output, dump, and final dependency removal.

BM25, boosts, phrase/fuzzy/geo, highlighting, facets, explanations, generic collectors, and plugin compatibility remain out of scope.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Completion criteria

  • Every leaf is merged and its named caller is native; no dormant native implementation lands.
  • Product, historical-fixture, two-binary, crash, corruption, race, resource, and performance gates pass for the applicable workstream.
  • Operational metrics, mismatch artifacts, recovery guidance, and rollback instructions are exercised.
  • NIDX-01 through NIDX-04 add no new case-insensitive bluge token or matching tracked path in active repository surfaces.
  • The last NIDX-05 removal leaf deletes every pre-existing case-insensitive bluge token and matching tracked path outside archived design/research evidence and Git history; no alias, filename, runtime name, fixture metadata, or generated license remains.
  • Final dependency scans find no retired runtime or CLI edge; compatibility fixtures use neutral oracle identity and immutable hashes.

Top-level workstreams

  1. NIDX-01 — bounded read-only access, with five ordered TDD leaves above
  2. NIDX-02 — Property shard index, decompose after NIDX-01 closes
  3. NIDX-03 — per-segment series sidx, decompose after NIDX-02 closes
  4. NIDX-04 — Stream element idx, decompose after NIDX-03 closes
  5. NIDX-05 — administration and dependency removal, decompose after NIDX-01 through NIDX-04 close and retirement is approved
Lingua principale
Java
Stelle
25k
Fork
6.6k
Merge medio
12h 56m
PR unite (30g)
47

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/skywalking

Tutte le issue di apache/skywalking

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.