[Feature] Replace Bluge with a native ICE-compatible inverted index

Cerrado
  • ¿Algo no está bien?
#13,990 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Activo
Área
backend, database

Línea de trabajo

Este es un tracker general, no un candidato de implementación. Empieza con la issue hoja ordenada #14008 y luego sigue con #14009 hasta #14012 a medida que se resuelvan los bloqueadores. El trabajo general termina cuando todos los callers mencionados usan el índice nativo, se superan los gates de compatibilidad y recuperación, y se cumplen los criterios de eliminación de la dependencia.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bydb file compatible change database feature

Summary

Replace runtime Bluge/ICE with a minimal BanyanDB-owned inverted index while preserving ICE v3 and snapshot v3. BanyanDB owns index behavior; this is not a new-format migration.

The behavioral and compatibility authority remains BDB-NIDX-SPEC-001 revision 0.2, introduced by skywalking-banyandb#1306 and refined by skywalking-banyandb#1307. The tracker hierarchy below refines implementation granularity so each automated TDD run owns one observable seam and one mergeable live-caller replacement.

Boundary

Public pkg/index/ contracts remain stable for callers. Native implementation types are internal; no Bluge, ICE, or bluge_segment_api type may escape. The five top-level NIDX issues are tracking workstreams, not implementation candidates:

  1. bounded direct read-only operations;
  2. Property shard index;
  3. per-segment series sidx;
  4. Stream element idx; and
  5. remaining administration plus dependency removal.

TDD execution model

The TDD workflow selects the lowest-number open issue carrying both database and Backlog. Therefore:

  • never apply Backlog to this umbrella or a workstream parent;
  • only the oldest unblocked leaf may carry Backlog;
  • a successor is not unblocked until its predecessor PR merges;
  • create downstream leaves just in time, after their blocker has taught us the real seam; and
  • every leaf must switch a named production/CLI caller in its merge—no fixture-only, parser-only, codec-only, shadow-only, or test-only leaf.
graph LR
  A["NIDX-01A<br/>baseline count"] --> B["NIDX-01B<br/>generations + deletions"]
  B --> C["NIDX-01C<br/>stored migration walks"]
  C --> D["NIDX-01D<br/>exact schema walks"]
  D --> E["NIDX-01E<br/>repair search-after"]
  E --> P["decompose NIDX-02<br/>from merged seams"]
  P --> S["decompose NIDX-03"] --> T["decompose NIDX-04"] --> R["decompose NIDX-05"]

Pre-filing all downstream leaves would be horizontal slicing: their contracts would describe interfaces that do not exist yet and would be stale when selected. File them only when the preceding workstream closes; their issue numbers will then naturally follow dependency order.

Current executable leaves: NIDX-01

  1. #14008 — baseline single-segment ReadOnlyDocCount
  2. #14009 — multi-segment generations and deletion masks, blocked by #14008
  3. #14010 — match-all stored walks for migration sources, blocked by #14009
  4. #14011 — exact-term/OR schema walks, blocked by #14010
  5. #14012 — Property repair tuple search-after, blocked by #14011

Each leaf owns checked-in compatibility-writer fixture bytes, literal expected values, one pre-agreed behavioral seam, RED and e2e commands, focused race/package suites, immutable-file proof, typed bounded corruption behavior, and explicit rollback.

Compatibility and durability contract

  • Native readers open supported historical ICE v3/snapshot directories and return canonical logical results.
  • During the retirement window, the pinned legacy oracle opens native output and can query, merge, restart, and externally receive it where applicable.
  • No new disk version, opening-time migration, replay, or rewrite is allowed merely to remove dependencies.
  • Required CRC32 fields remain layout bytes only; native code never calculates, validates, caches, or uses them.
  • Publication is atomic; restart selects a complete preceding or new generation, never a partial snapshot.
  • Malformed input returns bounded typed errors without panic, hang, leak, or unbounded allocation.
  • Native index code uses BanyanDB database ownership for writer exclusion; it creates and requires no index-local PID or lock file.
  • Read-only operations create no runtime file and do not change bytes, mtimes, or directory entries.

Required final product surface

  • Bounded reads: committed generation, visible count, match-all stored walk, exact term/OR, deletion masks, and Property repair search-after.
  • Property: upsert/delete, stored fields, required filters/sort, callbacks, snapshots, recovery, expiry merge, GC, repair, and backup.
  • Series: field-aware mutation/projection, identity matching/dictionaries, filters/sort, index-mode Measure, snapshots, cache reset, and mixed-version receive.
  • Stream elements: batch/NRT, required filters/analyzers, paired postings, doc-value sort, snapshot/restart/receive.
  • Administration: verify, rebuild, union/dedup, reconstruction, migration output, dump, and final dependency removal.

BM25, boosts, phrase/fuzzy/geo, highlighting, facets, explanations, generic collectors, and plugin compatibility remain out of scope.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Completion criteria

  • Every leaf is merged and its named caller is native; no dormant native implementation lands.
  • Product, historical-fixture, two-binary, crash, corruption, race, resource, and performance gates pass for the applicable workstream.
  • Operational metrics, mismatch artifacts, recovery guidance, and rollback instructions are exercised.
  • NIDX-01 through NIDX-04 add no new case-insensitive bluge token or matching tracked path in active repository surfaces.
  • The last NIDX-05 removal leaf deletes every pre-existing case-insensitive bluge token and matching tracked path outside archived design/research evidence and Git history; no alias, filename, runtime name, fixture metadata, or generated license remains.
  • Final dependency scans find no retired runtime or CLI edge; compatibility fixtures use neutral oracle identity and immutable hashes.

Top-level workstreams

  1. NIDX-01 — bounded read-only access, with five ordered TDD leaves above
  2. NIDX-02 — Property shard index, decompose after NIDX-01 closes
  3. NIDX-03 — per-segment series sidx, decompose after NIDX-02 closes
  4. NIDX-04 — Stream element idx, decompose after NIDX-03 closes
  5. NIDX-05 — administration and dependency removal, decompose after NIDX-01 through NIDX-04 close and retirement is approved
Lenguaje dominante
Java
Estrellas
25k
Forks
6.6k
Merge medio
14 h 17 min
PR fusionados (30 d)
39

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/skywalking

Todos los issues de apache/skywalking

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.