OME: SkillClusterUpdated persists a 1024-dim vector, inflating run_record ~18x

Abierto
  • ¿Algo no está bien?
#398 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Refactorización
Claridad
Necesita aclaración
Estado de actividad
Tranquilo
Stack tecnológico
python
Área
backend, database

Línea de trabajo

Read _stores/run_record.py, engine.py (around lines 686 and 749), and _background/crash_recovery.py (around line 63) to trace how event payloads are stored and replayed. Compare the audit-projection and re-embedding options, paying particular attention to whether recovery can select a different skill when the vector is unavailable. Done means the recovery-divergence question is resolved and a storage approach is chosen.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What

SkillClusterUpdated.case_vector (added in #393) carries the triggering case's 1024-dim embedding so extract_agent_skill avoids a second embedding call. The event payload is persisted verbatim, so the vector lands in two stores:

  • run_record.event_payload (_stores/run_record.py, column is TEXT NOT NULL)
  • the APScheduler jobstore, while a job is queued (engine.py:686 serializes the event into args)

Measured with a realistic payload:

with case_vector : 14,074 bytes
without          :    765 bytes   -> 18.4x, +13.3 KB/record

At the default max_records_per_strategy = 1000, the skill_cluster_updated ring buffer is ~14 MB instead of ~0.8 MB. It has no audit value — nobody reads 1024 floats out of a run record.

The vector is only consumed when a cluster holds more skills than MAX_SKILLS_IN_PROMPT (extract_agent_skill.py:265-273), which for a highly-aggregated artifact like a skill should be the minority of runs. Most of the time the 13 KB rides along unused.

Why it wasn't fixed in #393

The obvious fix — strip large fields when persisting to run_record — is not local, and this is the part worth carrying forward:

Crash recovery replays the persisted payload to rebuild the event (_background/crash_recovery.py:63 -> engine.py:749 model_validate_json). A trimmed persisted copy deserializes with case_vector is None, which is a legal value that silently routes the recovered run down the md-ordering fallback branch while the original run took the LanceDB-ranked branch. Same event id, different selection, no error. So trimming needs either a payload/audit split with recovery reading the untrimmed copy, or an explicit "vector unavailable after recovery" signal.

Options

  1. Audit projection. Give BaseEvent a hook (or pass model_dump_json(exclude=...)) used only for the run_record write, keeping the recovery path on the full payload. Fixes the audit bloat, leaves the jobstore copy.
  2. Don't carry the vector; re-embed in the > MAX_SKILLS_IN_PROMPT branch. Costs one embedding call on the minority path. Note this does not reintroduce the eventual-consistency dependency #393 removed: that branch already reads LanceDB (find_topk_relevant_in_cluster) and already degrades to md ordering when the index is stale.
  3. Keep as-is and document the sizing (done in CHANGELOG 1.2.3 + #393 body).

Option 2 is the smaller diff and removes the field entirely; option 1 preserves the saved call. Either needs the recovery-divergence question answered first.

Not urgent

Bounded by the ring buffer, so it does not grow without limit. Filed so ome.db sizing is traceable to a decision rather than an accident.

Found during post-merge-review of #393.

Lenguaje dominante
Python
Estrellas
13.3k
Forks
926
Merge medio
19 h 14 min
PR fusionados (30 d)
27

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de EverMind-AI/EverOS

Todos los issues de EverMind-AI/EverOS

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.