[Bug] Reflection V1: large single-cluster merges cause partial deprecated_by writes with no compensation, no per-merge source cap, and no prompt/language configuration

Start with everos/memory/reflection/orchestrator.py, especially _deprecate_lance_episodes, _deprecate_lance_facts, _deprecate, and _detect_orphans; then trace how reflect_episodes.py invokes EpisodeReflector.areflect. Review the write timeout in everos/core/persistence/lancedb/repository.py and the prompt in everalgo/user_memory/prompts/en/reflect.py. The issue lists three possible improvements; agree on scope first, then verify deprecation consistency, oversized-cluster handling, or configurable prompt/language behavior for the chosen change.

Summary

Three related defects in Reflection V1 (reflect_episodes) that compound on large clusters:

1. Partial deprecation with no compensation — the per-episode LanceDB deprecated_by updates run as N concurrent writes; when some exceed the write-lock deadline under load, the run is reported failed while the already-applied updates persist. Sources are left pointing deprecated_by at a merged entry that may be invalid or already removed, with no rollback or reconcile pass.
2. No per-cluster source cap — a merge consumes all members of a cluster (observed source_count=228 → one narrative). Only _MAX_CLUSTERS_PER_RUN = 10 is bounded. Oversized merges produce narratives whose cascade upsert exceeds the 15 s write-lock deadline, starving the same run's deprecate updates (feedback loop, see below).
3. No prompt/language entry point — the reflector prompt is hard-wired to the upstream English prompt; merged output language is left to the model (observed mixed Chinese/English/French, occasionally garbage), with no [reflection] config to inject instructions.

Environment

• EverOS 1.3.1, everalgo-core 0.3.0, LanceDB index
• LLM: mimo-v2.5 via an OpenAI-compatible proxy that does support strict structured outputs — the merge LLM call succeeds, so this is not a provider-shape failure

Observed behavior

Manual trigger over an existing 228-member cluster:

reflection_merged ... cluster_id=cl_bb3f91313016 mode=init source_count=228
lancedb_write_lock_deadline_exceeded ... op=update table=episode budget_seconds=15.0 acquired=False (repeated xN)
reflection_deprecate_failed ... cluster_id=cl_bb3f91313016

After reflection_deprecate_failed, LanceDB inspection showed 179–297 episode rows (and thousands of atomic_fact rows) with deprecated_by already written pointing at the merged entry — deprecation partially applied before failing, and nothing reconciles the leftovers. Concurrently cascade_worker_recoverable logged VectorStoreBusyError: upsert on table 'episode' exceeded its 15s write-lock deadline while indexing the same oversized narrative.

Root cause

Timeout mechanism (measured). The storm is queue-wait starvation behind the single per-table lock, not slow individual ops:

• lancedb_write_lock_deadline_exceeded: 3516× acquired=False vs 21× acquired=True → overwhelmingly "did not acquire the lock within 15 s" while someone else held it.
• Lock bodies are long on memory-bound hosts: lancedb_write_lock_slow_hold shows routine 8–16 s holds, and one maintenance op held a table lock 260.5 s (op=rebuild_indexes table=agent_case) on 2.4 GB RAM + swap. LanceDB writes, table-handle resolution and embedding are slow under swap churn.
• The cascade worker itself cannot index the merged narrative — its upsert (+embed) exceeds 15 s, so it both fails and holds/contends the same lock the deprecations need.
• Feedback loop: the giant markdown entry → cascade reprocesses it (upsert + downstream extract_atomic_facts) → more writes to the same table, while the run's N per-row deprecate updates queue behind them. The partial residuals keep cascade busy, so the storm continues for minutes after reflect_episodes stops — observed ~100–220 table=episode deadline events/min from 06:01→06:51 (UTC+8), including a stretch while the strategy was disabled, until the oversized markdown entries were removed.

Code-level defects (everos/memory/reflection/orchestrator.py unless noted):

• _deprecate_lance_episodes (L838) builds one update() coroutine per source episode via asyncio.gather(...) without return_exceptions (same for _deprecate_lance_facts L868 per parent id). Each update runs under the hard-coded _WRITE_TIMEOUT_SECONDS = 15.0 (everos/core/persistence/lancedb/repository.py:37).
• On failure, _deprecate (L642) catches AppError, logs reflection_deprecate_failed (L693), returns None — already-applied updates remain, no compensation. _detect_orphans (L505) only detects merged-entries-whose-sources-were-not-deprecated, not the inverse.
• reflect_episodes.py builds EpisodeReflector(llm=...) and the orchestrator calls areflect(algo_episodes) with no prompt; the prompt is hard-coded (English) in everalgo/user_memory/prompts/en/reflect.py, and no [reflection] section exists in everos/config/default.toml.
• Cluster size is governed only by [clustering] threshold = 0.65 / time_window_days = 7.0; reflection has no per-merge member cap.

Reproduction

1. Any cluster large enough that its merged narrative's cascade upsert exceeds ~15 s (memory-bound host, big cluster).
2. POST /api/v2/ome/trigger {"name":"reflect_episodes","timeout":180,"force":true}.
3. Observe reflection_merged source_count=N → lancedb_write_lock_deadline_exceeded storm → reflection_deprecate_failed; query episode/atomic_fact — a non-empty subset carries deprecated_by toward the (possibly removed) merged entry.
4. No later reflect_episodes run repairs these rows; cascade reconcile only rebuilds tables from markdown whose md changed, so the residuals can stay deprecated indefinitely.

Impact

• Searchable memory silently loses rows (deprecated_by IS NULL is the default filter) that point at failed/invalid merged episodes — data effectively hidden with dangling references.
• One large cluster can drive hundreds of lancedb_write_lock_deadline_exceeded warnings per minute on every weekly run.
• Merged episodes are stored in model-chosen (mixed) languages, not controllable by the operator.

Suggested fixes (independently useful)

1. Transactional / compensating deprecation (L838/L868): batch the deprecate into a single atomic update over entry_id IN (...), or revert the already-written deprecated_by on _deprecate failure, or extend orphan detection to flag (and auto-revert) "source deprecated toward a merged entry that does not exist / was never committed".
2. Per-merge source cap / oversized-cluster policy: add [reflection] max_sources_per_merge (skip or truncate clusters beyond N) so the merged narrative stays inside the write-lock budget, breaking the lock feedback loop.
3. Prompt & language configuration: expose [reflection] prompt / prompt_language, plumbed through reflect_episodes.py → EpisodeReflector.areflect(..., prompt=...) (the algo API already accepts prompt), so operators can pin output language without patching site-packages.

Related hardening already merged: #385 (deadline must cover table-handle resolution), #392 (stall class), #379/#384/#390. Related open: #316 (asyncio deadlock between cascade + extract + LanceDB write), #398 (run_record inflation). This issue documents what we hit with the 1.2.2/1.3.1 hardening in place — the remaining gaps are in the reflection orchestration layer above those primitives.