bug(sync): acknowledged hard-delete tombstones are re-journaled on every cloud sync, adding one redundant chunk per run

Abierto
  • ¿Algo no está bien?
#2,152 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@Reaan06 ya está trabajando en esto.

Desde el 29/10/2026.

  • #2198 de @Reaan06 — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
go, sqlite
Área
backend, databases

Línea de trabajo

Start by tracing the cloud sync export path that creates mutations from hard-delete tombstones, then inspect the journal acknowledgement and deduplication logic. Use the provided SQLite queries to reproduce the repeated delete rows and verify their acknowledgement state. Done means an acknowledged tombstone is not re-queued, pending reaches zero, and an immediate sync creates no additional chunk.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

status:needs-review type:bug
Existing Issue Search
  • I searched existing issues and will add evidence to a canonical tracker when applicable.
📝 Observed Consequence

Cloud sync for one project never settles. Seven hard-delete tombstones whose rows are already acknowledged are re-queued on every engram sync --cloud, so the pending counter never reaches 0 and each sync appends one more redundant chunk to both local and remote storage. Over time the journal grows without bound while nothing new is pending: 13 entity_keys already hold more than 100 journal rows, seven of them hold ~3.000 each.

🔄 Steps to Reproduce
  1. Use a store that was migrated from an older release (binary 3.6.1-0.20261026063815-f15b94ab0394).
  2. Inspect the journal churn: 
    sqlite3 ~/.engram/engram.db \
  "SELECT entity_key, count(*) FROM sync_mutations GROUP BY 1 HAVING count(*) > 100 ORDER BY 2 DESC;"
  3. Confirm the entities no longer exist and that their tombstones are recorded: 
    sqlite3 ~/.engram/engram.db \
  "SELECT count(*) FROM observations WHERE sync_id IN (<the churned keys>);"      -- 0
sqlite3 ~/.engram/engram.db \
  "SELECT count(*) FROM sync_delete_tombstones WHERE entity_key IN (<the churned keys>);" -- 7
  4. Note the chunk counts, then sync once: 
    engram sync --cloud --project <project> --status   # Local chunks: 9023 / Remote chunks: 9021
engram sync --cloud --project <project>            # Mutations: 7
engram sync --cloud --project <project> --status   # Local chunks: 9024 / Remote chunks: 9022
  5. Re-read the journal: seven new seq rows with the same entity_keys and op='delete', all acknowledged, and the previous generation already acked. Repeat step 4 and it happens again with fresh seqs (observed chain 84146 → 84197 → 84330 → 84357).
✅ Expected Behavior

An acknowledged hard-delete tombstone is terminal. Once its mutation has been exported and acked, it is not queued again, pending reaches 0 for the project, an immediate re-sync is a no-op, and no additional chunk is created.

❌ Actual Behavior

Every sync re-materializes the same seven deletes with a new seq and a new occurred_at, so:

  • pending never drains for that project (7 before and after each successful sync).
  • One extra chunk per run on both sides (local 9023 → 9024, remote 9021 → 9022; the two-sided gap stays constant, so remote is consistent — nothing is lost, it is duplicated).
  • The journal and the chunk store grow monotonically: 84.331 journal rows total (84.267 acked), of which 21.683 belong to these seven keys alone (2.966–3.171 rows each). Store size reached 809 MB.
  • sync_mutations.disposition is 'pending' for all 84.170 rows, so it cannot be used to distinguish settled rows.
Operating System

Windows (WSL)

Release Version (Report Context)

engram 3.6.1-0.20261026063815-f15b94ab0394

Agent / Client

Other

📋 Relevant Logs
# journal churn per entity_key (top rows)
obs-eaf9a2408f8333be | 3260 | 2026-10-26 18:40:10.664839435
obs-e1049f40d8eedd5c | 3260 | 2026-10-26 18:40:10.664691018
obs-b1840168fbbb9355 | 3260 | 2026-10-26 18:40:10.664518854
obs-a6f585b5116f8dc9 | 3260 | 2026-10-26 18:40:10.664296821
obs-f225868ff6ae2703 | 3171 | 2026-10-26 18:40:06.818199303

# op distribution for the seven churned keys
delete | 21683 | 21676
upsert |     7 |     7

# payload of the newest generation (seq 84357..84363), by design identical to the first
{"deleted": true, "hard_delete": true, "deleted_at": "2026-10-14 12:20:07"}

# same entity_keys, generations apart
obs-21959bc3c54b3164 | 3158 rows | first_seq 35671 | last_seq 84357
obs-f225868ff6ae2703 | 3171 rows | first_seq 24948 | last_seq 84363

# three consecutive syncs, each one chunk
Exporting memories for project "<project>" to cloud...
Created chunk d04ea78c   ... Mutations: 7
Created chunk 3452f824   ... Mutations: 7
Created chunk 421c3a44   ... Mutations: 7
💡 Additional Context
  • Each re-queued row is acknowledged almost immediately by the cloud, so this is not a rejected write: the export step itself regenerates the mutation from the tombstone/absence of the entity instead of honoring the already-settled journal row. The dedupe boundary looks like it keys on the entity instead of the settled (entity, entity_key, op) generation, and stamps a fresh occurred_at/seq each time.
  • Practical effect: engram sync --cloud cannot be used as a convergent "flush" operation on such a store — every call makes the store slightly bigger and never reports zero pending. I stopped re-syncing the affected project deliberately.
  • Related, not duplicates: #1851 (hard project delete writes no tombstone), #1230 (delete on an unenrolled project orphans its pending upsert), #1104 (unbounded journal growth), #1864, #1424.
  • A supported way to retire a settled tombstone (or to stop re-exporting it) would be enough; no manual DB surgery is documented.
Lenguaje dominante
Go
Estrellas
7.1k
Forks
729
Merge medio
10 h 12 min
PR fusionados (30 d)
272

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Gentleman-Programming/engram

Todos los issues de Gentleman-Programming/engram

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.