Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

bug(sync): acknowledged hard-delete tombstones are re-journaled on every cloud sync, adding one redundant chunk per run

Open
  • Is something wrong?
#2,152 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Reaan06 is already working on this.

Since Oct 29, 2026.

  • #2198 by @Reaan06 — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, sqlite
Domain
backend, databases

Research direction

Start by tracing the cloud sync export path that creates mutations from hard-delete tombstones, then inspect the journal acknowledgement and deduplication logic. Use the provided SQLite queries to reproduce the repeated delete rows and verify their acknowledgement state. Done means an acknowledged tombstone is not re-queued, pending reaches zero, and an immediate sync creates no additional chunk.

Written by the indexing model from the issue text.

Description

status:needs-review type:bug
Existing Issue Search
  • I searched existing issues and will add evidence to a canonical tracker when applicable.
📝 Observed Consequence

Cloud sync for one project never settles. Seven hard-delete tombstones whose rows are already acknowledged are re-queued on every engram sync --cloud, so the pending counter never reaches 0 and each sync appends one more redundant chunk to both local and remote storage. Over time the journal grows without bound while nothing new is pending: 13 entity_keys already hold more than 100 journal rows, seven of them hold ~3.000 each.

🔄 Steps to Reproduce
  1. Use a store that was migrated from an older release (binary 3.6.1-0.20261026063815-f15b94ab0394).
  2. Inspect the journal churn: 
    sqlite3 ~/.engram/engram.db \
  "SELECT entity_key, count(*) FROM sync_mutations GROUP BY 1 HAVING count(*) > 100 ORDER BY 2 DESC;"
  3. Confirm the entities no longer exist and that their tombstones are recorded: 
    sqlite3 ~/.engram/engram.db \
  "SELECT count(*) FROM observations WHERE sync_id IN (<the churned keys>);"      -- 0
sqlite3 ~/.engram/engram.db \
  "SELECT count(*) FROM sync_delete_tombstones WHERE entity_key IN (<the churned keys>);" -- 7
  4. Note the chunk counts, then sync once: 
    engram sync --cloud --project <project> --status   # Local chunks: 9023 / Remote chunks: 9021
engram sync --cloud --project <project>            # Mutations: 7
engram sync --cloud --project <project> --status   # Local chunks: 9024 / Remote chunks: 9022
  5. Re-read the journal: seven new seq rows with the same entity_keys and op='delete', all acknowledged, and the previous generation already acked. Repeat step 4 and it happens again with fresh seqs (observed chain 84146 → 84197 → 84330 → 84357).
✅ Expected Behavior

An acknowledged hard-delete tombstone is terminal. Once its mutation has been exported and acked, it is not queued again, pending reaches 0 for the project, an immediate re-sync is a no-op, and no additional chunk is created.

❌ Actual Behavior

Every sync re-materializes the same seven deletes with a new seq and a new occurred_at, so:

  • pending never drains for that project (7 before and after each successful sync).
  • One extra chunk per run on both sides (local 9023 → 9024, remote 9021 → 9022; the two-sided gap stays constant, so remote is consistent — nothing is lost, it is duplicated).
  • The journal and the chunk store grow monotonically: 84.331 journal rows total (84.267 acked), of which 21.683 belong to these seven keys alone (2.966–3.171 rows each). Store size reached 809 MB.
  • sync_mutations.disposition is 'pending' for all 84.170 rows, so it cannot be used to distinguish settled rows.
Operating System

Windows (WSL)

Release Version (Report Context)

engram 3.6.1-0.20261026063815-f15b94ab0394

Agent / Client

Other

📋 Relevant Logs
# journal churn per entity_key (top rows)
obs-eaf9a2408f8333be | 3260 | 2026-10-26 18:40:10.664839435
obs-e1049f40d8eedd5c | 3260 | 2026-10-26 18:40:10.664691018
obs-b1840168fbbb9355 | 3260 | 2026-10-26 18:40:10.664518854
obs-a6f585b5116f8dc9 | 3260 | 2026-10-26 18:40:10.664296821
obs-f225868ff6ae2703 | 3171 | 2026-10-26 18:40:06.818199303

# op distribution for the seven churned keys
delete | 21683 | 21676
upsert |     7 |     7

# payload of the newest generation (seq 84357..84363), by design identical to the first
{"deleted": true, "hard_delete": true, "deleted_at": "2026-10-14 12:20:07"}

# same entity_keys, generations apart
obs-21959bc3c54b3164 | 3158 rows | first_seq 35671 | last_seq 84357
obs-f225868ff6ae2703 | 3171 rows | first_seq 24948 | last_seq 84363

# three consecutive syncs, each one chunk
Exporting memories for project "<project>" to cloud...
Created chunk d04ea78c   ... Mutations: 7
Created chunk 3452f824   ... Mutations: 7
Created chunk 421c3a44   ... Mutations: 7
💡 Additional Context
  • Each re-queued row is acknowledged almost immediately by the cloud, so this is not a rejected write: the export step itself regenerates the mutation from the tombstone/absence of the entity instead of honoring the already-settled journal row. The dedupe boundary looks like it keys on the entity instead of the settled (entity, entity_key, op) generation, and stamps a fresh occurred_at/seq each time.
  • Practical effect: engram sync --cloud cannot be used as a convergent "flush" operation on such a store — every call makes the store slightly bigger and never reports zero pending. I stopped re-syncing the affected project deliberately.
  • Related, not duplicates: #1851 (hard project delete writes no tombstone), #1230 (delete on an unenrolled project orphans its pending upsert), #1104 (unbounded journal growth), #1864, #1424.
  • A supported way to retire a settled tombstone (or to stop re-exporting it) would be enough; no manual DB surgery is documented.
Dominant language
Go
Stars
7.1k
Forks
729
Avg merge
8h 35m
Merged PRs (30d)
319

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Gentleman-Programming/engram

All issues in Gentleman-Programming/engram

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.