bug(sync): autosync blocks local writes every cycle while a non-enrolled legacy backlog exists (v3.6.0)

Chiusa
  • Qualcosa non va?
#2,166 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@dnlrsls ci sta già lavorando.

Dal 1/11/2026.

  • #2215 di @dnlrsls — integrata

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go, sqlite
Ambito
backend, database

Direzione di ricerca

Read internal/cloud/autosync and internal/store/store.go, especially Manager.push and QuarantineIrreparableSyncMutations. Trace why each poll scans pending cloud mutations for projects that are not enrolled and how that holds the SQLite write lock; the report provides a reproducible large-backlog scenario and an offline server setup. Done when autosync no longer repeatedly blocks local requests on undeliverable backlog rows, while enrolled project sync continues to work.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

status:needs-review type:bug
Existing Issue Search
  • I searched existing issues and will add evidence to a canonical tracker when applicable.
📝 Observed Consequence

After enabling cloud autosync (one enrolled project), the local engram serve stops responding for most of every 30-second cycle:

  • Agent memory writes time out (mem_save, mem_delete, session registration). One delete was applied but reported as unconfirmed.
  • GET /health and GET /sync/status time out (2 s client limit) for about 25 of every 30 seconds.
  • engram doctor fails with migration: database is locked (5) (SQLITE_BUSY).
  • The process uses close to one CPU core continuously.

The enrolled project itself syncs correctly.

The store holds 35,239 pending cloud mutations from 85 non-enrolled projects, journaled from 2026-05-20 to 2026-09-13, before cloud was configured on this machine. None are newer.

🔄 Steps to Reproduce
  1. Take a store with a large backlog of pending cloud mutations for non-enrolled projects (here: 35,239 rows, ~25 MB of payload, 228 MB database).
  2. Run engram serve with ENGRAM_CLOUD_AUTOSYNC=1, a cloud server and a token.
  3. Poll GET /health every few seconds across the first poll cycles.

No network is needed: I reproduced it on a copy of the store with ENGRAM_CLOUD_SERVER=https://127.0.0.1:1 (unreachable).

✅ Expected Behavior

A backlog that autosync never delivers should not block local writes on every cycle.

❌ Actual Behavior

On the store copy with the unreachable server, /health timed out (>2 s) when the first poll cycle ran, at about 40 s. The process had used 14.3 s of CPU by then.

The same copy with only those 35,239 rows changed behaves normally. Same measurement (engram 3.6.0, offline):

Store copy /health at the first cycle CPU after ~40 s Autosync phase after the cycle
Unchanged timeout (>2 s) 14.3 s no answer
Rows moved to target_key='cloud:inbox' ~0.1 s 1.9 s degraded / transport_failed (expected offline)
Rows set to disposition='superseded' ~0.1 s 2.2 s degraded / transport_failed (expected offline)

Both changed copies kept every memory table and the sync_mutations row count identical; nothing was deleted. Running engram cloud enroll <project> on them backfilled the project's current state (67 observations, 23 sessions, 47 prompts, 12 relations), which matches its live rows.

Workaround applied on the live store: the inbox variant, in one transaction. Afterwards /health answered all 30 probes over ~65 s, the process used 2.4 s of CPU, autosync reported healthy, and a new memory of the enrolled project saved at once and reached the cloud.

Operating System

Windows

Release Version (Report Context)

3.6.0

Agent / Client

Other

📋 Relevant Logs
[autosync] started (server=https://<redacted>)
[autosync] quarantined 214 irreparable pending mutation(s); inspect with `engram doctor --check sync_mutation_required_fields`
engram: engram: migration: database is locked (5) (SQLITE_BUSY)

/sync/status last_error: "pending cloud sync mutations are blocked because project(s) are not enrolled: … (85 projects)"

💡 Additional Context

Client: Pi with gentle-engram, which starts engram serve with ENGRAM_CLOUD_AUTOSYNC=1. OS: Windows 11 Pro.

Possible cause. This is my reading of the source, not proven by a test:

  • Manager.push calls QuarantineIrreparableSyncMutations(targetKey, "", true) on every cycle: the 30 s poll, plus the debounce after each write.
  • With an empty project, that scans every pending cloud row of every project, including non-enrolled ones. It validates each row inside withTx, and the store opens transactions with _txlock=immediate, so the write lock is held for the whole scan.
  • The backlog is never delivered, so the scan repeats every cycle.
  • internal/cloud/autosync and internal/store/store.go are unchanged between v3.6.0 and main @ 9824739.

Related:

  • #1896: same symptom, different path (enrolled repair backfill); closed.
  • #1013: non-enrolled backlog; list/claim/drop not shipped.
  • #1188: inbox. docs/CLOUD-INBOX-STATE.md says legacy migration is not implemented, so moving old rows to the inbox has to be done by hand with SQL.
Lingua principale
Go
Stelle
7.1k
Fork
729
Merge medio
10h 51m
PR unite (30g)
277

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Gentleman-Programming/engram

Tutte le issue di Gentleman-Programming/engram

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.