deleted pages older than 30 days cause checksum differences between client and server

Aperta
#1,513 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
clojure

Direzione di ricerca

Start with src/main/frontend/worker/handler/transaction.cljs, deps/outliner/src/logseq/outliner/recycle.cljs, and src/main/frontend/worker/sync/apply_txs.cljs, following the issue’s links to the GC and transaction persistence paths. Reproduce with the CLI steps in the issue and compare local and remote checksums before and after 31 days. Done means recycle GC no longer leaves the client and server checksums inconsistent.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Search first
  • I searched and no similar issues were found
What Happened?

Once a page has been deleted for more than 30 days, it will be deleted by local cleanup, but the deletion is never sent to the server. This means that the local checksum changes, but the server checksum doesn't, and from there on they remain inconsistent.

This is quite possibly the underlying cause of #1341 , but I couldn't tell for sure so I've opted to create as a new more specific issue and link from there. Once this is fixed, the fix can be tried against #1341 to see if it was indeed the same problem.

Reproduce the Bug

Here are reproduction instructions using the CLI. These instructions use faketime to pretend a month has passed for convenience, but I have a real case of this issue, which I've seen with both the CLI and the desktop client, and both have the same checksum that's different to the server checksum.

I've put Claude's theory as to why this is happening into "Additional Context" below, but I wanted to start with something that made sense to me as a human!

An alias for convenience. This runs the Logseq CLI, using the Linux faketime command to make it think the time is different from the server's current time.

LOGSEQ_AT() { _offset="$1"; shift; FAKETIME_DONT_FAKE_MONOTONIC=1 faketime -f "$_offset" node /opt/logseq/static/logseq-cli.js --root-dir /data --graph recycle-gc-repro-2 --output json "$@"; }
  1. Create a new graph and sync it
$ LOGSEQ_AT +0d graph create --enable-sync
{"status":"ok","data":{"graph":"recycle-gc-repro-2","repo":"logseq_db_recycle-gc-repro-2", ...}}
$ LOGSEQ_AT +0d sync start
{"status":"ok","data":{"last-error":null,"ws-state":"open","local-tx":0,"pending-local":0,"pending-asset":0,"pending-server":0,"remote-tx":0,"local-checksum":"46fd0cee68a8017e","remote-checksum":"46fd0cee68a8017e","missing-asset-upload-files":[],"graph-id":"ebc6f1b5-26d7-4ad8-acec-ea720f22fee2","repo":"logseq_db_recycle-gc-repro-2"}}
  1. Add a page and remove it, so it ends up in the recycle
$ LOGSEQ_AT +0d upsert page --page Repro2
{"status":"ok","data":{"result":[196]}}
$ LOGSEQ_AT +0d upsert block --target-page Repro2 --content "delete me too"
{"status":"ok","data":{"result":[197]}}
$ LOGSEQ_AT +0d remove page --page Repro2
{"status":"ok","data":{"result":true,"permanently-deleted?":false}}

Confirming it's in the recycling ("result: 1"):

$ LOGSEQ_AT +0d query --query '[:find (count ?e) . :where [?e :logseq.property/deleted-at]]'
{"status":"ok","data":{"result":1}}

and confirm that local and remote checksums are the same, in this case 22deee402252901e:

$ LOGSEQ_AT +0d sync status
{"status":"ok","data":{"last-error":null,"ws-state":"open","local-tx":3,"pending-local":0,"pending-asset":0,"pending-server":0,"remote-tx":3,"local-checksum":"22deee402252901e","remote-checksum":"22deee402252901e","missing-asset-upload-files":[],"graph-id":"ebc6f1b5-26d7-4ad8-acec-ea720f22fee2","repo":"logseq_db_recycle-gc-repro-2"}}

Stop the worker process, pretend 31 days have passed, and start the sync again:

$ LOGSEQ_AT +0d server stop
[graph-lifecycle] {"event":"worker-termination","repo":"recycle-gc-repro-2","graphsDir":"/data/graphs","generation":"8921c430-d5ae-41da-957e-a7431b0dc5f6","ticket":"eb9db5f6-e6e4-4e8d-ab0b-e337fff0b9f6","pid":982,"stage":"graceful"}
{"status":"ok","data":{"repo":"logseq_db_recycle-gc-repro-2"}}
$ LOGSEQ_AT +31d sync start
{"status":"ok","data":{"last-error":null,"ws-state":"open","local-tx":3,"pending-local":1,"pending-asset":0,"pending-server":0,"remote-tx":3,"local-checksum":"46fd0cee68a8017e","remote-checksum":"22deee402252901e","missing-asset-upload-files":[],"graph-id":"ebc6f1b5-26d7-4ad8-acec-ea720f22fee2","repo":"logseq_db_recycle-gc-repro-2"}}

Once the sync has settled, notice that the local checksum has changed, and no longer matches the remote checksum (which hasn't changed):

$ LOGSEQ_AT +31d sync status
{"status":"ok","data":{"last-error":null,"ws-state":"open","local-tx":4,"pending-local":0,"pending-asset":0,"pending-server":0,"remote-tx":4,"local-checksum":"46fd0cee68a8017e","remote-checksum":"22deee402252901e","missing-asset-upload-files":[],"graph-id":"ebc6f1b5-26d7-4ad8-acec-ea720f22fee2","repo":"logseq_db_recycle-gc-repro-2"}}

See also that the recycled page has now been deleted on the client, presumably by the recycle GC process when the new process started ("result: null"):

$ LOGSEQ_AT +31d query --query '[:find (count ?e) . :where [?e :logseq.property/deleted-at]]'
{"status":"ok","data":{"result":null}}
Expected Behavior

After the recycle GC, local and server checksums still match.

Screenshots

No response

Files

No response

Browser, Desktop or Mobile Platform Information
  • Windows 11
    • desktop app version 2.0.2
  • Linux ARM64 running in Docker node:24.19.0-bookworm-slim container, OS version Debian bookworm
    • CLI app version 2.0.2 (Build time: 2026-10-07T14:07:14.501Z, Revision: 533ac43)
Additional Context

This is something I hit in my real database, and used Claude to help track it back to the reproducible case above. Here's Claude's description of what it thinks is happening:

Once a recycled page or block passes the 30-day retention, the next recycle GC on each client permanently deletes it locally only. The deletion is never sent to the sync server, so the server keeps the entity, and the client's local checksum (which counts it) no longer matches the server's. Nothing is pending and content still syncs, but the checksums never agree again.

Cause, as far as I can read it (permalinks to master d3b4ea8; 2.0.2 is the same):

  • maybe-run-recycle-gc! runs at graph open (db_core.cljs#L862) and after transactions, at most once per 24 h.
  • outliner-recycle/gc! retracts entities recycled more than 30 days ago with {:outliner-op :recycle-gc :persist-op? false}.
  • persistable-local-tx-meta? rejects :persist-op? false, so enqueue-local-tx! never queues the GC for upload.
  • The sync checksum counts recycled entities like any other block or page, and the server has no recycle GC of its own, so the server keeps them indefinitely.

Side effects: the sync indicator for every affected client is in a permanent mismatch, the sync indicator still shows green (an indicator fix, logseq/logseq#13169, was closed unmerged); a fresh download seems to bring long-expired items back into Recycle until that copy's next GC; the server's copy grows with entities every client has deleted.

Also seen while investigating: GET /sync//checksum/diagnostics returns HTTP 500 for my actual (E2EE) graph, so "(Dev) Recompute graph checksum" can't show a diff.

Some options for a fix:

  • sync the GC deletion (drop :persist-op? false from gc!; retracting an already-retracted entity should be a no-op on other clients), or
  • run the same GC on the server, or
  • leave recycled entities out of the checksum on both sides.
Are you willing to submit a PR? If you know how to fix the bug.
  • I'm willing to submit a PR (Thank you!)
Lingua principale
Nessun dato sulla lingua
Stelle
29
Fork
2
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di logseq/db-test

Tutte le issue di logseq/db-test

Issue simili

Altre issue su Databases

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.