A block's body is a Markdown string, so no text export or import can round-trip

Aperta
  • Qualcosa non va?
#1,442 1 commento 2 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
5/100
Tipo di issue
Refactoring
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
ocaml, sqlite
Ambito
data, database

Direzione di ricerca

This is a design proposal, not a task with a clear first step. The payload points to lib/syntax/inline.ml (the entity parser that drops the backslash), Type_op.md_escaped, astExportMarkdown, and the fuzz namespaces listed in the issue, which are the places to read to see the collisions. Done would mean a maintainer-agreed data model for block bodies (text runs, refs, code and math nodes) and a decision on :block/title, which needs a design discussion before any code.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

A DB graph stores a block's whole body in :block/title as a Markdown string, and every text format Logseq moves blocks through (Copy as text and paste, the Markdown mirror and the file-graph importer, the clipboard) is Markdown read by the same parser. A block's text and the format's structure share one grammar, so text that looks like structure comes back as structure. This cannot be fixed one bug at a time.

Why it cannot be patched

  1. mldoc's parse is not injective. \DeltaG and DeltaG both parse to Plain "DeltaG" (the entity parser in lib/syntax/inline.ml drops the backslash before a word that is not an entity, and mldoc's own test inline-entity / no separator expects it). Once two texts parse the same, no printer can give either one back.
  2. mldoc has no printer that inverts its parser. Type_op.md_escaped is failwith "not impl yet". astExportMarkdown is a lossy exporter: the text \# x parses as plain text # x and prints as # x, which parses as a heading.
  3. The outline grammar cannot hold some block text at all. A block whose text has a line - x shows it as a line of the block; in any outline file that line is a child block. Only an escape can tell them apart, and every escape needs every reader to agree on it.

The bugs it produces

Found by fuzzing the round-trips with no escaping added anywhere: master with the open structural PRs merged, harness on yuxi-liu-wired/logseq roundtrip-fuzz, namespaces frontend.handler.export-paste-fuzz-test, frontend.worker.markdown-mirror-import-fuzz-test, frontend.handler.copy-paste-fuzz-test, logseq.db.sqlite.export-import-fuzz-test, run with FUZZ_N=10 FUZZ_SEED=1 and FUZZ_SEED=11.

Copy as text, then paste:

block before block after
text émoji😀 \tail. text émoji😀 tail.
text >中文 \[^1] text 中文 [^1]
text émoji😀 \(x\)[^1] text émoji😀 $x$ [1]
empty block, tag research block whose text is tags:: research
block $$ with p-choice:: sad, then block \frac{a}{b} one block: $$, tags:: research, p-choice:: sad, - $$, frac{a}{b}, $$

Markdown mirror, then the importer:

block before block after
text DONE a done task, empty text
text ## a heading 2
text [#A] **bold** tail. #[[Beta Gamma]] priority high, tag Beta Gamma, text **bold** tail.
text >{{query (task TODO)}}>中文 a quote block (task TODO)>中文, tag Quote
text :PROPERTIES: / :x: 1 / :END: property x = 1
text ~~s~~key:: v42 empty text
text id:: 6aa|a|b| empty text
text émoji😀 / - a:b '42 the second line becomes a child block

The EDN export and import, which serialize data instead of printing Markdown, found no mismatches on the same seeds.

Patches withdrawn

Each of these escaped or special-cased one collision. They are closed: fixing symptoms hides the design problem, and the fuzzers kept finding new collisions after each one.

  • #13721 export: keep backslash escapes
  • #13743 mirror: escape text a Markdown reader takes as structure
  • #13719 paste: Markdown quote lines as a quote block
  • #13720 export and paste: property, status, priority, tag and collapsed lines
  • #13730 export: text before an embed macro
  • #13732 export: macro syntax, embeds in property values
  • #13742 mirror: tags and status written next to a math fence
  • #13754 paste: keep #tag syntax

Related symptoms already filed: #13755, logseq/db-test#1439, logseq/db-test#1440.

The fix

Store a block's body as data, not as a Markdown string: text runs, emphasis, refs and tags by id, code and math as their own nodes. Keep :block/title a plain-text name. Then exporting is a printer over that data and importing is a parser into it, and the DB's own formats round-trip by serialization, as EDN already does.

Lingua principale
Nessun dato sulla lingua
Stelle
29
Fork
2
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di logseq/db-test

Tutte le issue di logseq/db-test

Issue simili

Altre issue su Data Engineering

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.