A block's body is a Markdown string, so no text export or import can round-trip

Open
  • Is something wrong?
#1,442 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
5/100
Issue type
Refactor
Clarity
Needs clarification
Activity status
Active
Tech stack
ocaml, sqlite
Domain
data, database

Research direction

This is a design proposal, not a task with a clear first step. The payload points to lib/syntax/inline.ml (the entity parser that drops the backslash), Type_op.md_escaped, astExportMarkdown, and the fuzz namespaces listed in the issue, which are the places to read to see the collisions. Done would mean a maintainer-agreed data model for block bodies (text runs, refs, code and math nodes) and a decision on :block/title, which needs a design discussion before any code.

Written by the indexing model from the issue text.

Description

A DB graph stores a block's whole body in :block/title as a Markdown string, and every text format Logseq moves blocks through (Copy as text and paste, the Markdown mirror and the file-graph importer, the clipboard) is Markdown read by the same parser. A block's text and the format's structure share one grammar, so text that looks like structure comes back as structure. This cannot be fixed one bug at a time.

Why it cannot be patched

  1. mldoc's parse is not injective. \DeltaG and DeltaG both parse to Plain "DeltaG" (the entity parser in lib/syntax/inline.ml drops the backslash before a word that is not an entity, and mldoc's own test inline-entity / no separator expects it). Once two texts parse the same, no printer can give either one back.
  2. mldoc has no printer that inverts its parser. Type_op.md_escaped is failwith "not impl yet". astExportMarkdown is a lossy exporter: the text \# x parses as plain text # x and prints as # x, which parses as a heading.
  3. The outline grammar cannot hold some block text at all. A block whose text has a line - x shows it as a line of the block; in any outline file that line is a child block. Only an escape can tell them apart, and every escape needs every reader to agree on it.

The bugs it produces

Found by fuzzing the round-trips with no escaping added anywhere: master with the open structural PRs merged, harness on yuxi-liu-wired/logseq roundtrip-fuzz, namespaces frontend.handler.export-paste-fuzz-test, frontend.worker.markdown-mirror-import-fuzz-test, frontend.handler.copy-paste-fuzz-test, logseq.db.sqlite.export-import-fuzz-test, run with FUZZ_N=10 FUZZ_SEED=1 and FUZZ_SEED=11.

Copy as text, then paste:

block before block after
text émoji😀 \tail. text émoji😀 tail.
text >中文 \[^1] text 中文 [^1]
text émoji😀 \(x\)[^1] text émoji😀 $x$ [1]
empty block, tag research block whose text is tags:: research
block $$ with p-choice:: sad, then block \frac{a}{b} one block: $$, tags:: research, p-choice:: sad, - $$, frac{a}{b}, $$

Markdown mirror, then the importer:

block before block after
text DONE a done task, empty text
text ## a heading 2
text [#A] **bold** tail. #[[Beta Gamma]] priority high, tag Beta Gamma, text **bold** tail.
text >{{query (task TODO)}}>中文 a quote block (task TODO)>中文, tag Quote
text :PROPERTIES: / :x: 1 / :END: property x = 1
text ~~s~~key:: v42 empty text
text id:: 6aa|a|b| empty text
text émoji😀 / - a:b '42 the second line becomes a child block

The EDN export and import, which serialize data instead of printing Markdown, found no mismatches on the same seeds.

Patches withdrawn

Each of these escaped or special-cased one collision. They are closed: fixing symptoms hides the design problem, and the fuzzers kept finding new collisions after each one.

  • #13721 export: keep backslash escapes
  • #13743 mirror: escape text a Markdown reader takes as structure
  • #13719 paste: Markdown quote lines as a quote block
  • #13720 export and paste: property, status, priority, tag and collapsed lines
  • #13730 export: text before an embed macro
  • #13732 export: macro syntax, embeds in property values
  • #13742 mirror: tags and status written next to a math fence
  • #13754 paste: keep #tag syntax

Related symptoms already filed: #13755, logseq/db-test#1439, logseq/db-test#1440.

The fix

Store a block's body as data, not as a Markdown string: text runs, emphasis, refs and tags by id, code and math as their own nodes. Keep :block/title a plain-text name. Then exporting is a printer over that data and importing is a parser into it, and the DB's own formats round-trip by serialization, as EDN already does.

Dominant language
No language data
Stars
29
Forks
2
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from logseq/db-test

All issues in logseq/db-test

Similar issues

More Data Engineering issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.