A block's body is a Markdown string, so no text export or import can round-trip

オープン
  • 何か問題がありますか？
#1,442 コメント 1 件 リアクション 2 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
5/100
issue の種類
リファクタリング
明瞭さ
説明が足りない
活発さ
活発
技術スタック
ocaml, sqlite
領域
data, database

調査の方向性

This is a design proposal, not a task with a clear first step. The payload points to lib/syntax/inline.ml (the entity parser that drops the backslash), Type_op.md_escaped, astExportMarkdown, and the fuzz namespaces listed in the issue, which are the places to read to see the collisions. Done would mean a maintainer-agreed data model for block bodies (text runs, refs, code and math nodes) and a decision on :block/title, which needs a design discussion before any code.

索引モデルが issue の本文から書いたものです。

説明

A DB graph stores a block's whole body in :block/title as a Markdown string, and every text format Logseq moves blocks through (Copy as text and paste, the Markdown mirror and the file-graph importer, the clipboard) is Markdown read by the same parser. A block's text and the format's structure share one grammar, so text that looks like structure comes back as structure. This cannot be fixed one bug at a time.

Why it cannot be patched

  1. mldoc's parse is not injective. \DeltaG and DeltaG both parse to Plain "DeltaG" (the entity parser in lib/syntax/inline.ml drops the backslash before a word that is not an entity, and mldoc's own test inline-entity / no separator expects it). Once two texts parse the same, no printer can give either one back.
  2. mldoc has no printer that inverts its parser. Type_op.md_escaped is failwith "not impl yet". astExportMarkdown is a lossy exporter: the text \# x parses as plain text # x and prints as # x, which parses as a heading.
  3. The outline grammar cannot hold some block text at all. A block whose text has a line - x shows it as a line of the block; in any outline file that line is a child block. Only an escape can tell them apart, and every escape needs every reader to agree on it.

The bugs it produces

Found by fuzzing the round-trips with no escaping added anywhere: master with the open structural PRs merged, harness on yuxi-liu-wired/logseq roundtrip-fuzz, namespaces frontend.handler.export-paste-fuzz-test, frontend.worker.markdown-mirror-import-fuzz-test, frontend.handler.copy-paste-fuzz-test, logseq.db.sqlite.export-import-fuzz-test, run with FUZZ_N=10 FUZZ_SEED=1 and FUZZ_SEED=11.

Copy as text, then paste:

block before block after
text émoji😀 \tail. text émoji😀 tail.
text >中文 \[^1] text 中文 [^1]
text émoji😀 \(x\)[^1] text émoji😀 $x$ [1]
empty block, tag research block whose text is tags:: research
block $$ with p-choice:: sad, then block \frac{a}{b} one block: $$, tags:: research, p-choice:: sad, - $$, frac{a}{b}, $$

Markdown mirror, then the importer:

block before block after
text DONE a done task, empty text
text ## a heading 2
text [#A] **bold** tail. #[[Beta Gamma]] priority high, tag Beta Gamma, text **bold** tail.
text >{{query (task TODO)}}>中文 a quote block (task TODO)>中文, tag Quote
text :PROPERTIES: / :x: 1 / :END: property x = 1
text ~~s~~key:: v42 empty text
text id:: 6aa|a|b| empty text
text émoji😀 / - a:b '42 the second line becomes a child block

The EDN export and import, which serialize data instead of printing Markdown, found no mismatches on the same seeds.

Patches withdrawn

Each of these escaped or special-cased one collision. They are closed: fixing symptoms hides the design problem, and the fuzzers kept finding new collisions after each one.

  • #13721 export: keep backslash escapes
  • #13743 mirror: escape text a Markdown reader takes as structure
  • #13719 paste: Markdown quote lines as a quote block
  • #13720 export and paste: property, status, priority, tag and collapsed lines
  • #13730 export: text before an embed macro
  • #13732 export: macro syntax, embeds in property values
  • #13742 mirror: tags and status written next to a math fence
  • #13754 paste: keep #tag syntax

Related symptoms already filed: #13755, logseq/db-test#1439, logseq/db-test#1440.

The fix

Store a block's body as data, not as a Markdown string: text runs, emphasis, refs and tags by id, code and math as their own nodes. Keep :block/title a plain-text name. Then exporting is a printer over that data and importing is a parser into it, and the DB's own formats round-trip by serialization, as EDN already does.

主要言語
言語のデータがありません
スター
29
フォーク
2
30日以内にマージされた PR はありません

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

logseq/db-test のほかの issue

logseq/db-test の issue をすべて見る

似ている issue

Data Engineering の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。