Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Problems and future plans on token system

未关闭
  • 哪里不对？
#75 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
20/100
Issue 类型
重构
描述清晰度
需要澄清
活跃度
停滞
技术栈
react, typescript
领域
frontend, performance

调研方向

Start with the linked token system documentation and the mentioned files: src/tokenize/utils.js, normalizeToLines.js, pickRanges.js, and backToTree.js. Review the proposed clone, path, and node-shape changes and their breaking-change implications; this issue is complete only after a concrete migration plan and agreed implementation scope are defined.

由索引模型根据 Issue 内容生成。

描述

pinned

Token system is a key feature of react-diff-view and is the source of its powerfulness.

I looked into current implementation of token system recently and discovered some vulnerabilities about it, mostly about performance.

Too many clones

To ensure token system works safely by default, we introduce a clone function to copy an entire path, that is clone every node inside this path. This function is used in almost every key functions like normalizeToLines and other manipulation utilities.

Actually, while we guarantee that token node is immutable, we can get rid of all these clones, boosting performance considerably.

In current situation we don't have a statement saying "you cannot mutate token node" and removing clone operations can introduce some potential breaking changes, but I think it should be safe in most cases.

Too many slices

Previously we represent a path as an array of token nodes, the last node in this array is always a text node:

[
  {type: 'element'},
  {type: 'element'},
  {type: 'mark'},
  {type: 'text', value: 'Hello World'}
]

In nearly all cases when we need to manipulate this path, we were not changing the text, we frequently insert a node before text node, the implement could be:

const parents = path.slice(0, -1);
const text = path[path.length - 1];
const newPath = [...parents, customNode, text];

We notice a .slice(0, -1) inside this code which can cause some dropdown on performance.

To optimize this, we can separate node path and its containing text into a tuple like:

[
  [
    {type: 'element'},
    {type: 'element'},
    {type: 'mark'},
  ],
  'Hello World'
]

We can insert a node before text more efficiently:

const [parents, text] = path;
const newPath = [[...parents, customNode], text];

In this solution we also unwrap text node into a single string to reduce the creation and garbage collection of node objects.

This is a breaking change since any custom enhancers written previously will fail to work on the new data structure.

Too dynamic nodes

For the lifetime of react-diff-view, we didn't have a strictly defined type for token node, that is, users can put any properties into a node and use them in custom renderToken function.

This is easy to use, but a dynamic node causes more than expected node clones and a poor performance node merge algorithm.

We decide to later define the node more strictly, leaving only type and a dynamic properties for it, the clone and compare can be much simpler:

const cloneNode = node => {
  return {
    type: node.type,
    properties: node.properties,
  }
};

const areNodesEquals = (x, y) => (x.type === y.type && shallowEquals(x.properties, y.properties));

This is also a breaking change since a more strict structure is forced.

Reuse token system

As a result of performance inspection and reusability consideration, I developed a tiny library source-tokenizer to handle these, on the initial implement we get about 25% performance boost:

highlighting a json file with 11029 lines, 10 times
legacy: 22.384s
current: 16.038s

I'm preparing to develop a react-source-view component based on this library to render source codes, react-diff-view which focused on render diff contents will have a major version to use this library in 2020, an upgrade guide will be published then.

主要语言
TypeScript
星标
1k
派生
88
30 天内没有已合并 PR

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

otakustay/react-diff-view 的其他 Issue

查看 otakustay/react-diff-view 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。