Better default minisearch tokenizer for Chinese documents

Description

no

Contributor guide

Tech stack

typescriptnodejsvue

Domain

searchdocumentationfrontend

Issue type

feature

DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.

3

Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.

1-3 hours

Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.

fresh

ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.

clear

Prerequisites

Basic knowledge of full text searchUnderstanding of tokenization

Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.

40

Research direction

Investigate the current minisearch tokenizer configuration in VitePress (likely in src/node/search.ts or similar). The issue is about improving the default tokenizer to handle Chinese text. Research possible integrations with Chinese segmentation libraries like 'jieba' or 'nodejieba'. Consider making the tokenizer configurable or adding a custom default that uses a simple unigram or bigram approach for CJK characters. Look at existing discussions in the issue comments for specific suggestions.