yoshuawuyts/components

[component] chunker — Text → chunks for retrieval

Open

#7 opened on 2026年5月1日

GitHub で見る
 (0 comments) (0 reactions) (0 assignees)Rust (1 fork)github user discovery
good first issue

Repository metrics

Stars
 (1 star)
PR merge metrics
 (PR metrics pending)

説明

Goal

Component to split Markdown/semantic text into offset chunks for chunked retrieval (RAG). Might use headings/paragraphs as boundaries or basic token counts with overlaps.

Scope/Features

  • Input: string
  • Output: structured array of chunks (text, offset/indices)
  • Configurable: chunk size, overlap, min paragraph, etc.

Rough WIT shape

  • split(input: string, params: chunk_params) -> list<chunk>
  • Chunk struct: {text, start, end, ...}

Labels

  • good first issue

コントリビューターガイド