Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[BUG] rag: sentence content can cross the reported sentence boundaries

Open Beginner friendly
  • Is something wrong?
#1,469 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@LJK-ikun is already working on this.

Since Oct 21, 2026.

  • #1470 by @LJK-ikun — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
75/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
nodejs, typescript
Domain
backend, data

Research direction

Start by locating SentenceChunker and reading how it turns sentence match ranges into chunk content. Reproduce the reported boundary mismatch with the default tokenizer, then add regression coverage for that example and sentence overlap. Done means content matches each reported source range while sentence grouping, token accounting, and overlap behavior remain unchanged.

Written by the indexing model from the issue text.

Description

bug
Describe the bug

SentenceChunker computes character ranges from sentence matches, but constructs content using token counts of independently encoded prefixes as indexes into the full document's BPE tokens. Prefix tokenization is not additive: a token can extend beyond a sentence boundary. Returned content can include part of the next sentence and omit part of its own reported source range. This affects sentence-level context grouping and consumers that use the returned range to highlight retrieved content.

Steps To Reproduce

On current main 72a46c76b4507e1f98b41d5050a92bd30587eb70, using the real default tokenizer:

import { SentenceChunker } from "@voltagent/rag";
const text = "foo.bar baz.qux end.";
const chunks = new SentenceChunker().chunk(text, { maxTokens: 3 });
console.log(chunks.map(c => ({
  content: c.content,
  source: text.slice(c.start, c.end),
})));

Actual pairs are foo.bar / foo., baz.qu / bar baz., and x end. / qux end.. A second reproduction, First sentence.\n\nSecond sentence! Last one? at the same budget, puts the two newlines in the first chunk's content while its range excludes them; the next range includes the newlines while its content excludes them.

Expected behavior

Sentence chunk content should match the already-computed sentence character range. It should not cut into an adjacent sentence because an independently encoded prefix has a different token boundary. Existing sentence grouping, token accounting and overlap options should stay unchanged.

Packages

@voltagent/rag 1.0.2 source at the baseline above; Node.js 24.19.0 on Windows, js-tiktoken 1.0.21 (within the declared range). Reproduced through exported public classes, without a model/provider or patched collaborators.

Additional Context

I plan to work on a scoped fix that slices the original text by the computed sentence range, with regressions using the real default BPE tokenizer and sentence overlap. This follows the contribution guide's request to mention work in an issue before implementation. This is independent of #1466 (recursive IDs) and #1468 (table delegation offsets); neither of those changes SentenceChunker's content construction. Exact SentenceChunker/prefix/boundary searches and the current open-PR list found no active fix for this defect. No downstream production incident is claimed.

AI assistance: Codex performed the source inspection and local reproduction under LJK-ikun's direction. Automated verification is distinguished from human review.

Dominant language
TypeScript
Stars
10.9k
Forks
1.3k
Avg merge
4h 57m
Merged PRs (30d)
2

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from VoltAgent/voltagent

All issues in VoltAgent/voltagent

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.