[Feature]: Bounded MCP text reads with lossless, version-checked continuation

Read openviking/server/mcp_endpoint.py for the MCP read arguments, openviking/service/fs_service.py for visible-text reading and slicing, and openviking/utils/token_estimation.py for the existing estimator. Start by reviewing the MCP integration contract in docs/en/guides/06-mcp-integration.md. Done means bounded JSON pages reconstruct text exactly, reject changed versions, preserve authorization and legacy behavior, and pass the listed acceptance checks.

Issue Origin

Observed or reproduced in a real environment

Problem Statement

MCP clients need to read exact source text within a bounded tool-response size. The current read tool provides line-based offset/limit and reads the whole file by default, subject to the existing visible-read limits. It has no explicit per-call token budget for the returned text. A single Markdown table row or compact JSON line can still be very large when limit=1.

Two related requirements arise when the client reads the source in several calls:

1. Bound each response while allowing the client to recover the complete text, including a line larger than the budget.
2. Detect source changes between pages so that the client cannot silently combine a prefix from version A with a suffix from version B.

search(mode="context", max_tokens=...) already budgets assembled retrieval context. This request concerns exact source reads and their continuation, rather than another retrieval-context assembler.

Proposed Solution

Add an opt-in, bounded page mode for a single stored text file to MCP read. Keep existing calls and their return format unchanged when page mode is not requested. Names below are illustrative and open to maintainer feedback.

• An optional max_tokens enables the page response. Define its counting scope precisely: the serialized page returned to the model, including the URI, escaped content, version and continuation metadata. JSON-RPC transport framing and host-generated tool labels can remain outside that scope. If text and structured output repeat the payload, avoid the duplication or account for it explicitly.
• Reuse the existing token-estimation utility and document that the limit is measured by that estimator, not every model's tokenizer. No mandatory model-specific tokenizer dependency is needed.
• Return valid JSON in a single text block with content, has_more, next_cursor and version, without duplicating the page in structuredContent. Continue at Unicode code-point boundaries, including within an oversized line. Preserve the exact authorized text representation, including its line endings.
• The cursor binds the canonical URI, text position and content version to the relevant identity/scope. Repeating a valid cursor against unchanged content returns the same page. At EOF, has_more=false and next_cursor=null.
• On each continuation, recheck authorization and the content version. A changed file returns an existing precondition/conflict error with an instruction to restart; do not return a page from the new version. Deleted or unauthorized resources retain their normal error semantics.
• Calculate the version and page from one consistent authorized text representation, rather than checking file metadata and then reading potentially different bytes. This can be change detection; retaining a historical copy is not required.
• Keep existing visible-read, privacy and authorization limits in force. The response budget is an additional constraint, not a way to bypass them.
• If a valid minimum budget cannot fit metadata and any source text, return a clear error instead of an empty, non-advancing page or a silently increased budget.

For an initial implementation, reject ambiguous mixtures of cursor mode and legacy offset/limit, as well as budgeted multi-file reads. Images, audio and Live Context evaluation are outside this initial text-page contract. Ordinary legacy calls keep their existing behavior. The implementation must distinguish stored text from Live Context rather than accidentally execute a node or page through an evaluated result.

Illustrative JSON page (not a current upstream API):

{
  "uri": "viking://resources/example/large.md",
  "content": "First portion of the exact source text...",
  "version": "<opaque-content-version>",
  "has_more": true,
  "next_cursor": "<opaque-version-bound-cursor>"
}

Alternatives Considered

• Smaller line limits do not bound a very long line.
• Truncating a string or cutting serialized JSON loses content or breaks the response structure.
• Client-side truncation occurs after the large response has already crossed the tool boundary.
• Retrieval-context assembly or LLM summaries can be useful, but do not provide exact reconstruction of the original text.
• Persistent historical snapshots could support a stronger read contract, but are not necessary for a first version that detects changes and asks the client to restart.

Feature Area

Filesystem Operations

Use Case

Coding agents inspect long project documents, generated JSON and Markdown tables in several bounded calls. They can stop after obtaining the needed evidence, or concatenate all pages to reconstruct the source exactly. Concurrent document updates are reported rather than silently mixed into an answer.

Example API (Optional)

# Proposed API, not supported on current upstream main.
{"name":"read","arguments":{"uri":"viking://resources/example/large.md","max_tokens":2000}}

# Continue with next_cursor from the response:
{"name":"read","arguments":{"uri":"viking://resources/example/large.md","max_tokens":2000,"cursor":"<opaque-version-bound-cursor>"}}

Additional Context

Reproduction environment

The current behavior was reproduced through real MCP Streamable HTTP at commit 5caada112a60783d4ec21234be90e95f0261fbd5, using an isolated server, API-key authentication, native local storage and synthetic files. Python server code and the rebuilt RAGFS extension came from that commit; the C++ vector engine used the unmodified 0.5.0 release binary. Only models used the upstream API-test doubles. This is baseline integration evidence, not a complete fresh native build or upstream feature acceptance. No production service or project data was used.

Observed through the actual MCP client/server path:

• tools/list advertised read arguments uri, uris, offset and limit; no response-budget or content-version argument.
• Reading one synthetic line with limit=1 returned all 120,000 Unicode code points, estimated at 55,500 tokens by the upstream helper. The line limit was honored; it is not a response-token limit.
• Reading four pages of an unchanged CRLF document reconstructed it exactly. An empty file also returned correctly.
• After reading A-first\n, the fixture was replaced through the real content API. The next page returned B-middle\nB-last\n without a version conflict, producing text belonging to neither complete source version.
• Passing an unsupported max_tokens=2000 argument did not bound that long-line response in the tested runtime. It is not part of the current advertised API; this observation is not a claim that a supported token-budget contract was violated.

A local prototype based on b8e88959f3ab437b4d08514eceaa7221cca470dc now implements the proposed mode. 67 selected read-related tests passed, including the new page-mode cases and existing compatibility checks; this was not a full repository test run, and remote CI has not run. A real HTTP/MCP fixture verified exact reconstruction of the 120,000-code-point line in 32 pages at a 2,000-token estimated JSON budget, changed-version rejection, deletion errors, cross-identity cursor rejection, actual ACL revocation and Privacy-visible reconstruction. Model calls use upstream test doubles; native provenance has the same C++ limitation noted above. This is local validation, not an upstream release or maintainer approval. The interface is proposed for discussion before opening a code PR.

The prototype uses process-local signed cursors: they expire on restart and require the same worker. Each continuation still reads the visible full text under the existing size limit, so this bounds the tool response, not server I/O or memory. I would like maintainer feedback on the opt-in read API, JSON text page format, version source, and acceptable cursor lifetime before preparing the upstream code PR.

Minimal baseline reproduction with synthetic text:

line = '中文🙂 "quoted" \\ path ' * 6000
assert len(line) == 120000

1. Store that single line as a resource using the normal content-write API.
2. Call MCP read with that URI, offset=0, limit=1: the complete line is returned (55,500 estimated tokens in the tested baseline).
3. For the consistency case, store A-first\nA-middle\nA-last\n; read offset=0, limit=1.
4. Replace it with B-first\nB-middle\nB-last\n, then read offset=1, limit=-1: the combined pages mix versions without an error.

Reviewed upstream entrypoints:

• MCP read and its current arguments
• Filesystem visible-text reading and slicing
• Existing token estimation
• Existing MCP integration contract

Suggested acceptance checks:

1. Every successful page fits the documented estimator budget after serialization, including metadata and JSON escaping.
2. A long single line, CJK, emoji, quotes, backslashes and CRLF can be reconstructed exactly without missing or duplicated text.
3. Continuation advances until EOF; repeating a cursor against unchanged text is deterministic.
4. A changed content version is rejected before returning further text; deletion and permission changes retain their own errors.
5. A cursor cannot redirect the read to another URI or cross an authorization boundary.
6. Invalid cursors, unsupported argument combinations and insufficient budgets fail clearly.
7. Legacy text, batch, media and Live Context calls retain their behavior when page mode is absent.

A custom local adapter demonstrates bounded JSON responses, within-line continuation and content-fingerprint checks. Its synthetic checks are feasibility evidence only, separate from the baseline MCP observations above; neither establishes task accuracy/token savings or acceptance of an upstream feature implementation. Implementation should follow the upstream owning modules rather than copy the adapter wholesale. Listing and retrieval-tool budgets can be considered separately after the text-read contract is agreed.

Contribution readiness

I can contribute the tested prototype and regression cases after agreement on the interface and scope. The implementation and local self-review were AI-assisted; the human requester set the scope and required the real MCP validation. Upstream review is still needed.

Contribution

• I am willing to contribute to implementing this feature