Issues pipeline chunk_size (1500) exceeds the TEI embedding limit (1000) — tail of long chunks is unsearchable

Open Beginner friendly
#216 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
data-engineering, search

Research direction

Start in docs-agent-mcp/pipelines/issues-pipeline.py at lines 301 and 481, tracing how chunk_size and max_tei_chars flow through ingestion. Verify that the resulting chunk content and TEI input cover the same text, then run the pipeline's available validation or ingestion checks to confirm long issue chunks are fully searchable.

Written by the indexing model from the issue text.

Description

Problem

The GitHub issues ingestion pipeline chunks issue content up to 1500 characters, but truncates every chunk to 1000 characters before sending it to the TEI embeddings service:

  • docs-agent-mcp/pipelines/issues-pipeline.py#L481 — chunk_size: int = 1500
  • docs-agent-mcp/pipelines/issues-pipeline.py#L301 — max_tei_chars = 1000, applied as r["content_text"][:max_tei_chars]

So for any chunk longer than 1000 chars, the last ~500 characters are stored in Milvus and shown to users, but were never embedded — that text is invisible to semantic search.

Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from kubeflow/docs-agent

All issues in kubeflow/docs-agent

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.