embed: hardcoded 30-minute session maxDuration aborts large-corpus embeddings (store.js:1377)

Open Beginner friendly
#673 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
node.js, typescript

Research direction

Start at dist/store.js:1377 and inspect how qmd embed currently reads QMD_EMBED_CONTEXT_SIZE and QMD_EXPAND_CONTEXT_SIZE. Make the embedding session duration configurable while preserving the 30-minute default, then run qmd embed on a corpus that exceeds 30 minutes and verify it completes without session expiry.

Written by the indexing model from the issue text.

Description

Summary

The generateEmbeddings LLM session in dist/store.js:1377 has a hardcoded maxDuration: 30 * 60 * 1000 (30 minutes). On large corpora that take longer than 30 min to fully embed, the session aborts mid-run via abortController, removeIncompleteEmbeddings() purges partially-embedded documents, and the user is forced into many resume runs that may never converge.

Reproduction (qmd 2.5.1, Apple Silicon CPU)

Corpus: ~6,400 markdown docs across two collections (mostly Chinese, ~66,000 chunks total after re-chunk).
Model: Qwen3-Embedding-0.6B-Q8_0 via QMD_EMBED_MODEL.

Run Cmd Wall time Result needsEmbedding after
1 qmd embed -f 30:02 "Session expired — skipping 2047 remaining chunks". 8,382 chunks embedded. 5,594
2 qmd embed (resume) 30:02 "Session expired — skipping 1279 remaining chunks". 9,436 chunks embedded. 5,364
3 qmd embed after patching maxDuration to 4 * 60 * 60 * 1000 3h 3m Clean completion: "Embedded 48023 chunks from 5364 documents". No expiry. 0

Why resume doesn't converge: doc-level completion requires all chunks for a content hash to finish in a single session before removeIncompleteEmbeddings() keeps them. Each 30-min run re-chunks the not-fully-done set, persists ~8-9k chunks, then aborts — so vectors grow but the doc-level "done" set only advances by the docs that happened to fit entirely within a 30-min window. At the observed rate (~230-323 content hashes flipped to done per run), convergence on this corpus would take ~17-24 additional runs.

Memory note

The 2.1.0 fix (#500 / #393) holds. Memory stayed bounded at ~3 GB throughout the 3h+ patched run on Apple Silicon. The 30-min cap appears to be over-defensive given that the underlying memory leak is fixed.

Workaround (used in production)

Monkey-patch dist/store.js:1377:

// before
}, { maxDuration: 30 * 60 * 1000, name: 'generateEmbeddings' });

// after
}, { maxDuration: 4 * 60 * 60 * 1000, name: 'generateEmbeddings' });

Survives until next npm install -g @tobilu/qmd.

Proposed fix

Make it env-configurable, in the spirit of QMD_EMBED_CONTEXT_SIZE / QMD_EXPAND_CONTEXT_SIZE:

maxDuration: Number(process.env.QMD_EMBED_MAX_DURATION_MS ?? 30 * 60 * 1000)

Or add a --max-duration <ms> flag on qmd embed. The 30-min default is sensible as a safety net; large corpora need an escape hatch.

Environment
  • qmd 2.5.1 (installed via npm install -g @tobilu/qmd)
  • Node — system default via Homebrew
  • macOS, Apple Silicon
  • Embedding model: hf:Qwen/Qwen3-Embedding-0.6B-GGUF/Qwen3-Embedding-0.6B-Q8_0.gguf
  • Reranker: default qwen3-reranker-0.6b
Dominant language
TypeScript
Stars
29.9k
Forks
1.9k
Avg merge
10d 12h
Merged PRs (30d)
6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tobi/qmd

All issues in tobi/qmd

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.