embed: hardcoded 30-minute session maxDuration aborts large-corpus embeddings (store.js:1377)
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- node.js, typescript
- Domain
- cli, machine-learning, performance
Research direction
Start at dist/store.js:1377 and inspect how qmd embed currently reads QMD_EMBED_CONTEXT_SIZE and QMD_EXPAND_CONTEXT_SIZE. Make the embedding session duration configurable while preserving the 30-minute default, then run qmd embed on a corpus that exceeds 30 minutes and verify it completes without session expiry.
Written by the indexing model from the issue text.
Description
Summary
The generateEmbeddings LLM session in dist/store.js:1377 has a hardcoded maxDuration: 30 * 60 * 1000 (30 minutes). On large corpora that take longer than 30 min to fully embed, the session aborts mid-run via abortController, removeIncompleteEmbeddings() purges partially-embedded documents, and the user is forced into many resume runs that may never converge.
Reproduction (qmd 2.5.1, Apple Silicon CPU)
Corpus: ~6,400 markdown docs across two collections (mostly Chinese, ~66,000 chunks total after re-chunk).
Model: Qwen3-Embedding-0.6B-Q8_0 via QMD_EMBED_MODEL.
| Run | Cmd | Wall time | Result | needsEmbedding after |
|---|---|---|---|---|
| 1 | qmd embed -f |
30:02 | "Session expired — skipping 2047 remaining chunks". 8,382 chunks embedded. | 5,594 |
| 2 | qmd embed (resume) |
30:02 | "Session expired — skipping 1279 remaining chunks". 9,436 chunks embedded. | 5,364 |
| 3 | qmd embed after patching maxDuration to 4 * 60 * 60 * 1000 |
3h 3m | Clean completion: "Embedded 48023 chunks from 5364 documents". No expiry. | 0 |
Why resume doesn't converge: doc-level completion requires all chunks for a content hash to finish in a single session before removeIncompleteEmbeddings() keeps them. Each 30-min run re-chunks the not-fully-done set, persists ~8-9k chunks, then aborts — so vectors grow but the doc-level "done" set only advances by the docs that happened to fit entirely within a 30-min window. At the observed rate (~230-323 content hashes flipped to done per run), convergence on this corpus would take ~17-24 additional runs.
Memory note
The 2.1.0 fix (#500 / #393) holds. Memory stayed bounded at ~3 GB throughout the 3h+ patched run on Apple Silicon. The 30-min cap appears to be over-defensive given that the underlying memory leak is fixed.
Workaround (used in production)
Monkey-patch dist/store.js:1377:
// before
}, { maxDuration: 30 * 60 * 1000, name: 'generateEmbeddings' });
// after
}, { maxDuration: 4 * 60 * 60 * 1000, name: 'generateEmbeddings' });
Survives until next npm install -g @tobilu/qmd.
Proposed fix
Make it env-configurable, in the spirit of QMD_EMBED_CONTEXT_SIZE / QMD_EXPAND_CONTEXT_SIZE:
maxDuration: Number(process.env.QMD_EMBED_MAX_DURATION_MS ?? 30 * 60 * 1000)
Or add a --max-duration <ms> flag on qmd embed. The 30-min default is sensible as a safety net; large corpora need an escape hatch.
Environment
- qmd 2.5.1 (installed via
npm install -g @tobilu/qmd) - Node — system default via Homebrew
- macOS, Apple Silicon
- Embedding model:
hf:Qwen/Qwen3-Embedding-0.6B-GGUF/Qwen3-Embedding-0.6B-Q8_0.gguf - Reranker: default
qwen3-reranker-0.6b
- Dominant language
- TypeScript
- Stars
- 29.9k
- Forks
- 1.9k
- Avg merge
- 10d 12h
- Merged PRs (30d)
- 6
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from tobi/qmd
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Similar issues
-
calcite-components needs triage refactor
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Esri/calcite-design-system#15203 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 91/100
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100