extract --out <dir>: semantic cache never saved, docs missing from manifest (relative source_file resolved against --out)

Open Beginner friendly
  • Is something wrong?
#4,831 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Deepjyoti-Sarmah is already working on this.

Since Oct 28, 2026.

  • #4850 by @Deepjyoti-Sarmah — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Start in __main__.py at extract, then inspect cache.save_semantic_cache and check_semantic_cache to trace how source_file paths are resolved. Compare the paths used for _sem_extracted and files_by_type when --out differs from the scanned directory. Done means semantic cache entries are saved and the manifest records their semantic_hash, so a second run reuses the files.

Written by the indexing model from the issue text.

Description

Version: graphify 0.9.7 (uv tool install graphifyy), macOS, --backend claude-cli

Problem
graphify extract <dir> --out <other> never reuses the semantic cache. Every run re-extracts every doc and reports 0 files cached/unchanged. <other>/graphify-out/cache/semantic/ stays empty.

Cause
The LLM returns source_file relative to the scanned dir (e.g. n1.md). In __main__.py (extract), two places treat it as a path under the output root (out_root) instead:

  1. cache.save_semantic_cache(..., root=out_root) builds Path(root) / "n1.md", is_file() is false, so the entry is skipped. Nothing is saved.
  2. _sem_extracted (used to build _manifest_files) is compared against the absolute paths in files_by_type. A relative source_file never matches, so no doc gets a semantic_hash in manifest.json and detect_incremental re-queues all of them.

When --out is omitted (out_root == target), (1) resolves correctly.

Repro

mkdir vault out cwd
for i in 1 2 3; do echo "# Note $i. Alice uses Redis." > vault/n$i.md; done
cd cwd
graphify extract ../vault --out ../out --backend claude-cli --max-concurrency 1   # run twice

Run 2: incremental summary: 0 files cached/unchanged, 3 re-extracted, tokens spent again, out/graphify-out/cache/semantic/ empty, manifest.json is {}.

Workaround
Omit --out and set GRAPHIFY_OUT=/abs/path/graphify-out. Run 2 then reports 3 files cached/unchanged, 0 re-extracted and spends no tokens. On a ~106-doc vault this took a nightly run from ~340k in / ~66k out tokens to ~17k / ~7k.

Suggested fix
Resolve relative source_file against target (the scanned dir) in save_semantic_cache and when building _sem_extracted, while keeping the cache key relative to out_root as check_semantic_cache already does.

Dominant language
Python
Stars
124k
Forks
11.9k
Avg merge
2h 35m
Merged PRs (30d)
6

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Graphify-Labs/graphify

All issues in Graphify-Labs/graphify

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.