Bundle provider: cut the per-file metadata calls of the filesystem walk (Windows ×2.3 on flat bundles)

Open
#208 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
43/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp

Research direction

Read CLAUDE.md and inspect Walk, the ReparsePoints seam, Enumerate, GetVersion, and Read; the issue compares three optimization options but does not identify their file paths. Start by reproducing the flat bundle measurements on Linux and Windows, then trace metadata calls per file. Done means implementing and validating a chosen approach that reduces those calls while preserving the stated fail-closed behavior and version token semantics.

Written by the indexing model from the issue text.

Description

Follow-up of lot A (#139, PR #207), after the per-scope directory memo (#205).

Measured (independent differential, median of 3 runs, origin/dev 32a48bc → branch):

Bundle Verb Linux Windows
3 000 flat concepts validate 188 → 250 ms 501 → 1 142 ms
3 000 flat concepts info 117 → 182 ms 449 → 1 052 ms
acme_retail validate 71 → 94 ms 114 → 138 ms

Cost is linear, with a constant of roughly ten metadata calls per file: Enumerate and GetVersion each walk the tree (per entry: ReparsePoints.Inspect = attributes + link target, Directory.Exists/File.Exists, a FileInfo for the length), then every Read re-inspects the leaf (STRICT) before opening it.

Options

  1. In Walk, take the reparse flag and the file/directory kind from the FileSystemInfo entries the enumeration already returns (Attributes, LinkTarget are populated by the enumerator without extra syscalls on Windows) instead of re-inspecting each entry. Touches the ReparsePoints seam and the rule that the walk is one of the four lenient sites (CLAUDE.md): needs its own argument that an entry whose attributes could not be read is still treated fail-closed where a guard needs it.
  2. Make GetVersion reuse the listing of Enumerate within one scope (lengths and mtimes are already read by the walk), halving the walks per load. Changes what the weak version token covers only if the two walks no longer observe the same moment — which is arguably more correct.
  3. Skip re-inspecting the leaf in Read when the same scope already inspected it during the walk (the window grows from one call to one scope, as for the name memo).

Not in scope: the exact-name memo (done, #205).

Dominant language
C#
Stars
4
Forks
1
Avg merge
1d 4h
Merged PRs (30d)
32

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from jchable/okf4net

All issues in jchable/okf4net

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.