Self-scan context pack: per-issue "read file:X-Y" pointers from the master DB for churn workers

Open
  • Is something wrong?
#5,521 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
44/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
github, python
Domain
devtools, tooling

Research direction

Start by reading tests/tools/self_scan.py, the self-scan orientation doc, and the master DB schema for file_data, function_data, class_data, fcall_data, and edge_data. Build the context pack around the issue text, then check its output and token cap against the listed fixture cases and five recent closed churn issues. Done means the tests pass, each pack stays within 3,000 tokens and names files changed by the merged PRs, runtime stays under 10 seconds, and the stated doc fixes are handled.

Written by the indexing model from the issue text.

Description

enhancement performance priority: low scope:infra

Why

A cost audit of nyx's churn workers (2026-10-18, 16 worker sessions, 10 merged PRs at ~$2/PR) found:

  • Workers get only the issue number and title. Each one finds the code itself, taking 5-14 grep/sed -n requests before its first edit. Orientation accounts for ~45% of worker spend, though most of that is the fixed ~40k-token starting context that every request re-reads.
  • Whole-file reads are the main trap. gitgalaxy/core/detector.py is ~135k tokens, and seven files are over 60k tokens each. The golden-master JSON fixtures run up to 606k lines.
  • The self-scan already has what is needed to point a worker straight at the code, but nothing turns it into a per-issue slice:
    • The master DB's function_data has start_line, loc, complexity, fan-in/fan-out and token_mass.
    • fcall_data holds the resolved call graph and edge_data the import edges.
    • file_data has token_mass, churn and blast radius.
    • The existing LLM brief (_llm.md) is ~137 KB (~34k tokens), too big to inject.

What

Add a tool, e.g. tests/tools/context_pack.py <issue> [--db docs/self_scan/gitgalaxy_master.db] [--max-tokens 3000]. Given an issue, it would:

  1. Pull out the paths, file names, function, class and symbol names, and backticked identifiers named in the issue body and title (gh issue view --json title,body).
  2. Resolve them against the master DB (file_data, function_data, class_data). Fuzzy-match bare names and rank matches by how often they are mentioned.
  3. For each hit, emit path:start-end (from start_line + loc), complexity and token size. Add one hop of callers and callees from fcall_data, each with its own path:start-end, and import fan-in from edge_data.
  4. Flag large files (token_mass > 8k, as dev_agent_firewall does) with "read only the named ranges".
  5. Print a Markdown block capped at about 1-3k tokens that a worker brief can include verbatim. If nothing resolves, print nothing; never pad.

Freshness: keep one DB per origin/main commit (tests/tools/self_scan.py already does delta mode, ~6-8 s; a full scan is ~1.5 min). Run the pack before the worker launches, so a stale DB is off by at most the commits since the last refresh. Line ranges are hints and the worker still reads the code.

Integration (separate PR in squid-protocol/promoter-bot): churn worker runs the tool and adds its output to prompt.md under "Where to look". That is a few lines once the tool exists.

Related cleanups found during the audit

  • CLAUDE.md:103 points agents at docs/gitgalaxy_architecture_brief.md. That file moved to the generated branch, so a local checkout has nothing there.
  • docs/agents/self-scan-orientation.md says the DB "has no line numbers". That is stale: function_data.start_line is populated.
  • Delta-mode briefs written into worker worktrees (e.g. churn-5464_galaxy_llm.md) report "0 analyzed artifact(s), 0 LOC". Either fix them or stop writing them.

Acceptance

  • context_pack.py produces a pack of 3k tokens or less for at least 5 recent closed churn issues, and for each one the pack names the file(s) the merged PR actually changed. Show the hit rate in the PR body. (#6354)
  • Unit tests on a small fixture DB: exact path, bare function name, class name, no matches (empty output) and the token cap. (#6145)
  • Runs in under 10 s against an existing DB. It needs no network access apart from gh issue view, and it accepts --body-file for offline use. (#6145)
  • The doc fixes above. (#6145)
  • Follow-up issue or PR filed on promoter-bot to wire it into churn worker.
Dominant language
Python
Stars
57
Forks
11
Avg merge
1h 35m
Merged PRs (30d)
975

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from squid-protocol/gitgalaxy

All issues in squid-protocol/gitgalaxy

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.