epic: Declarative Link-Driven Knowledge Base & Granular Incremental Indexing

Open
#700 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@DerekRoberts is already working on this.

Since Sep 21, 2026.

Assessment

This issue has not been assessed yet.

Description

enhancement

Overview

Transition the Agreement Navigator knowledge base from a repository of committed static Markdown files and binary PDFs to a declarative link-driven library (sources.yaml) with granular per-document caching and incremental FAISS re-indexing.

Core Invariants & Goals

  • Gradual & Safe: No committed files deleted without proven 100% substantive parity via shadow audits.
  • Incremental Rebuilds: Document modifications re-embed only the changed files, keeping FAISS re-indexing under <50ms.
  • Network Efficiency: Conditional HTTP requests (ETag / 304 Not Modified) prevent unnecessary downloads during scheduled drift runs.

Child Issues

  • #697: feat(indexing): Implement per-document chunk and embedding caching (.npy) for incremental FAISS builds
  • #698: feat(ingestion): Support conditional HTTP headers (ETag / If-Modified-Since) and 304 skipping in sync_sources.py
  • #699: feat(ingestion): Add PDF extraction strategy and shadow-audit multi-part BCLaws statutes into sources.yaml

Architectural Contract

Documented in app/SPEC.md (Section 6: Forensic Markdown Pipeline & Section 12: Future Roadmap).

Dominant language
Python
Stars
2
Forks
1
Avg merge
4h 16m
Merged PRs (30d)
38

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from MinionTech/vexilon

All issues in MinionTech/vexilon

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.