[Phase 16] S10 — Data-layer fidelity, scale and privacy

Offen
  • Stimmt etwas nicht?
#896 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
10/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
data, security

Rechercherichtung

This is a multi-unit roadmap tracker, not a single fix. Start with docs/roadmap/phase-16-trust-surface-hardening.md (lines 583-611) and CLAUDE.md for the gauntlet. The public functions find_near_duplicates and find_near_duplicates_minhash in data_audit.all must keep their pair-list shape, so changes must be additive. Done means the listed exit criteria pass, each with its own red-then-green reproducer.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

phase: 16 source: roadmap

Roadmap · Phase 16, step S10 · size L · 7 unit(s) · planned in docs/roadmap/phase-16-trust-surface-hardening.md:583-611

Summary

Two threads that share one report schema. The data layer reports a clean or complete result it did not compute — dropped Presidio rows, a no-op script-sanity check, silent decode replacement, a duplicate scan that cannot finish — and a declared data intent is silently not honoured (multimodal.text_column is dead; mix_ratio does not produce the requested blend). CORE-13 is the same layer's privacy contract: a synthetic empty-response WARNING logs raw prompt content.

Units

Unit Tracked by (October 2026 review)
F-W20260729-TRUST-06 #348 · #547 · #601 · #610 · #652
F-W20260729-TRUST-09 #348 · #547 · #551 · #601 · #610
F-W20260729-TRUST-11 #348 · #547 · #601 · #610
F-W20260729-TRUST-12 #348 · #478 · #547 · #601 · #610
F-W20260729-CORE-08 #399 · #588 · #596 · #621 · #739
F-W20260729-CORE-11 #545 · #559 · #588 · #596 · #621 · #688
F-W20260729-CORE-13 #699

Plan

  • find_near_duplicates and find_near_duplicates_minhash are in data_audit.__all__, so the pair-list shape is public API and the fix must be additive.
  • Group on the exact fingerprint before LSH, so k identical rows contribute k(k-1)/2 arithmetically instead of materialising k(k-1)/2 tuples.

Exit criteria

  • 5,000 identical records still yield exactly 12,497,500 pairs with subquadratic peak memory under tracemalloc, and a property test holds count_… == len(find_…) across random inputs, including the brute-force fallback.
  • An every-row Presidio failure yields coverage=0, incomplete=true.
  • tr / TR / Tr normalise byte-identically.
  • A single invalid byte in a long file reports decode_replacements=1.
  • A text_column absent from the dataset fails early.
  • Realised mix ratios and row counts appear in the result artefact.
  • A caplog test proves known PII tokens appear in no log record, while the warning still identifies the prompt by index and length.

How to deliver

  • One root-cause family, committed on its own; then an Opus review round and a Sonnet review round, each with verified findings fixed and committed before the next step begins.
  • A reproducer that is red before the fix and green after; the full gauntlet from CLAUDE.md passes under ./.venv/bin/python.
  • Every English documentation edit ships with its Turkish mirror in the same commit; a new or renamed audit event gets its EN and TR catalog rows in the same PR.
  • A step that grows a deferred module carries its budget_history justification in the same diff; a widened guard rolls out non-strict first, then flips.
  • Commit bodies cite Absorbs F-W20260729-…; the CHANGELOG [Unreleased] section gets an entry only for a change a user or library consumer can observe (decision C-20).

Planned in docs/roadmap/phase-16-trust-surface-hardening.md (read at f94595f). Unit IDs (F-W20260729-…) come from the 2026-07-29/30 full-project review; the issue numbers next to them are the October 2026 review's records of the same defects.

Vorherrschende Sprache
Python
Sterne
9
Forks
1
Ø Merge
8 Std. 20 Min.
Gemergte PRs (30 T.)
59

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus HodeTech/ForgeLM

Alle Issues in HodeTech/ForgeLM

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.