[Phase 16] S10 — Data-layer fidelity, scale and privacy
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 10/100
調査の方向性
This is a multi-unit roadmap tracker, not a single fix. Start with docs/roadmap/phase-16-trust-surface-hardening.md (lines 583-611) and CLAUDE.md for the gauntlet. The public functions find_near_duplicates and find_near_duplicates_minhash in data_audit.all must keep their pair-list shape, so changes must be additive. Done means the listed exit criteria pass, each with its own red-then-green reproducer.
索引モデルが issue の本文から書いたものです。
説明
Roadmap · Phase 16, step S10 · size L · 7 unit(s) · planned in
docs/roadmap/phase-16-trust-surface-hardening.md:583-611
Summary
Two threads that share one report schema. The data layer reports a clean or complete result it did not compute — dropped Presidio rows, a no-op script-sanity check, silent decode replacement, a duplicate scan that cannot finish — and a declared data intent is silently not honoured (multimodal.text_column is dead; mix_ratio does not produce the requested blend). CORE-13 is the same layer's privacy contract: a synthetic empty-response WARNING logs raw prompt content.
Units
| Unit | Tracked by (October 2026 review) |
|---|---|
F-W20260729-TRUST-06 |
#348 · #547 · #601 · #610 · #652 |
F-W20260729-TRUST-09 |
#348 · #547 · #551 · #601 · #610 |
F-W20260729-TRUST-11 |
#348 · #547 · #601 · #610 |
F-W20260729-TRUST-12 |
#348 · #478 · #547 · #601 · #610 |
F-W20260729-CORE-08 |
#399 · #588 · #596 · #621 · #739 |
F-W20260729-CORE-11 |
#545 · #559 · #588 · #596 · #621 · #688 |
F-W20260729-CORE-13 |
#699 |
Plan
find_near_duplicatesandfind_near_duplicates_minhashare indata_audit.__all__, so the pair-list shape is public API and the fix must be additive.- Group on the exact fingerprint before LSH, so k identical rows contribute k(k-1)/2 arithmetically instead of materialising k(k-1)/2 tuples.
Exit criteria
- 5,000 identical records still yield exactly 12,497,500 pairs with subquadratic peak memory under
tracemalloc, and a property test holdscount_… == len(find_…)across random inputs, including the brute-force fallback. - An every-row Presidio failure yields
coverage=0, incomplete=true. -
tr/TR/Trnormalise byte-identically. - A single invalid byte in a long file reports
decode_replacements=1. - A
text_columnabsent from the dataset fails early. - Realised mix ratios and row counts appear in the result artefact.
- A
caplogtest proves known PII tokens appear in no log record, while the warning still identifies the prompt by index and length.
How to deliver
- One root-cause family, committed on its own; then an Opus review round and a Sonnet review round, each with verified findings fixed and committed before the next step begins.
- A reproducer that is red before the fix and green after; the full gauntlet from
CLAUDE.mdpasses under./.venv/bin/python. - Every English documentation edit ships with its Turkish mirror in the same commit; a new or renamed audit event gets its EN and TR catalog rows in the same PR.
- A step that grows a deferred module carries its
budget_historyjustification in the same diff; a widened guard rolls out non-strict first, then flips. - Commit bodies cite
Absorbs F-W20260729-…; the CHANGELOG[Unreleased]section gets an entry only for a change a user or library consumer can observe (decision C-20).
Planned in docs/roadmap/phase-16-trust-surface-hardening.md (read at f94595f). Unit IDs (F-W20260729-…) come from the 2026-07-29/30 full-project review; the issue numbers next to them are the October 2026 review's records of the same defects.
- 主要言語
- Python
- スター
- 9
- フォーク
- 1
- 平均マージ
- 8時間 10分
- マージ済み PR（30日）
- 5
環境構築
- Dockerfile または Docker Compose ファイルあり
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
HodeTech/ForgeLM のほかの issue
-
area: dev-tooling bug severity: low source: roadmap wave: 4
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
-
area: site bug good first issue severity: low source: review-2026-09 wave: 4
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
-
documentation good first issue severity: medium source: review-2026-09 wave: 3
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
-
documentation good first issue severity: low source: review-2026-09 wave: 4
難易度 1/5 1〜3時間 初心者へのやさしさ 80/100
-
documentation severity: medium source: review-2026-09 wave: 3
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
HodeTech/ForgeLM の issue をすべて見る
似ている issue
-
bug
難易度 1/5 1時間未満 初心者へのやさしさ 88/100
openwisp/openwisp-utils#777 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 63/100
google-research/weatherbenchX#310 ·
メンテナーはふだん 3 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
huggingface/peft#3955 ·
メンテナーはふだん 1 日以内に返信
-
author-page correction
難易度 2/5 1〜3時間 初心者へのやさしさ 60/100
acl-org/acl-anthology#10313 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
AOSSIE-Org/Gitcord-GithubDiscordBot#147 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信