Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Require source-grounded positive identity evidence before automatically merging namesakes

Open
  • Is something wrong?
#1,193 7 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@virtue039 is already working on this.

Since Oct 22, 2026.

  • #1377 by @virtue039 — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
postgresql, rust
Domain
ai, databases, testing

Research direction

Start by reading the existing CI test setup and the resolution and adjudication paths referenced in #1136; the payload does not name specific files. Add a fixed namesake case set with scripted model replies, including the failure paths and true-merge controls. Run the strict gold check and the separate known-failure budget check; done means CI rejects new or worse failures while reporting existing gold failures without presenting a budget pass as identity correctness.

Written by the indexing model from the issue text.

Description

question

Current status (2026-10-26): decision-path measurement ready; production evidence policy remains open

#1251 is merged and covers the resolution-side passage-over-profile defect. The remaining adjudication-side path in #1136 is measured by #1377, now ready for review after syncing current dev. Its 48 scripted probes check actual A/B evidence delivery, investigation/source reads and graph/review outcomes; green CI does not mean identity gold errors are fixed. The original eight inputs/gold and named-failure rules remain intact.

The current measurement and next-policy question proposes a limited existing-field source-input/second-look experiment for automatic exact-name same, including ordinary cache and governance paths. Whether that experiment and its possible added human/call cost are acceptable is awaiting maintainer confirmation. No production gate is implemented by #1377, and no accuracy gain is claimed from the earlier F1 deltas.

#1199 stays closed on hold. Its uniqueness gate, kept behavior and old model results below are historical review material, not the current agreed implementation contract. Any additional persisted/public contract would be discussed and recorded before implementation.

Historical first slice (held)

Implemented in https://github.com/deeplethe/utopia/pull/1199, independently of #1192, following the maintainer's scope agreement.

  • Only automatic pairs with byte-identical canonical names.
  • Current direct facts on both records must establish an existing, explicitly eternal uniqueness relation; each non-empty quote must occur in its current matching source passage. Name, employer/title/topic overlap and noncontradiction alone do not authorize merging.
  • Missing checkable identity evidence closes the pair as kept with a reason, without another model request or a new human review. Cached same verdicts recheck current sources.
  • Existing fields only; no schema/API/ontology-field changes. Human decisions and existing execution/rollback safeguards remain authoritative. The kept reason preserves an unresolved-left role when the original human namesake review carried it; #1192 consumes that existing-field marker so later mentions cannot reopen human work or grow another bucket.

Completed checks and evaluation

Candidate 883e440, dev baseline da2c45e: DB/PDF guards enabled, workspace 1501 passed / 0 failed / 6 existing ignored; focused suite 7 passed. fmt, clippy, frozen-lockfile web build and diff checks passed. Controlled true-identity fixtures preserve two qualified merges and intentionally lose one employer-only true merge, with zero human reviews added.

Independent official identity bench: qwen3.7-flash-2026-07-15, thinking disabled, pinned local BGE-M3; two rounds, eight fresh KBs per side, all 16 complete, no gateway errors. Mean F1 0.5246 → 0.5285; pending human reviews 0.500 → 0.125.

The gate withholds 3 observed true same-entity merge pairs established by singleton official gold groups on both current records. The PR lists those pairs and all per-KB F1/entity/human/false-merge/false-split/missing counts. False-merge and false-split means increased in this small experiment; this is a narrow evidence gate, not a demonstrated end-to-end quality gain. These measurements and withheld pairs are descriptive, not a causal estimate from F1.

Combined with #1192, a deterministic DB regression verifies 20 further ambiguous mentions after machine-kept reviews: one unresolved entity and zero reopened human reviews. The two live-model experiments are independent and do not measure the combined change.

Original proposal and historical DeepSeek investigation, preserved for context

What I am trying to do

Follow-up to #1136, separate from the mention-resolution change in #1192. I would like namesake adjudication to merge only when there is inspectable evidence connecting the two records, while preserving genuine aliases and full-name/short-name matches.

The original #1136 report includes a bare-name merge at 0.85 justified by different achievements not contradicting one another. Absence of contradiction does not establish identity. The current second-look mechanism can inspect facts and passages, but a model's same verdict and confidence are not themselves a checkable identity link.

This is a proposal to discuss the smallest adjudication change, not a claim that every namesake merge is unsafe or that an implementation has already been evaluated.

Evidence and limits

We measured e9057ac and the resolution-only candidate d8e3cf8 in the same session on 2026-10-11: two rounds, eight fresh identity-bench bases per side, DeepSeek deepseek-flash with thinking enabled/high, and pinned local BGE-M3 on both sides. All document jobs completed.

Four candidate reverse-order bases still contain mixed finance/engineering facts. These should not be attributed to bad adjudication merely because a merge exists. In three, the second look correctly connects the d04 budget mention to the finance director using its explicit title. Those merges (0.86 / 0.90 / 0.90) happen before d03, the engineering document, is ingested; subsequent resolution attaches engineering facts to the survivor. The fourth has no namesake merge. This ordering has been checked, and #1192 documents the remaining resolution limitation.

Separately, byte-identical adjudication requests yielded a changed i/verdict signature in one of nine repeated request groups, and changed confidence-threshold bands in three. Those groups are a variance diagnostic, not proof of a namesake-specific failure or of this proposal's effectiveness. Per-base results and the scope of the resolution experiment are in #1192.

What would help

A small gate before automatic namesake merges:

  • Require the decision to identify positive evidence connecting these particular records, using the existing facts/passages/decision trace where possible.
  • Check that cited evidence exists, is still applicable, and belongs to the current pair. Same name, topic overlap, a shared employer/project, or merely non-conflicting facts should not satisfy the gate by themselves.
  • If a link is missing or cannot be checked, retain the separation or use the existing human-review path, with an explicit reason.
  • Apply the requirement to both batch/second-look results and cached same results, so a cache hit cannot bypass the gate.

The evidence categories need agreement first. An explicit source statement of co-reference or a stable unique identifier may be strong evidence; matching department/job title alone can still fit more than one person. Checking provenance and references also does not prove the model's semantic interpretation is correct.

I would start by reusing the existing execution safeguards and decision trace rather than adding another general-purpose model verifier. This should be a separate PR from #1192, not a recall redesign, threshold increase, or rewrite of the identity architecture in #725. If additional persisted fields, ontology/public contracts are necessary, I would discuss the contract and ADR before writing that part.

Acceptance and evaluation

  • A high-confidence same based only on identical names and absence of contradiction cannot automatically merge.
  • A supported true same-entity example still merges; missing, stale or unrelated evidence cannot silently authorize a merge.
  • Cached decisions obey the same requirement.
  • Existing contradiction, human-decision and rollback safeguards remain effective.
  • Add minimal regressions that fail without the change and run in the existing CI suite.
  • Evaluate before/after against a same-session baseline, several fresh bases and at least two rounds per side, as requested in #1136. Report missing/type-unbound anchors, false merges, false splits, human-review load and model usage alongside F1; do not treat a single day's count as a causal effect.

What I do today instead

The resolution guard in #1192 prevents profile scores or fallbacks from guessing between exact-name candidates when passage corroboration cannot distinguish them. That does not establish an evidence contract for adjudication.

I would be happy to take the smallest follow-up once the evidence boundary and scope are agreed. Would this be a useful first adjudication step, or is there an existing identity-evidence cut that this should join?

Dominant language
Rust
Stars
8.1k
Forks
1.1k
Avg merge
21h 30m
Merged PRs (30d)
251

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deeplethe/utopia

All issues in deeplethe/utopia

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.