fix(detection): preserve source spelling in augmented and split-name spans

Open Beginner friendly
  • Is something wrong?
#327 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Read engine/detection/postprocess.py, focusing on apply_augmented_entities() and _split_full_names(), where the issue identifies the two span construction sites. Start by comparing their value handling with expand_entity_occurrences(), which already preserves source slices. Add regression coverage for the described case variants and name splits without inference; done means both source matches are replaced while matching rules, offsets, labels, provenance, and strict replacement checks remain unchanged.

Written by the indexing model from the issue text.

Description

Priority Level

Medium (has a local workaround)

Describe the bug

Augmented entities and split full-name parts use case-insensitive occurrence matching, but their EntitySpan.value retains the suggestion's spelling rather than the text at the matched offsets. The replacement primitive correctly requires exact agreement between the source slice and entity value, so it skips matches whose capitalization differs.

Steps/Code to reproduce bug

This example uses invented names and no model calls:

import pandas as pd

from anonymizer.config.replace_strategies import Redact
from anonymizer.engine.constants import (
    COL_FINAL_ENTITIES,
    COL_REPLACED_TEXT,
    COL_REPLACEMENT_APPLICATION,
    COL_TEXT,
)
from anonymizer.engine.detection.postprocess import apply_augmented_entities
from anonymizer.engine.replace.strategies import apply_local_replace_strategy

text = "Alice met ALICE."
entities = apply_augmented_entities(
    text=text,
    entities=[],
    augmented_output={"entities": [{"value": "Alice", "label": "first_name"}]},
)
result = apply_local_replace_strategy(
    pd.DataFrame({
        COL_TEXT: [text],
        COL_FINAL_ENTITIES: [{"entities": [entity.as_dict() for entity in entities]}],
    }),
    strategy=Redact(),
)
print(result[COL_REPLACED_TEXT].iloc[0])
print(result[COL_REPLACEMENT_APPLICATION].iloc[0])

Current output is [REDACTED_FIRST_NAME] met ALICE. with one applied span and one skipped span. _split_full_names() has the same problem for standalone variants such as JOHN after a John Smith full-name detection.

Expected behavior

Each matched entity should use text[start:end] as its value, preserving the exact source spelling. Both matches should be replaced. Matching rules, offsets, labels, provenance, and strict replacement checks should stay the same.

Additional context

Observed on main at 1872309. The detector parser and expand_entity_occurrences() already preserve source slices. The proposed correction applies the same rule to the two remaining construction sites in engine/detection/postprocess.py. Regression coverage can exercise case variants, Unicode spelling, and first/middle/last-name splits without inference.

Dominant language
Python
Stars
136
Forks
22
Avg merge
1d 22h
Merged PRs (30d)
47

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-NeMo/Anonymizer

All issues in NVIDIA-NeMo/Anonymizer

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.