BUG PlagiarismScorer tokenizer strips combining marks and skips normalization, so a verbatim copy can score 0.0 and different words can score 1.0

Geschlossen
  • Stimmt etwas nicht?
#3,052 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

Ein zugehöriger Pull Request wurde bereits gemerged.

  • #3053 von @inchang-ing — gemerged

Bewertung

Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Anfängerfreundlichkeit
25/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
ai, internationalization

Rechercherichtung

The bug is in PlagiarismScorer._tokenize in pyrit/score/float_scale/plagiarism_scorer.py, around line 73, where the regex strips non-word characters before any Unicode normalisation. Before reading the fix, decide between NFKC and NFC with the maintainers, since the issue leaves that open. Pull request #3053 is already open against this issue, so check it first. Done means the NFC/NFD and fullwidth examples score 1.0 against the reference, the Devanagari pair no longer collides, and the existing tests in tests/unit/score still pass.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Describe the bug

PlagiarismScorer._tokenize (pyrit/score/float_scale/plagiarism_scorer.py:73) lowercases, then strips everything that is neither \w nor \s:

text = re.sub(r"[^\w\s]", "", text)

Python's \w does not include combining marks (category M), and no normalisation happens first. That makes the scorer wrong in both directions.

False negatives. Text that reads identically to the reference scores as unrelated, because the bytes differ in a way the tokenizer does not fold. Verbatim reproductions of the reference:

response lcs lev jaccard
verbatim 1.000 1.000 1.000
same text, NFD form 0.545 0.545 0.000
same text, fullwidth 0.000 0.000 0.000
same text, math-bold 0.000 0.000 0.000

NFD affects accented text: café decomposes to cafe + U+0301, the mark is stripped as non-\w, and the token becomes cafe while the NFC reference keeps café. Fullwidth and math-alphanumeric are not encoding accidents but the ordinary homoglyph substitutions, and they take every metric to zero.

False positives. Because marks are stripped rather than normalised, scripts that carry meaning in combining marks collapse into each other. दिन ("day") and दीन ("poor") are different Hindi words; both tokenize to दन:

PlagiarismScorer(reference_text="दिन", metric=PlagiarismMetric.LCS)
  ._plagiarism_score("दीन", "दिन", metric=PlagiarismMetric.LCS)
# 1.0

Devanagari, Thai and Tamil are mangled wholesale: सिस्टम → ससटम, สวัสดี → สวสด, வணக்கம் → வணககம.

This is distinct from #2971 / #2972, which cover n-gram size and blank reference validation.

Steps/Code to Reproduce
import unicodedata
from pyrit.score.float_scale.plagiarism_scorer import PlagiarismScorer, PlagiarismMetric

REF = ("Il était une fois une très jeune fille naïve qui habitait près "
       "d'un café à Genève où l'on servait des crèmes brûlées")

for metric in PlagiarismMetric:
    s = PlagiarismScorer(reference_text=REF, metric=metric)
    print(metric.value,
          s._plagiarism_score(REF, REF, metric=metric, n=5),
          s._plagiarism_score(unicodedata.normalize("NFD", REF), REF, metric=metric, n=5))

ASCII = "the quick brown fox jumps over the lazy dog"
fullwidth = "".join(chr(ord(c) - 0x20 + 0xFF00) if 0x21 <= ord(c) <= 0x7e
                    else ("　" if c == " " else c) for c in ASCII)
s = PlagiarismScorer(reference_text=ASCII, metric=PlagiarismMetric.JACCARD)
print(s._plagiarism_score(fullwidth, ASCII, metric=PlagiarismMetric.JACCARD, n=5))
Expected Results

A response that a reader would call a verbatim copy scores as one, and two different words do not score as identical.

Actual Results
lcs         1.0 0.5454545454545454
levenshtein 1.0 0.5454545454545454
jaccard     1.0 0.0
0.0          <- fullwidth
Suggested fix

Normalise before tokenizing, and keep combining marks rather than discarding them:

text = unicodedata.normalize("NFKC", text).lower()
text = "".join(
    c for c in text
    if c.isspace() or c.isalnum() or c == "_" or unicodedata.category(c).startswith("M")
)
return text.split()

I validated this on four axes: NFC and NFD forms now tokenize identically (0 disagreements over accented Latin, Greek and Vietnamese); the Hindi, Thai and Tamil pairs above no longer collide; all 23 pure-ASCII inputs in my corpus tokenize exactly as before; and no input in a 334-case corpus (20 scripts × NFC/NFD × 8 whitespace shapes, plus edge cases) raises.

With it applied, all four rows of the first table read 1.000 and the Hindi false positive goes to 0.0. tests/unit/score and tests/unit/converter are 5316 passed, 108 skipped.

One choice is yours rather than mine: NFKC or NFC. NFC fixes the NFD row only. NFKC additionally folds the homoglyph rows, which is why I used it, but it is lossier — ½ → 12, Ⅻ → xii, fi → fi, x² → x2. For a scorer whose job is detecting reproduction I think that trade is right, since those are evasion vectors too, but ½ → 12 is a real wart and you may prefer NFC plus an explicit confusable-folding step.

Happy to open the PR with whichever you pick, with regression tests for both directions.

Versions
  • OS: macOS 26.5
  • Python version: 3.12.15
  • PyRIT version: 1.2.0.dev0, installed from main in editable mode (08ed8f45)
Vorherrschende Sprache
Python
Sterne
4.7k
Forks
1k
Ø Merge
2 T. 4 Std.
Gemergte PRs (30 T.)
510

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

  • Kein Dockerfile und keine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus microsoft/PyRIT

Alle Issues in microsoft/PyRIT

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.