Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

BUG PlagiarismScorer tokenizer strips combining marks and skips normalization, so a verbatim copy can score 0.0 and different words can score 1.0

Abierto
  • ¿Algo no está bien?
#3,052 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

@inchang-ing ya está trabajando en esto.

Desde el 9/10/2026.

  • #3053 de @inchang-ing — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
ai, internationalization

Línea de trabajo

The bug is in PlagiarismScorer._tokenize in pyrit/score/float_scale/plagiarism_scorer.py, around line 73, where the regex strips non-word characters before any Unicode normalisation. Before reading the fix, decide between NFKC and NFC with the maintainers, since the issue leaves that open. Pull request #3053 is already open against this issue, so check it first. Done means the NFC/NFD and fullwidth examples score 1.0 against the reference, the Devanagari pair no longer collides, and the existing tests in tests/unit/score still pass.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

PlagiarismScorer._tokenize (pyrit/score/float_scale/plagiarism_scorer.py:73) lowercases, then strips everything that is neither \w nor \s:

text = re.sub(r"[^\w\s]", "", text)

Python's \w does not include combining marks (category M), and no normalisation happens first. That makes the scorer wrong in both directions.

False negatives. Text that reads identically to the reference scores as unrelated, because the bytes differ in a way the tokenizer does not fold. Verbatim reproductions of the reference:

response lcs lev jaccard
verbatim 1.000 1.000 1.000
same text, NFD form 0.545 0.545 0.000
same text, fullwidth 0.000 0.000 0.000
same text, math-bold 0.000 0.000 0.000

NFD affects accented text: café decomposes to cafe + U+0301, the mark is stripped as non-\w, and the token becomes cafe while the NFC reference keeps café. Fullwidth and math-alphanumeric are not encoding accidents but the ordinary homoglyph substitutions, and they take every metric to zero.

False positives. Because marks are stripped rather than normalised, scripts that carry meaning in combining marks collapse into each other. दिन ("day") and दीन ("poor") are different Hindi words; both tokenize to दन:

PlagiarismScorer(reference_text="दिन", metric=PlagiarismMetric.LCS)
  ._plagiarism_score("दीन", "दिन", metric=PlagiarismMetric.LCS)
# 1.0

Devanagari, Thai and Tamil are mangled wholesale: सिस्टम → ससटम, สวัสดี → สวสด, வணக்கம் → வணககம.

This is distinct from #2971 / #2972, which cover n-gram size and blank reference validation.

Steps/Code to Reproduce
import unicodedata
from pyrit.score.float_scale.plagiarism_scorer import PlagiarismScorer, PlagiarismMetric

REF = ("Il était une fois une très jeune fille naïve qui habitait près "
       "d'un café à Genève où l'on servait des crèmes brûlées")

for metric in PlagiarismMetric:
    s = PlagiarismScorer(reference_text=REF, metric=metric)
    print(metric.value,
          s._plagiarism_score(REF, REF, metric=metric, n=5),
          s._plagiarism_score(unicodedata.normalize("NFD", REF), REF, metric=metric, n=5))

ASCII = "the quick brown fox jumps over the lazy dog"
fullwidth = "".join(chr(ord(c) - 0x20 + 0xFF00) if 0x21 <= ord(c) <= 0x7e
                    else ("　" if c == " " else c) for c in ASCII)
s = PlagiarismScorer(reference_text=ASCII, metric=PlagiarismMetric.JACCARD)
print(s._plagiarism_score(fullwidth, ASCII, metric=PlagiarismMetric.JACCARD, n=5))
Expected Results

A response that a reader would call a verbatim copy scores as one, and two different words do not score as identical.

Actual Results
lcs         1.0 0.5454545454545454
levenshtein 1.0 0.5454545454545454
jaccard     1.0 0.0
0.0          <- fullwidth
Suggested fix

Normalise before tokenizing, and keep combining marks rather than discarding them:

text = unicodedata.normalize("NFKC", text).lower()
text = "".join(
    c for c in text
    if c.isspace() or c.isalnum() or c == "_" or unicodedata.category(c).startswith("M")
)
return text.split()

I validated this on four axes: NFC and NFD forms now tokenize identically (0 disagreements over accented Latin, Greek and Vietnamese); the Hindi, Thai and Tamil pairs above no longer collide; all 23 pure-ASCII inputs in my corpus tokenize exactly as before; and no input in a 334-case corpus (20 scripts × NFC/NFD × 8 whitespace shapes, plus edge cases) raises.

With it applied, all four rows of the first table read 1.000 and the Hindi false positive goes to 0.0. tests/unit/score and tests/unit/converter are 5316 passed, 108 skipped.

One choice is yours rather than mine: NFKC or NFC. NFC fixes the NFD row only. NFKC additionally folds the homoglyph rows, which is why I used it, but it is lossier — ½ → 12, Ⅻ → xii, fi → fi, x² → x2. For a scorer whose job is detecting reproduction I think that trade is right, since those are evasion vectors too, but ½ → 12 is a real wart and you may prefer NFC plus an explicit confusable-folding step.

Happy to open the PR with whichever you pick, with regression tests for both directions.

Versions
  • OS: macOS 26.5
  • Python version: 3.12.15
  • PyRIT version: 1.2.0.dev0, installed from main in editable mode (08ed8f45)
Lenguaje dominante
Python
Estrellas
4.6k
Forks
944
Merge medio
3 d 1 h
PR fusionados (30 d)
278

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/PyRIT

Todos los issues de microsoft/PyRIT

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.