Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Quadratic comment split on a requirements file line (pip_requirements_parser.py:1058)

Aperta Adatta ai principianti
  • Qualcosa non va?
#30 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

@Anurag-S1ngh ci sta già lavorando.

Dal 24/10/2026.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
72/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
tooling

Direzione di ricerca

Il pattern lento è COMMENT_RE in src/pip_requirements_parser.py vicino alla riga 1058, usato dal ciclo di suddivisione dei commenti vicino alla riga 1569. Leggi perché il gruppo di spazi bianchi viene scartato lì, e riproduci i tempi con una lunga riga di spazi bianchi tramite RequirementsFile.from_file(). Il lavoro è completato quando l'output della suddivisione per gli input esistenti resta invariato, una riga di spazi bianchi da 64 KB viene analizzata in millisecondi e la suite di test esistente non mostra nuovi errori.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Parsing a requirements file takes time quadratic in the length of any line that is mostly whitespace. Measured on 32.0.1 through RequirementsFile.from_file(), with one whitespace line in an otherwise ordinary three-line file:

whitespace line file size before after
4 KB 4,027 B 0.1165 s 0.0007 s
16 KB 16,027 B 1.4871 s 0.0011 s
64 KB 64,027 B 20.5483 s 0.0030 s
128 KB 128,027 B - 0.0057 s

Two controls, both measured, and together they show the cost is the shape and not the size:

  • 4,000 ordinary pinned lines with trailing comments - a 149,780-byte file, more than twice the size - costs 0.7479 s, against 20.5483 s for the 64,027-byte hostile one.
  • The same 64 KB of whitespace followed by a # costs 0.0008 s. The cost needs the match to fail.

Who this matters to. The package exists to parse requirements files without installing anything - the README stresses that it works offline - which is the SBOM and software-composition-analysis use: reading requirements files out of third-party or unknown codebases. Dependents include CycloneDX/cyclonedx-python and BentoML. So the payload is one long whitespace line in an otherwise ordinary file, and a small commit to a scanned repository costs minutes of scanner CPU. (The same shape exists in pip itself, at src/pip/_internal/req/req_file.py:41, still with \s+ on current master - but pip install -r on an untrusted requirements file already runs arbitrary code, so there is nothing to protect there. Mentioned only so you know it is not a fork-only divergence.)

Cause. src/pip_requirements_parser.py:1058:

COMMENT_RE = re.compile(r"(^|\s+)(#.*)$")

applied at :1569, once per line:

parts = [l.strip() for l in COMMENT_RE.split(line) if l.strip()]

split tries every offset, and at each one \s+ consumes the rest of the whitespace run before failing to find the #. O(n) work at O(n) start positions. The two COMMENT_RE.match() uses in join_lines() (:1539, :1540) are anchored and stay linear - measured at 2.5 ms on a 256 KB line - so only the split is affected.

Fix - one character, and no cap. The caller strips every part and drops the empty ones, so whatever the whitespace group captured is discarded. A single whitespace character therefore produces the same list as a run, and makes each start position cost O(1):

 COMMENT_RE = re.compile(r"(^|\s+)(#.*)$")
+
+COMMENT_SPLIT_RE = re.compile(r"(^|\s)(#.*)$")
...
-        parts = [l.strip() for l in COMMENT_RE.split(line) if l.strip()]
+        parts = [l.strip() for l in COMMENT_SPLIT_RE.split(line) if l.strip()]

COMMENT_RE itself is left alone, because the two match() uses in join_lines() do rely on the run: with a single \s they would stop recognising " # comment" as a comment line.

This has no divergence class. Over 311,161 inputs - every string of length 0 to 5 over # space tab \x0b \x0c \r a = . -, 200,000 random strings over the same alphabet, hand-written real requirement lines, and whitespace runs either side of 200 characters - the list the caller builds is identical in every case. Zero divergences, no cap to justify.

Two alternatives were tested and should not be used.

  1. Bounding the run ((^|\s{1,200})) is also exactly equivalent on those inputs, but it is about 90 times slower than the single-character fix - 0.1253 s against 0.0014 s on a 64 KB run - and it introduces a cap that has to be justified. There is no reason to take that trade.
  2. Narrowing the class ((^|[ \t]{1,200})) changes behaviour and must not be used. \s covers \x0b, \x0c and \r, and [ \t] does not, so on the same 311,161 inputs it differs on 21,427 - every one of them because of those characters. The failure is not cosmetic: "a\x0b#b" currently splits into ['a', '#b'] and under that pattern stays ['a\x0b#b'], so the comment is no longer separated and the comment text is carried into the requirement line.

Verification.

  • Differential as described above, compared at the caller's output rather than at the regex.
  • Project test suite on both trees: 8,246 passed, 20 failed, 5 skipped, 1 xfailed, the 20 failures identical by name. They are pre-existing error-message comparisons - the vendored packaging in this environment words "Invalid requirement" differently from the checked-in expectations - and they fail on unmodified 32.0.1.
  • Control test: ten real lines split identically and a three-line file is parsed to the right requirement names, plus a timing assertion on a 16 KB whitespace line. On 32.0.1 the splitting assertions pass and the timing assertion fails at 1.3453 s; patched, both pass in 0.002 s.
  • patch -p1 verified against the 32.0.1 sdist.

For context rather than as prior art: pypa/pip #3662 (2016, closed 2019) discusses this expression, but as a Jython maximum recursion depth exceeded error with no cost figure, and pip declined to change it on the grounds that "keeping existing behaviour is more important". Nobody in that thread identified the CPython behaviour, and the expression is unchanged in pip today.

I am happy to open a pull request with the patch if that is easier.

Found with AI assistance (Claude) and verified by hand.

Patch: full patch 
--- a/src/pip_requirements_parser.py
+++ b/src/pip_requirements_parser.py
@@ -1057,6 +1057,15 @@
 SCHEME_RE = re.compile(r"^(http|https|file):", re.I)
 COMMENT_RE = re.compile(r"(^|\s+)(#.*)$")
 
+# Used only by split_comments(), which strips every part and drops the empty
+# ones, so what the whitespace group captures is discarded. A single whitespace
+# character therefore gives the same result as a run, and makes each start
+# position cost O(1): with "\s+" every offset in a run of whitespace starts an
+# attempt that consumes the rest of the line before failing to find the "#", so
+# splitting a line is quadratic in its length. COMMENT_RE itself is unchanged,
+# because the two match() uses in join_lines() rely on the run.
+COMMENT_SPLIT_RE = re.compile(r"(^|\s)(#.*)$")
+
 SUPPORTED_OPTIONS: List[Callable[..., optparse.Option]] = [
     index_url,
     extra_index_url,
@@ -1566,7 +1575,7 @@
     Yield TextLine or Commentline
     """
     for line_number, line in lines_enum:
-        parts = [l.strip() for l in COMMENT_RE.split(line) if l.strip()]
+        parts = [l.strip() for l in COMMENT_SPLIT_RE.split(line) if l.strip()]
 
         if len(parts) == 1:
             part = parts[0]
Lingua principale
Python
Stelle
20
Fork
16
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di aboutcode-org/pip-requirements-parser

Tutte le issue di aboutcode-org/pip-requirements-parser

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.