Quadratic comment split on a requirements file line (pip_requirements_parser.py:1058)

Open Beginner friendly
#30 1 comment 0 reactions 0 assignees View on GitHub

@Anurag-S1ngh is already working on this.

Since Oct 24, 2026.

  • #31 by @Anurag-S1ngh — closed without merging

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
tooling

Research direction

The slow pattern is COMMENT_RE in src/pip_requirements_parser.py near line 1058, used by the comment-splitting loop near line 1569. Read why the whitespace group is discarded there, and reproduce the timing with a long whitespace line through RequirementsFile.from_file(). Done means the split output for existing inputs is unchanged, a 64 KB whitespace line parses in milliseconds, and the existing test suite shows no new failures.

Written by the indexing model from the issue text.

Description

Parsing a requirements file takes time quadratic in the length of any line that is mostly whitespace. Measured on 32.0.1 through RequirementsFile.from_file(), with one whitespace line in an otherwise ordinary three-line file:

whitespace line file size before after
4 KB 4,027 B 0.1165 s 0.0007 s
16 KB 16,027 B 1.4871 s 0.0011 s
64 KB 64,027 B 20.5483 s 0.0030 s
128 KB 128,027 B - 0.0057 s

Two controls, both measured, and together they show the cost is the shape and not the size:

  • 4,000 ordinary pinned lines with trailing comments - a 149,780-byte file, more than twice the size - costs 0.7479 s, against 20.5483 s for the 64,027-byte hostile one.
  • The same 64 KB of whitespace followed by a # costs 0.0008 s. The cost needs the match to fail.

Who this matters to. The package exists to parse requirements files without installing anything - the README stresses that it works offline - which is the SBOM and software-composition-analysis use: reading requirements files out of third-party or unknown codebases. Dependents include CycloneDX/cyclonedx-python and BentoML. So the payload is one long whitespace line in an otherwise ordinary file, and a small commit to a scanned repository costs minutes of scanner CPU. (The same shape exists in pip itself, at src/pip/_internal/req/req_file.py:41, still with \s+ on current master - but pip install -r on an untrusted requirements file already runs arbitrary code, so there is nothing to protect there. Mentioned only so you know it is not a fork-only divergence.)

Cause. src/pip_requirements_parser.py:1058:

COMMENT_RE = re.compile(r"(^|\s+)(#.*)$")

applied at :1569, once per line:

parts = [l.strip() for l in COMMENT_RE.split(line) if l.strip()]

split tries every offset, and at each one \s+ consumes the rest of the whitespace run before failing to find the #. O(n) work at O(n) start positions. The two COMMENT_RE.match() uses in join_lines() (:1539, :1540) are anchored and stay linear - measured at 2.5 ms on a 256 KB line - so only the split is affected.

Fix - one character, and no cap. The caller strips every part and drops the empty ones, so whatever the whitespace group captured is discarded. A single whitespace character therefore produces the same list as a run, and makes each start position cost O(1):

 COMMENT_RE = re.compile(r"(^|\s+)(#.*)$")
+
+COMMENT_SPLIT_RE = re.compile(r"(^|\s)(#.*)$")
...
-        parts = [l.strip() for l in COMMENT_RE.split(line) if l.strip()]
+        parts = [l.strip() for l in COMMENT_SPLIT_RE.split(line) if l.strip()]

COMMENT_RE itself is left alone, because the two match() uses in join_lines() do rely on the run: with a single \s they would stop recognising " # comment" as a comment line.

This has no divergence class. Over 311,161 inputs - every string of length 0 to 5 over # space tab \x0b \x0c \r a = . -, 200,000 random strings over the same alphabet, hand-written real requirement lines, and whitespace runs either side of 200 characters - the list the caller builds is identical in every case. Zero divergences, no cap to justify.

Two alternatives were tested and should not be used.

  1. Bounding the run ((^|\s{1,200})) is also exactly equivalent on those inputs, but it is about 90 times slower than the single-character fix - 0.1253 s against 0.0014 s on a 64 KB run - and it introduces a cap that has to be justified. There is no reason to take that trade.
  2. Narrowing the class ((^|[ \t]{1,200})) changes behaviour and must not be used. \s covers \x0b, \x0c and \r, and [ \t] does not, so on the same 311,161 inputs it differs on 21,427 - every one of them because of those characters. The failure is not cosmetic: "a\x0b#b" currently splits into ['a', '#b'] and under that pattern stays ['a\x0b#b'], so the comment is no longer separated and the comment text is carried into the requirement line.

Verification.

  • Differential as described above, compared at the caller's output rather than at the regex.
  • Project test suite on both trees: 8,246 passed, 20 failed, 5 skipped, 1 xfailed, the 20 failures identical by name. They are pre-existing error-message comparisons - the vendored packaging in this environment words "Invalid requirement" differently from the checked-in expectations - and they fail on unmodified 32.0.1.
  • Control test: ten real lines split identically and a three-line file is parsed to the right requirement names, plus a timing assertion on a 16 KB whitespace line. On 32.0.1 the splitting assertions pass and the timing assertion fails at 1.3453 s; patched, both pass in 0.002 s.
  • patch -p1 verified against the 32.0.1 sdist.

For context rather than as prior art: pypa/pip #3662 (2016, closed 2019) discusses this expression, but as a Jython maximum recursion depth exceeded error with no cost figure, and pip declined to change it on the grounds that "keeping existing behaviour is more important". Nobody in that thread identified the CPython behaviour, and the expression is unchanged in pip today.

I am happy to open a pull request with the patch if that is easier.

Found with AI assistance (Claude) and verified by hand.

Patch: full patch
--- a/src/pip_requirements_parser.py
+++ b/src/pip_requirements_parser.py
@@ -1057,6 +1057,15 @@
 SCHEME_RE = re.compile(r"^(http|https|file):", re.I)
 COMMENT_RE = re.compile(r"(^|\s+)(#.*)$")
 
+# Used only by split_comments(), which strips every part and drops the empty
+# ones, so what the whitespace group captures is discarded. A single whitespace
+# character therefore gives the same result as a run, and makes each start
+# position cost O(1): with "\s+" every offset in a run of whitespace starts an
+# attempt that consumes the rest of the line before failing to find the "#", so
+# splitting a line is quadratic in its length. COMMENT_RE itself is unchanged,
+# because the two match() uses in join_lines() rely on the run.
+COMMENT_SPLIT_RE = re.compile(r"(^|\s)(#.*)$")
+
 SUPPORTED_OPTIONS: List[Callable[..., optparse.Option]] = [
     index_url,
     extra_index_url,
@@ -1566,7 +1575,7 @@
     Yield TextLine or Commentline
     """
     for line_number, line in lines_enum:
-        parts = [l.strip() for l in COMMENT_RE.split(line) if l.strip()]
+        parts = [l.strip() for l in COMMENT_SPLIT_RE.split(line) if l.strip()]
 
         if len(parts) == 1:
             part = parts[0]
Dominant language
Python
Stars
20
Forks
17
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aboutcode-org/pip-requirements-parser

All issues in aboutcode-org/pip-requirements-parser

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.