Quadratic comment split on a requirements file line (pip_requirements_parser.py:1058)

慢的模式是 src/pip_requirements_parser.py 中第 1058 行附近的 COMMENT_RE，它被第 1569 行附近的注释拆分循环使用。请阅读为什么那里会丢弃空白分组，并通过 RequirementsFile.from_file() 用一个很长的空白行复现耗时。完成标准是：现有输入的拆分输出保持不变，64 KB 的空白行能在毫秒级解析完成，并且现有测试套件没有出现新的失败。

Parsing a requirements file takes time quadratic in the length of any line that is mostly whitespace. Measured on 32.0.1 through RequirementsFile.from_file(), with one whitespace line in an otherwise ordinary three-line file:

Two controls, both measured, and together they show the cost is the shape and not the size:

• 4,000 ordinary pinned lines with trailing comments - a 149,780-byte file, more than twice the size - costs 0.7479 s, against 20.5483 s for the 64,027-byte hostile one.
• The same 64 KB of whitespace followed by a # costs 0.0008 s. The cost needs the match to fail.

Who this matters to. The package exists to parse requirements files without installing anything - the README stresses that it works offline - which is the SBOM and software-composition-analysis use: reading requirements files out of third-party or unknown codebases. Dependents include CycloneDX/cyclonedx-python and BentoML. So the payload is one long whitespace line in an otherwise ordinary file, and a small commit to a scanned repository costs minutes of scanner CPU. (The same shape exists in pip itself, at src/pip/_internal/req/req_file.py:41, still with \s+ on current master - but pip install -r on an untrusted requirements file already runs arbitrary code, so there is nothing to protect there. Mentioned only so you know it is not a fork-only divergence.)

Cause. src/pip_requirements_parser.py:1058:

COMMENT_RE = re.compile(r"(^|\s+)(#.*)$")

applied at :1569, once per line:

parts = [l.strip() for l in COMMENT_RE.split(line) if l.strip()]

split tries every offset, and at each one \s+ consumes the rest of the whitespace run before failing to find the #. O(n) work at O(n) start positions. The two COMMENT_RE.match() uses in join_lines() (:1539, :1540) are anchored and stay linear - measured at 2.5 ms on a 256 KB line - so only the split is affected.

Fix - one character, and no cap. The caller strips every part and drops the empty ones, so whatever the whitespace group captured is discarded. A single whitespace character therefore produces the same list as a run, and makes each start position cost O(1):

 COMMENT_RE = re.compile(r"(^|\s+)(#.*)$")
+
+COMMENT_SPLIT_RE = re.compile(r"(^|\s)(#.*)$")
...
-        parts = [l.strip() for l in COMMENT_RE.split(line) if l.strip()]
+        parts = [l.strip() for l in COMMENT_SPLIT_RE.split(line) if l.strip()]

COMMENT_RE itself is left alone, because the two match() uses in join_lines() do rely on the run: with a single \s they would stop recognising " # comment" as a comment line.

This has no divergence class. Over 311,161 inputs - every string of length 0 to 5 over # space tab \x0b \x0c \r a = . -, 200,000 random strings over the same alphabet, hand-written real requirement lines, and whitespace runs either side of 200 characters - the list the caller builds is identical in every case. Zero divergences, no cap to justify.

Two alternatives were tested and should not be used.

1. Bounding the run ((^|\s{1,200})) is also exactly equivalent on those inputs, but it is about 90 times slower than the single-character fix - 0.1253 s against 0.0014 s on a 64 KB run - and it introduces a cap that has to be justified. There is no reason to take that trade.
2. Narrowing the class ((^|[ \t]{1,200})) changes behaviour and must not be used. \s covers \x0b, \x0c and \r, and [ \t] does not, so on the same 311,161 inputs it differs on 21,427 - every one of them because of those characters. The failure is not cosmetic: "a\x0b#b" currently splits into ['a', '#b'] and under that pattern stays ['a\x0b#b'], so the comment is no longer separated and the comment text is carried into the requirement line.

Verification.

• Differential as described above, compared at the caller's output rather than at the regex.
• Project test suite on both trees: 8,246 passed, 20 failed, 5 skipped, 1 xfailed, the 20 failures identical by name. They are pre-existing error-message comparisons - the vendored packaging in this environment words "Invalid requirement" differently from the checked-in expectations - and they fail on unmodified 32.0.1.
• Control test: ten real lines split identically and a three-line file is parsed to the right requirement names, plus a timing assertion on a 16 KB whitespace line. On 32.0.1 the splitting assertions pass and the timing assertion fails at 1.3453 s; patched, both pass in 0.002 s.
• patch -p1 verified against the 32.0.1 sdist.

For context rather than as prior art: pypa/pip #3662 (2016, closed 2019) discusses this expression, but as a Jython maximum recursion depth exceeded error with no cost figure, and pip declined to change it on the grounds that "keeping existing behaviour is more important". Nobody in that thread identified the CPython behaviour, and the expression is unchanged in pip today.

I am happy to open a pull request with the patch if that is easier.

Found with AI assistance (Claude) and verified by hand.

--- a/src/pip_requirements_parser.py
+++ b/src/pip_requirements_parser.py
@@ -1057,6 +1057,15 @@
 SCHEME_RE = re.compile(r"^(http|https|file):", re.I)
 COMMENT_RE = re.compile(r"(^|\s+)(#.*)$")
 
+# Used only by split_comments(), which strips every part and drops the empty
+# ones, so what the whitespace group captures is discarded. A single whitespace
+# character therefore gives the same result as a run, and makes each start
+# position cost O(1): with "\s+" every offset in a run of whitespace starts an
+# attempt that consumes the rest of the line before failing to find the "#", so
+# splitting a line is quadratic in its length. COMMENT_RE itself is unchanged,
+# because the two match() uses in join_lines() rely on the run.
+COMMENT_SPLIT_RE = re.compile(r"(^|\s)(#.*)$")
+
 SUPPORTED_OPTIONS: List[Callable[..., optparse.Option]] = [
     index_url,
     extra_index_url,
@@ -1566,7 +1575,7 @@
     Yield TextLine or Commentline
     """
     for line_number, line in lines_enum:
-        parts = [l.strip() for l in COMMENT_RE.split(line) if l.strip()]
+        parts = [l.strip() for l in COMMENT_SPLIT_RE.split(line) if l.strip()]
 
         if len(parts) == 1:
             part = parts[0]