Note greediness of PEP 723 reference parser

Open Beginner friendly
#1,960 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
documentation

Research direction

Start with the PEP 723 reference parser and the PyPA documentation that lists it, then reproduce the issue using the REGEX and script_A/script_B examples in the report. Update the relevant documentation to warn about greedy collation of adjacent blocks, and confirm that the warning accurately describes the resulting invalid TOML edge case.

Written by the indexing model from the issue text.

Description

Issue Description

While preparing a PR for PEP 723 support in pip, I noticed that the reference parser defined by the PEP and listed in the PyPA docs will collate multiple adjacent /// TYPE blocks as a single match, even when separated by a comment line (the spec refers to it as a "content line"). This greedy collation is surprising and makes distinguishing error cases a little complicated, so I think it merits a warning in the docs if it is not possible to update the specification itself.

I believe this quirk is caused by the last + in the reference regex being greedy and matching all the way to the trailing /// instead of to the first available one. In my limited experimentation, replacing this quantifier with +? resolves the issue, producing the expected number of matches.

This shouldn't slip through anybody's code unnoticed, as the collation will produce invalid TOML (the interior /// is invalid syntax), but it is a surprising enough edge case that I thought to report it here.

click for code 
import re

script_A = """
# /// script
# data (1)
# ///
#
# /// script
# data (2)
# ///
"""

script_B = """
# /// script
# data (1)
# ///

# /// script
# data (2)
# ///
"""

# These lines adapted from PEP 723's reference parser:
# https://peps.python.org/pep-0723/#reference-implementation

REGEX = r"(?m)^# /// (?P<type>[a-zA-Z0-9-]+)$\s(?P<content>(^#(| .*)$\s)+)^# ///$"
name = "script"
matches_A = list(
    filter(lambda m: m.group("type") == name, re.finditer(REGEX, script_A))
)
matches_B = list(
    filter(lambda m: m.group("type") == name, re.finditer(REGEX, script_B))
)

# output:
# 1
# 2
print(len(matches_A))
print(len(matches_B))
Code of Conduct
  • I am aware that participants in this repository must follow the PSF Code of Conduct.
Dominant language
Python
Stars
1.7k
Forks
1.7k
Avg merge
3d 12h
Merged PRs (30d)
4

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pypa/packaging.python.org

All issues in pypa/packaging.python.org

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.