[Bug]: robots.txt rules ending in `$` never match on Python 3.13 and older

Offen
#2,544 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@adityajai119 arbeitet bereits daran.

Seit 12.11.2026.

  • #2545 von @adityajai119 — offen

Bewertung

Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Anfängerfreundlichkeit
30/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
backend

Rechercherichtung

Start with patched_applies_to in crawl4ai/utils.py, where the robots.txt rule path is checked for a trailing $ or %24, and with the RobotsParser code that calls it. Run the reproduction script on Python 3.13 to confirm the four can_fetch results. Done means *.pdf$ blocks report.pdf, Allow: /$ allows the home page, and the tests cover both cases. The author says they are opening a PR for this, so check for one before starting.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

crawl4ai version

0.9.5 (also reproduced on develop at 1a6eddb)

Expected Behavior

A robots.txt rule that ends in $ only matches URLs that end at that point (RFC 9309, section 2.2.3). With check_robots_txt=True, Disallow: /*.pdf$ keeps the crawler away from /files/report.pdf, and Allow: /$ lets it fetch the home page.

Current Behavior

On Python 3.13 and older, rules ending in $ never match:

  • Disallow: /*.pdf$ does not block /files/report.pdf, so the crawler fetches URLs the site disallows.
  • With Allow: /$ followed by Disallow: /, the home page is disallowed as well.

Below 3.14, urllib.robotparser.RuleLine quotes the rule path, so the $ is already %24 when patched_applies_to in crawl4ai/utils.py sees it. Its pattern.endswith('\\$') check never fires, and the regex looks for a literal %24. Rules without * (like Allow: /$) don't reach that code and are compared with startswith('/%24').

Python 3.14 supports $ natively and isn't affected. The description of #2278 mentions this gap ("Happy to do it separately"); I couldn't find an issue or PR for it.

Is this reproducible?

Yes

Inputs Causing the Bug
User-agent: *
Disallow: /*.pdf$

and

User-agent: *
Allow: /$
Disallow: /
Steps to Reproduce

Run the script below with Python 3.13. It serves the two robots.txt files from local HTTP servers, so no network is needed.

Code snippets
import asyncio
import tempfile
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

from crawl4ai.utils import RobotsParser

ROBOTS = {
    8765: b"User-agent: *\nDisallow: /*.pdf$\n",
    8766: b"User-agent: *\nAllow: /$\nDisallow: /\n",
}


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(ROBOTS[self.server.server_port])

    def log_message(self, *args):
        pass


for port in ROBOTS:
    server = HTTPServer(("127.0.0.1", port), Handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()


async def main():
    parser = RobotsParser(cache_dir=tempfile.mkdtemp())
    for url in [
        "http://127.0.0.1:8765/files/report.pdf",
        "http://127.0.0.1:8765/files/report.pdf.html",
        "http://127.0.0.1:8766/",
        "http://127.0.0.1:8766/page",
    ]:
        print(url, "->", await parser.can_fetch(url, "bot"))


asyncio.run(main())

Output:

http://127.0.0.1:8765/files/report.pdf -> True
http://127.0.0.1:8765/files/report.pdf.html -> True
http://127.0.0.1:8766/ -> False
http://127.0.0.1:8766/page -> False

Expected:

http://127.0.0.1:8765/files/report.pdf -> False
http://127.0.0.1:8765/files/report.pdf.html -> True
http://127.0.0.1:8766/ -> True
http://127.0.0.1:8766/page -> False
OS

Windows 11

Python version

3.13.5 (3.11 and 3.12 quote the rule path the same way)

Browser

Not used (RobotsParser only fetches robots.txt with aiohttp)

Browser version
Error logs & Screenshots (if applicable)

No error; can_fetch() returns the wrong answer.

I'm opening a PR that treats a trailing %24 as the end anchor in patched_applies_to, with tests.

AI-assisted: found and reproduced with an AI coding agent; the example above was run locally.

Vorherrschende Sprache
Python
Sterne
87.2k
Forks
9.1k
Ø Merge
1 T. 6 Std.
Gemergte PRs (30 T.)
25

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus unclecode/crawl4ai

Alle Issues in unclecode/crawl4ai

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.