[Bug]: robots.txt rules ending in `$` never match on Python 3.13 and older

Aperta
#2,544 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@adityajai119 ci sta già lavorando.

Dal 12/11/2026.

  • #2545 di @adityajai119 — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
30/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start with patched_applies_to in crawl4ai/utils.py, where the robots.txt rule path is checked for a trailing $ or %24, and with the RobotsParser code that calls it. Run the reproduction script on Python 3.13 to confirm the four can_fetch results. Done means *.pdf$ blocks report.pdf, Allow: /$ allows the home page, and the tests cover both cases. The author says they are opening a PR for this, so check for one before starting.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

crawl4ai version

0.9.5 (also reproduced on develop at 1a6eddb)

Expected Behavior

A robots.txt rule that ends in $ only matches URLs that end at that point (RFC 9309, section 2.2.3). With check_robots_txt=True, Disallow: /*.pdf$ keeps the crawler away from /files/report.pdf, and Allow: /$ lets it fetch the home page.

Current Behavior

On Python 3.13 and older, rules ending in $ never match:

  • Disallow: /*.pdf$ does not block /files/report.pdf, so the crawler fetches URLs the site disallows.
  • With Allow: /$ followed by Disallow: /, the home page is disallowed as well.

Below 3.14, urllib.robotparser.RuleLine quotes the rule path, so the $ is already %24 when patched_applies_to in crawl4ai/utils.py sees it. Its pattern.endswith('\\$') check never fires, and the regex looks for a literal %24. Rules without * (like Allow: /$) don't reach that code and are compared with startswith('/%24').

Python 3.14 supports $ natively and isn't affected. The description of #2278 mentions this gap ("Happy to do it separately"); I couldn't find an issue or PR for it.

Is this reproducible?

Yes

Inputs Causing the Bug
User-agent: *
Disallow: /*.pdf$

and

User-agent: *
Allow: /$
Disallow: /
Steps to Reproduce

Run the script below with Python 3.13. It serves the two robots.txt files from local HTTP servers, so no network is needed.

Code snippets
import asyncio
import tempfile
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

from crawl4ai.utils import RobotsParser

ROBOTS = {
    8765: b"User-agent: *\nDisallow: /*.pdf$\n",
    8766: b"User-agent: *\nAllow: /$\nDisallow: /\n",
}


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(ROBOTS[self.server.server_port])

    def log_message(self, *args):
        pass


for port in ROBOTS:
    server = HTTPServer(("127.0.0.1", port), Handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()


async def main():
    parser = RobotsParser(cache_dir=tempfile.mkdtemp())
    for url in [
        "http://127.0.0.1:8765/files/report.pdf",
        "http://127.0.0.1:8765/files/report.pdf.html",
        "http://127.0.0.1:8766/",
        "http://127.0.0.1:8766/page",
    ]:
        print(url, "->", await parser.can_fetch(url, "bot"))


asyncio.run(main())

Output:

http://127.0.0.1:8765/files/report.pdf -> True
http://127.0.0.1:8765/files/report.pdf.html -> True
http://127.0.0.1:8766/ -> False
http://127.0.0.1:8766/page -> False

Expected:

http://127.0.0.1:8765/files/report.pdf -> False
http://127.0.0.1:8765/files/report.pdf.html -> True
http://127.0.0.1:8766/ -> True
http://127.0.0.1:8766/page -> False
OS

Windows 11

Python version

3.13.5 (3.11 and 3.12 quote the rule path the same way)

Browser

Not used (RobotsParser only fetches robots.txt with aiohttp)

Browser version
Error logs & Screenshots (if applicable)

No error; can_fetch() returns the wrong answer.

I'm opening a PR that treats a trailing %24 as the end anchor in patched_applies_to, with tests.

AI-assisted: found and reproduced with an AI coding agent; the example above was run locally.

Lingua principale
Python
Stelle
87.2k
Fork
9.1k
Merge medio
1g 6h
PR unite (30g)
25

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.