[Bug]: robots.txt rules ending in `$` never match on Python 3.13 and older

Ouverte
#2,544 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@adityajai119 y travaille déjà.

Depuis le 12/11/2026.

  • #2545 par @adityajai119 — ouverte

Évaluation

Difficulté
2/5
Temps estimé
1-3 heures
Accessibilité débutants
30/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python
Domaine
backend

Piste de recherche

Start with patched_applies_to in crawl4ai/utils.py, where the robots.txt rule path is checked for a trailing $ or %24, and with the RobotsParser code that calls it. Run the reproduction script on Python 3.13 to confirm the four can_fetch results. Done means *.pdf$ blocks report.pdf, Allow: /$ allows the home page, and the tests cover both cases. The author says they are opening a PR for this, so check for one before starting.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

crawl4ai version

0.9.5 (also reproduced on develop at 1a6eddb)

Expected Behavior

A robots.txt rule that ends in $ only matches URLs that end at that point (RFC 9309, section 2.2.3). With check_robots_txt=True, Disallow: /*.pdf$ keeps the crawler away from /files/report.pdf, and Allow: /$ lets it fetch the home page.

Current Behavior

On Python 3.13 and older, rules ending in $ never match:

  • Disallow: /*.pdf$ does not block /files/report.pdf, so the crawler fetches URLs the site disallows.
  • With Allow: /$ followed by Disallow: /, the home page is disallowed as well.

Below 3.14, urllib.robotparser.RuleLine quotes the rule path, so the $ is already %24 when patched_applies_to in crawl4ai/utils.py sees it. Its pattern.endswith('\\$') check never fires, and the regex looks for a literal %24. Rules without * (like Allow: /$) don't reach that code and are compared with startswith('/%24').

Python 3.14 supports $ natively and isn't affected. The description of #2278 mentions this gap ("Happy to do it separately"); I couldn't find an issue or PR for it.

Is this reproducible?

Yes

Inputs Causing the Bug
User-agent: *
Disallow: /*.pdf$

and

User-agent: *
Allow: /$
Disallow: /
Steps to Reproduce

Run the script below with Python 3.13. It serves the two robots.txt files from local HTTP servers, so no network is needed.

Code snippets
import asyncio
import tempfile
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

from crawl4ai.utils import RobotsParser

ROBOTS = {
    8765: b"User-agent: *\nDisallow: /*.pdf$\n",
    8766: b"User-agent: *\nAllow: /$\nDisallow: /\n",
}


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(ROBOTS[self.server.server_port])

    def log_message(self, *args):
        pass


for port in ROBOTS:
    server = HTTPServer(("127.0.0.1", port), Handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()


async def main():
    parser = RobotsParser(cache_dir=tempfile.mkdtemp())
    for url in [
        "http://127.0.0.1:8765/files/report.pdf",
        "http://127.0.0.1:8765/files/report.pdf.html",
        "http://127.0.0.1:8766/",
        "http://127.0.0.1:8766/page",
    ]:
        print(url, "->", await parser.can_fetch(url, "bot"))


asyncio.run(main())

Output:

http://127.0.0.1:8765/files/report.pdf -> True
http://127.0.0.1:8765/files/report.pdf.html -> True
http://127.0.0.1:8766/ -> False
http://127.0.0.1:8766/page -> False

Expected:

http://127.0.0.1:8765/files/report.pdf -> False
http://127.0.0.1:8765/files/report.pdf.html -> True
http://127.0.0.1:8766/ -> True
http://127.0.0.1:8766/page -> False
OS

Windows 11

Python version

3.13.5 (3.11 and 3.12 quote the rule path the same way)

Browser

Not used (RobotsParser only fetches robots.txt with aiohttp)

Browser version
Error logs & Screenshots (if applicable)

No error; can_fetch() returns the wrong answer.

I'm opening a PR that treats a trailing %24 as the end anchor in patched_applies_to, with tests.

AI-assisted: found and reproduced with an AI coding agent; the example above was run locally.

Langage dominant
Python
Étoiles
87.2k
Forks
9.1k
Merge moyen
1 j 6 h
PR mergées (30 j)
25

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de unclecode/crawl4ai

Toutes les issues de unclecode/crawl4ai

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.