[Bug]: normalize_url lowercases the userinfo in a URL, so credentials in a link are rewritten

Aperta
#2,529 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Steve-20-ai ci sta già lavorando.

Dal 10/11/2026.

  • #2530 di @Steve-20-ai — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
20/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Read the three normalizers named in the issue: normalize_url, normalize_url_for_deep_crawl, and efficient_normalize_url_for_deep_crawl. Check how each handles parsed.netloc and add regression coverage for preserved userinfo casing and lowercase hostnames. The issue says the author has a patch and six tests ready, and linked PR #2530 is open.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

normalize_url, normalize_url_for_deep_crawl and efficient_normalize_url_for_deep_crawl all lowercase the whole netloc:

netloc = parsed.netloc.lower()

netloc also carries the user:password@ part of a URL. A hostname is case-insensitive, so lowering it is right; credentials are not, so the same call silently rewrites them. The comment on the efficient normalizer even says what was intended — # 1. Lowercase hostname.

Steps to Reproduce

from crawl4ai.utils import normalize_url, normalize_url_for_deep_crawl

base = "https://example.com/dir/page"
print(normalize_url("https://User:[email protected]/path", base))
print(normalize_url_for_deep_crawl("https://User:[email protected]/path", base))

Actual:

https://user:[email protected]/path
https://user:[email protected]/path

Expected:

https://User:[email protected]/path
https://User:[email protected]/path

Why it matters

quick_extract_links resolves every <a href> through normalize_url_for_deep_crawl, so any link carrying credentials in its URL is rewritten before the crawl follows it. A page that links to https://user:SecRet@host/ is then fetched with user:secret, and the server answers 401 — or, worse, authenticates as a different account on systems where the userinfo is an identifier rather than a password. The same happens to URLs assembled by hand and passed to the deep-crawl path.

Environment

  • crawl4ai main at cae74b2
  • Python 3.13, Windows
  • No browser involved: the three functions are pure string handling

Proposed patch

Lowercase only the host, keeping whatever precedes the @ as written:

def _lowercase_host(netloc: str) -> str:
    userinfo, separator, hostport = netloc.rpartition('@')
    return f"{userinfo}{separator}{hostport.lower()}"

rpartition keeps a %40-encoded @ inside the password intact, since urlparse leaves it encoded in the netloc. Used at the three call sites, the diff is one line each.

Behaviour for URLs without userinfo is unchanged, including the port: Example.COM:8080 still becomes example.com:8080.

I have the patch and six regression tests ready (userinfo casing, host-only casing, userinfo with port, encoded @ in the password, and the two deep-crawl normalizers) and will open the PR against this issue.

Lingua principale
Python
Stelle
87.2k
Fork
9.1k
Merge medio
1g 6h
PR unite (30g)
25

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.