[Bug]: normalize_url lowercases the userinfo in a URL, so credentials in a link are rewritten
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 2/5
- Tempo stimato
- 1-3 ore
- Idoneità per principianti
- 20/100
Direzione di ricerca
Read the three normalizers named in the issue: normalize_url, normalize_url_for_deep_crawl, and efficient_normalize_url_for_deep_crawl. Check how each handles parsed.netloc and add regression coverage for preserved userinfo casing and lowercase hostnames. The issue says the author has a patch and six tests ready, and linked PR #2530 is open.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Describe the bug
normalize_url, normalize_url_for_deep_crawl and efficient_normalize_url_for_deep_crawl all lowercase the whole netloc:
netloc = parsed.netloc.lower()
netloc also carries the user:password@ part of a URL. A hostname is case-insensitive, so lowering it is right; credentials are not, so the same call silently rewrites them. The comment on the efficient normalizer even says what was intended — # 1. Lowercase hostname.
Steps to Reproduce
from crawl4ai.utils import normalize_url, normalize_url_for_deep_crawl
base = "https://example.com/dir/page"
print(normalize_url("https://User:[email protected]/path", base))
print(normalize_url_for_deep_crawl("https://User:[email protected]/path", base))
Actual:
https://user:[email protected]/path
https://user:[email protected]/path
Expected:
https://User:[email protected]/path
https://User:[email protected]/path
Why it matters
quick_extract_links resolves every <a href> through normalize_url_for_deep_crawl, so any link carrying credentials in its URL is rewritten before the crawl follows it. A page that links to https://user:SecRet@host/ is then fetched with user:secret, and the server answers 401 — or, worse, authenticates as a different account on systems where the userinfo is an identifier rather than a password. The same happens to URLs assembled by hand and passed to the deep-crawl path.
Environment
crawl4aimainatcae74b2- Python 3.13, Windows
- No browser involved: the three functions are pure string handling
Proposed patch
Lowercase only the host, keeping whatever precedes the @ as written:
def _lowercase_host(netloc: str) -> str:
userinfo, separator, hostport = netloc.rpartition('@')
return f"{userinfo}{separator}{hostport.lower()}"
rpartition keeps a %40-encoded @ inside the password intact, since urlparse leaves it encoded in the netloc. Used at the three call sites, the diff is one line each.
Behaviour for URLs without userinfo is unchanged, including the port: Example.COM:8080 still becomes example.com:8080.
I have the patch and six regression tests ready (userinfo casing, host-only casing, userinfo with port, encoded @ in the password, and the two deep-crawl normalizers) and will open the PR against this issue.
- Lingua principale
- Python
- Stelle
- 87.2k
- Fork
- 9.1k
- Merge medio
- 1g 6h
- PR unite (30g)
- 25
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackForse già presa @HhH666-ux l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
I maintainer di solito rispondono entro 1 giorno
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configForse già presa Una pull request collegata a questa issue è aperta o già unita. Aperta🐞 Bug 📌 Root caused
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
unclecode/crawl4ai#2494 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresAperta⚙ Done ✨ Enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
I maintainer di solito rispondono entro 1 giorno
-
⚙ Done 🐞 Bug 📌 Root caused
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 1 giorno
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaAperta⚙ Done 🐞 Bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
unclecode/crawl4ai#2469 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di unclecode/crawl4ai
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 73/100
Netcracker/qubership-ai-packages#266 ·
I maintainer di solito rispondono entro 1 giorno
-
PROCESSED_DATA_FOLDER: strip whitespace and treat empty as unsetForse già presa @TorecLuik l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
NL-BioImaging/BIOMERO.importer#63 ·
I maintainer di solito rispondono entro 1 giorno
-
MLX training-complete chat/merge suggestions omit --backend mlxForse già presa @vaputa l’ha presa oggi. Apertaneeds-triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
MakazhanAlpamys/Soup#2913 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
area:security effort:S priority:P3 status:needs-triage type:chore
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
randomparity/kdive#3657 ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
aws/mcp-proxy-for-aws#476 ·
I maintainer di solito rispondono entro 1 giorno