scrapy/protego

A pure-Python robots.txt parser with support for modern conventions.

DIGITAL Command LanguageStars 74Forks 29Watchers 74Open issues 7License BSD 3-Clause "New" or "Revised" License

Details

仓库信息

Ownerscrapy

Homepage—

Last pushed2025-12-09

Last updated2025-12-15

Issues fetched at—

Community at a glance