protego
Pure-Python robots.txt parser from the Scrapy project that follows RFC 9309, for testing which user agents a site allows.
An unrated entity, not a level-rated lesson. A repo from Scrapy project; catalog added 2026-10-19, last verified 2026-10-19.
- Category
- AI search visibility (SEO/GEO/AEO)
- Tags
- ai-crawler-access-and-robots, ai-search-visibility-seo-geo-aeo
- Added
- 2026-10-19
- Last verified
- 2026-10-19
- Status
- ok
- Resource kind
- entity
- Type
- repo
- Original language
- en
- Access
- free
- Last modified
- 2026-10-19
- Subcategory
- AI crawler access and robots
- Verification basis
- Supplier-reported import check. Subsequent source corrections and content reviews are recorded separately.
- Access evidence
- Supplier-approved free inventory; not independently verified by this site.
- Source domain
- AI search visibility (SEO/GEO/AEO)
- Source subdomain
- AI crawler access and robots
Summary in our own words. Rights remain with the original creator. Credit and correction policy.