---
id: "scrapy-protego"
title: "protego"
creator: "Scrapy project"
creator_url: "https://github.com/scrapy"
url: "https://github.com/scrapy/protego"
category: "ai-search-visibility-seo-geo-aeo"
subcategory: "ai-crawler-access-and-robots"
access: "free"
date_added: "2026-10-19"
resource_kind: "entity"
verification_basis: "supplier-inventory-v2.4"
access_evidence: "Supplier-approved free inventory; not independently verified by this site."
domain_top: "AI search visibility (SEO/GEO/AEO)"
domain_sub: "AI crawler access and robots"
source_type: "repo"
last_verified: "2026-10-19"
last_content_modified: "2026-10-19"
---

[Home](/index.md) · [AI search visibility (SEO/GEO/AEO)](/ai-search-visibility-seo-geo-aeo/index.md) · [AI crawler access and robots](/ai-search-visibility-seo-geo-aeo/ai-crawler-access-and-robots/index.md)

# protego

Pure-Python robots.txt parser from the Scrapy project that follows RFC 9309, for testing which user agents a site allows.

An unrated entity, not a level-rated lesson. A repo from Scrapy project; catalog added 2026-10-19, last verified 2026-10-19.

by [Scrapy project](https://github.com/scrapy)

[Read the original →](https://github.com/scrapy/protego)

Category: [AI search visibility (SEO/GEO/AEO)](/ai-search-visibility-seo-geo-aeo/index.md)
Tags: [ai-crawler-access-and-robots](/tags/ai-crawler-access-and-robots/index.md), [ai-search-visibility-seo-geo-aeo](/tags/ai-search-visibility-seo-geo-aeo/index.md)
Added: 2026-10-19
Last verified: 2026-10-19
Status: ok
Resource kind: entity
Type: repo
Original language: en
Access: free
Last modified: 2026-10-19
Subcategory: [AI crawler access and robots](/ai-search-visibility-seo-geo-aeo/ai-crawler-access-and-robots/index.md)
Verification basis: Supplier-reported import check. Subsequent source corrections and content reviews are recorded separately.
Access evidence: Supplier-approved free inventory; not independently verified by this site.
Source domain: AI search visibility (SEO/GEO/AEO)
Source subdomain: AI crawler access and robots

Summary in our own words. Rights remain with the original creator. [Credit and correction policy](/about/index.md#credit-policy).

[Plain-text Markdown copy](/e/scrapy-protego/index.md)