AI crawler access and robots: 15 curated resources
AI crawler access and robots: 15 curated resources brings together 15 resources with credited original sources. Use prerequisite levels to find learning material; unrated entities remain available in the full collection.
AI crawler access and robots
15 resources · page 1 of 1
ai.robots.txt
Community list of AI crawler user agents with ready-made robots.txt, .htaccess and web server blocking rules generated from one source file.
indexnow-action
by Bojie Yang
GitHub Action that submits a site's URLs to IndexNow-participating search engines from a deploy workflow.
Content Signals
by Cloudflare
Robots.txt extension with search, ai-input and ai-train signals that state how a site's content may be used after it is crawled.
AI Preferences (aipref)
by IETF
IETF working group drafting a shared vocabulary, attachable through robots.txt and HTTP headers, for stating whether content may be used for AI training or search.
indexnow-wordpress-plugin
by Microsoft
Microsoft's WordPress plugin that hosts an IndexNow key and submits URLs to participating engines when posts are created, updated or deleted.
RSL: Really Simple Licensing
Open standard for publishing machine-readable licensing and payment terms for AI crawlers, linked from robots.txt, with an optional nonprofit collective.
protego
Pure-Python robots.txt parser from the Scrapy project that follows RFC 9309, for testing which user agents a site allows.
About Applebot
by Apple
Apple's page on Applebot and Applebot-Extended, showing how to keep content out of Apple model training while staying crawlable for Siri and Spotlight.
Bing Webmaster Guidelines
by Microsoft
Bing's rules for crawling, indexing and ranking, which also set eligibility for Copilot answers, grounding API results and citations.
Cloudflare AI Crawl Control
by Cloudflare
Cloudflare docs for seeing which AI crawlers visit a site and allowing, blocking or charging each one at the network edge.
Introducing pay per crawl
by Cloudflare
Cloudflare's July 2025 post on pay per crawl, where site owners set a per-request price and AI crawlers get a payment-required response until they pay.
Common Crawl CCBot
Common Crawl's page on CCBot, its user agent and robots.txt handling; the Common Crawl archive is a frequent source of LLM training text.
Google's common crawlers
by Google
Google's list of its crawlers and robots.txt tokens, including Google-Extended, which controls Gemini training and grounding use without affecting Search.
Search generative AI control
by Google
Search Console setting, rolled out worldwide in August 2026, that includes or excludes a site from AI Overviews, AI Mode and Discover generative features.
Perplexity Crawlers
by Perplexity
Perplexity's page on PerplexityBot and Perplexity-User, with their user-agent strings, published IP ranges and how robots.txt applies to each.