--- id: "/tags/ai-crawler-access-and-robots/" title: "Ai Crawler Access And Robots" url: "https://skillissue.dev/tags/ai-crawler-access-and-robots/" --- [Home](/index.md) # Ai Crawler Access And Robots Ai Crawler Access And Robots brings together 15 resources with credited original sources. Use prerequisite levels to find learning material; unrated entities remain available in the full collection. [Beginner](/levels/beginner/index.md) [Intermediate](/levels/intermediate/index.md) [Advanced](/levels/advanced/index.md) [Expert](/levels/expert/index.md) 15 resources · page 1 of 1 - ### [ai.robots.txt](/e/ai-robots-txt/index.md) by [ai.robots.txt](https://github.com/ai-robots-txt) Community list of AI crawler user agents with ready-made robots.txt, .htaccess and web server blocking rules generated from one source file. entity repo West en - ### [indexnow-action](/e/bojieyang-indexnow-action/index.md) by [Bojie Yang](https://github.com/bojieyang) GitHub Action that submits a site's URLs to IndexNow-participating search engines from a deploy workflow. entity repo en - ### [Content Signals](/e/content-signals/index.md) by [Cloudflare](https://www.cloudflare.com) Robots.txt extension with search, ai-input and ai-train signals that state how a site's content may be used after it is crawled. entity standard-policy West en - ### [AI Preferences (aipref)](/e/ietf-aipref/index.md) by [IETF](https://www.ietf.org) IETF working group drafting a shared vocabulary, attachable through robots.txt and HTTP headers, for stating whether content may be used for AI training or search. entity standard-policy en - ### [indexnow-wordpress-plugin](/e/microsoft-indexnow-wordpress-plugin/index.md) by [Microsoft](https://github.com/microsoft) Microsoft's WordPress plugin that hosts an IndexNow key and submits URLs to participating engines when posts are created, updated or deleted. entity repo West en - ### [RSL: Really Simple Licensing](/e/rsl-standard/index.md) by [RSL Collective](https://rslstandard.org) Open standard for publishing machine-readable licensing and payment terms for AI crawlers, linked from robots.txt, with an optional nonprofit collective. entity standard-policy West en - ### [protego](/e/scrapy-protego/index.md) by [Scrapy project](https://github.com/scrapy) Pure-Python robots.txt parser from the Scrapy project that follows RFC 9309, for testing which user agents a site allows. entity repo en - ### [About Applebot](/e/applebot/index.md) by [Apple](https://www.apple.com) Apple's page on Applebot and Applebot-Extended, showing how to keep content out of Apple model training while staying crawlable for Siri and Spotlight. Intermediate–Advanced learning_resource docs West en - ### [Bing Webmaster Guidelines](/e/bing-webmaster-guidelines/index.md) by [Microsoft](https://www.microsoft.com) Bing's rules for crawling, indexing and ranking, which also set eligibility for Copilot answers, grounding API results and citations. Intermediate–Advanced learning_resource docs West en - ### [Cloudflare AI Crawl Control](/e/cloudflare-ai-crawl-control/index.md) by [Cloudflare](https://www.cloudflare.com) Cloudflare docs for seeing which AI crawlers visit a site and allowing, blocking or charging each one at the network edge. Intermediate–Advanced learning_resource docs West en - ### [Introducing pay per crawl](/e/cloudflare-pay-per-crawl/index.md) by [Cloudflare](https://www.cloudflare.com) Cloudflare's July 2025 post on pay per crawl, where site owners set a per-request price and AI crawlers get a payment-required response until they pay. Intermediate learning_resource blog West en - ### [Common Crawl CCBot](/e/common-crawl-ccbot/index.md) by [Common Crawl Foundation](https://commoncrawl.org) Common Crawl's page on CCBot, its user agent and robots.txt handling; the Common Crawl archive is a frequent source of LLM training text. Intermediate–Advanced learning_resource docs West en - ### [Google's common crawlers](/e/google-common-crawlers/index.md) by [Google](https://developers.google.com) Google's list of its crawlers and robots.txt tokens, including Google-Extended, which controls Gemini training and grounding use without affecting Search. Intermediate–Advanced learning_resource docs West en - ### [Search generative AI control](/e/google-search-generative-ai-control/index.md) by [Google](https://developers.google.com) Search Console setting, rolled out worldwide in August 2026, that includes or excludes a site from AI Overviews, AI Mode and Discover generative features. Intermediate–Advanced learning_resource docs West en - ### [Perplexity Crawlers](/e/perplexity-crawlers/index.md) by [Perplexity](https://www.perplexity.ai) Perplexity's page on PerplexityBot and Perplexity-User, with their user-agent strings, published IP ranges and how robots.txt applies to each. Intermediate–Advanced learning_resource docs West en [Plain-text Markdown copy](/tags/ai-crawler-access-and-robots/index.md)