--- id: "/agents-ai-engineering/agent-readable-web-standards/" title: "agent-readable web standards: 18 curated resources" url: "https://skillissue.dev/agents-ai-engineering/agent-readable-web-standards/" --- # agent-readable web standards: 18 curated resources agent-readable web standards [Beginner](/agents-ai-engineering/beginner/index.md) [Intermediate](/agents-ai-engineering/intermediate/index.md) [Advanced](/agents-ai-engineering/advanced/index.md) [Expert](/agents-ai-engineering/expert/index.md) 18 resources ยท page 1 of 1 - ### [Does Anthropic crawl data from the web, and how can site owners block the crawler?](/agents/agent-design-patterns/anthropic-crawler-info/index.md) by [Anthropic](https://www.anthropic.com/) Anthropic support article on its web crawler and how site owners can block it. entity standard-policy West en - ### [Canonical URLs](/agents/agent-design-patterns/canonical-urls/index.md) by [Google](https://developers.google.com/) Google Search Central page on specifying canonical URLs with rel="canonical" and other methods. entity standard-policy West en - ### [Google AI features and your website](/agents/agent-design-patterns/google-ai-features-and-your-website/index.md) by [Google](https://developers.google.com/) Google Search Central page: how AI features such as AI Overviews use and show website content. entity standard-policy West en - ### [Google Dataset structured data](/agents/agent-design-patterns/google-dataset-structured-data/index.md) by [Google](https://developers.google.com/) Google Search Central page: Dataset structured data markup for appearing in Dataset Search. entity standard-policy West en - ### [Google sitemaps overview](/agents/agent-design-patterns/google-sitemaps-overview/index.md) by [Google](https://developers.google.com/) Google Search Central overview of sitemaps: what they are and when a site needs one. entity standard-policy West en - ### [Google structured data intro](/agents/agent-design-patterns/google-structured-data-intro/index.md) by [Google](https://developers.google.com/) Google Search Central introduction to structured data markup and how Search uses it. entity standard-policy West en - ### [IndexNow](/agents/agent-design-patterns/indexnow/index.md) by [Microsoft Bing](https://www.bing.com/) Bing IndexNow page: protocol for notifying search engines of new and changed URLs. entity standard-policy West en - ### [JSON Feed](/agents/agent-design-patterns/json-feed/index.md) by [Manton Reece and Brent Simmons (JSON Feed)](https://www.jsonfeed.org/) JSON Feed version 1.1 specification: JSON-based syndication format similar to RSS and Atom. entity standard-policy West en - ### [JSON-LD 1.1](/agents/agent-design-patterns/json-ld-1-1/index.md) by [W3C](https://www.w3.org/) W3C JSON-LD 1.1 recommendation: JSON-based format for serialising linked data. entity standard-policy West en - ### [JSON Schema](/agents/agent-design-patterns/json-schema/index.md) by [JSON Schema project](https://json-schema.org/) JSON Schema site: specification and tooling for describing and validating JSON data. entity standard-policy West en - ### [llms.txt proposal](/agents/agent-design-patterns/llms-txt-proposal/index.md) by [Jeremy Howard (llms.txt proposal)](https://llmstxt.org/) llms.txt proposal: a standard /llms.txt file giving agents concise information about a website. entity standard-policy West en - ### [OpenAI crawlers (GPTBot, OAI-SearchBot)](/agents/agent-design-patterns/openai-crawlers-gptbot-oai-searchbot/index.md) by [OpenAI](https://openai.com/) OpenAI documentation on its web crawlers GPTBot and OAI-SearchBot and how to control them. entity standard-policy West en - ### [RFC 9309 robots.txt](/agents/agent-design-patterns/rfc-9309-robots-txt/index.md) by [IETF (RFC Editor)](https://www.rfc-editor.org/) RFC 9309: the Robots Exclusion Protocol specification for robots.txt. entity standard-policy West en - ### [schema.org Dataset](/agents/agent-design-patterns/schema-org-dataset/index.md) by [Schema.org](https://schema.org/) schema.org Dataset type: vocabulary for describing datasets in structured data. entity standard-policy West en - ### [schema.org DefinedTerm](/agents/agent-design-patterns/schema-org-definedterm/index.md) by [Schema.org](https://schema.org/) schema.org DefinedTerm type: vocabulary for terms with formal definitions in a glossary. entity standard-policy West en - ### [schema.org ItemList](/agents/agent-design-patterns/schema-org-itemlist/index.md) by [Schema.org](https://schema.org/) schema.org ItemList type: vocabulary for ordered or unordered lists of items. entity standard-policy West en - ### [schema.org LearningResource](/agents/agent-design-patterns/schema-org-learningresource/index.md) by [Schema.org](https://schema.org/) schema.org LearningResource type: vocabulary for describing educational resources. entity standard-policy West en - ### [Data Catalog Vocabulary (DCAT) Version 3](/agents/agent-design-patterns/w3c-dcat-3-data-catalog-vocabulary/index.md) by [W3C](https://www.w3.org/) W3C DCAT version 3: RDF vocabulary for describing data catalogues. entity standard-policy West en [Plain-text Markdown copy](/agents-ai-engineering/agent-readable-web-standards/index.md)