--- id: "/tags/agent-readable-web-standards/" title: "Agent Readable Web Standards" creator: null creator_url: null creator_x: null url: "https://skillissue.dev/tags/agent-readable-web-standards/" level: null level_max: null category: null subcategory: null region: null license: null license_scope: null access: null models: null source_date: null date_added: null resource_kind: null verification_basis: null access_evidence: null domain_top: null domain_sub: null last_verified: null --- # Agent Readable Web Standards [Beginner](/levels/beginner/index.md) [Intermediate](/levels/intermediate/index.md) [Advanced](/levels/advanced/index.md) [Expert](/levels/expert/index.md) 18 resources · page 1 of 1 - ### [Does Anthropic crawl data from the web, and how can site owners block the crawler?](/agents/agent-design-patterns/anthropic-crawler-info/index.md) Creator not supplied; see original source. Anthropic support article on its web crawler and how site owners can block it. Not applicable (entity) · unknown · unknown resource · en - ### [Canonical URLs](/agents/agent-design-patterns/canonical-urls/index.md) Creator not supplied; see original source. Google Search Central page on specifying canonical URLs with rel="canonical" and other methods. Not applicable (entity) · unknown · unknown resource · en - ### [Google AI features and your website](/agents/agent-design-patterns/google-ai-features-and-your-website/index.md) Creator not supplied; see original source. Google Search Central page: how AI features such as AI Overviews use and show website content. Not applicable (entity) · unknown · unknown resource · en - ### [Google Dataset structured data](/agents/agent-design-patterns/google-dataset-structured-data/index.md) Creator not supplied; see original source. Google Search Central page: Dataset structured data markup for appearing in Dataset Search. Not applicable (entity) · unknown · unknown resource · en - ### [Google sitemaps overview](/agents/agent-design-patterns/google-sitemaps-overview/index.md) Creator not supplied; see original source. Google Search Central overview of sitemaps: what they are and when a site needs one. Not applicable (entity) · unknown · unknown resource · en - ### [Google structured data intro](/agents/agent-design-patterns/google-structured-data-intro/index.md) Creator not supplied; see original source. Google Search Central introduction to structured data markup and how Search uses it. Not applicable (entity) · unknown · unknown resource · en - ### [IndexNow](/agents/agent-design-patterns/indexnow/index.md) Creator not supplied; see original source. Bing IndexNow page: protocol for notifying search engines of new and changed URLs. Not applicable (entity) · unknown · unknown resource · en - ### [JSON Feed](/agents/agent-design-patterns/json-feed/index.md) Creator not supplied; see original source. JSON Feed version 1.1 specification: JSON-based syndication format similar to RSS and Atom. Not applicable (entity) · unknown · unknown resource · und - ### [JSON-LD 1.1](/agents/agent-design-patterns/json-ld-1-1/index.md) Creator not supplied; see original source. W3C JSON-LD 1.1 recommendation: JSON-based format for serialising linked data. Not applicable (entity) · unknown · unknown resource · en - ### [JSON Schema](/agents/agent-design-patterns/json-schema/index.md) Creator not supplied; see original source. JSON Schema site: specification and tooling for describing and validating JSON data. Not applicable (entity) · unknown · unknown resource · en - ### [llms.txt proposal](/agents/agent-design-patterns/llms-txt-proposal/index.md) Creator not supplied; see original source. llms.txt proposal: a standard /llms.txt file giving agents concise information about a website. Not applicable (entity) · unknown · unknown resource · en - ### [OpenAI crawlers (GPTBot, OAI-SearchBot)](/agents/agent-design-patterns/openai-crawlers-gptbot-oai-searchbot/index.md) Creator not supplied; see original source. OpenAI documentation on its web crawlers GPTBot and OAI-SearchBot and how to control them. Not applicable (entity) · unknown · unknown resource · en - ### [RFC 9309 robots.txt](/agents/agent-design-patterns/rfc-9309-robots-txt/index.md) Creator not supplied; see original source. RFC 9309: the Robots Exclusion Protocol specification for robots.txt. Not applicable (entity) · unknown · unknown resource · en - ### [schema.org Dataset](/agents/agent-design-patterns/schema-org-dataset/index.md) Creator not supplied; see original source. schema.org Dataset type: vocabulary for describing datasets in structured data. Not applicable (entity) · unknown · unknown resource · en - ### [schema.org DefinedTerm](/agents/agent-design-patterns/schema-org-definedterm/index.md) Creator not supplied; see original source. schema.org DefinedTerm type: vocabulary for terms with formal definitions in a glossary. Not applicable (entity) · unknown · unknown resource · en - ### [schema.org ItemList](/agents/agent-design-patterns/schema-org-itemlist/index.md) Creator not supplied; see original source. schema.org ItemList type: vocabulary for ordered or unordered lists of items. Not applicable (entity) · unknown · unknown resource · en - ### [schema.org LearningResource](/agents/agent-design-patterns/schema-org-learningresource/index.md) Creator not supplied; see original source. schema.org LearningResource type: vocabulary for describing educational resources. Not applicable (entity) · unknown · unknown resource · en - ### [Data Catalog Vocabulary (DCAT) Version 3](/agents/agent-design-patterns/w3c-dcat-3-data-catalog-vocabulary/index.md) Creator not supplied; see original source. W3C DCAT version 3: RDF vocabulary for describing data catalogues. Not applicable (entity) · unknown · unknown resource · en [Plain-text Markdown copy](/tags/agent-readable-web-standards/index.md)