Agent Readable Web Standards

18 resources · page 1 of 1

• Does Anthropic crawl data from the web, and how can site owners block the crawler?

  Creator not supplied; see original source.

  Anthropic support article on its web crawler and how site owners can block it.

  Not applicable (entity) · unknown · unknown resource · en

• Canonical URLs

  Creator not supplied; see original source.

  Google Search Central page on specifying canonical URLs with rel="canonical" and other methods.

  Not applicable (entity) · unknown · unknown resource · en

• Google AI features and your website

  Creator not supplied; see original source.

  Google Search Central page: how AI features such as AI Overviews use and show website content.

  Not applicable (entity) · unknown · unknown resource · en

• Google Dataset structured data

  Creator not supplied; see original source.

  Google Search Central page: Dataset structured data markup for appearing in Dataset Search.

  Not applicable (entity) · unknown · unknown resource · en

• Google sitemaps overview

  Creator not supplied; see original source.

  Google Search Central overview of sitemaps: what they are and when a site needs one.

  Not applicable (entity) · unknown · unknown resource · en

• Google structured data intro

  Creator not supplied; see original source.

  Google Search Central introduction to structured data markup and how Search uses it.

  Not applicable (entity) · unknown · unknown resource · en

• IndexNow

  Creator not supplied; see original source.

  Bing IndexNow page: protocol for notifying search engines of new and changed URLs.

  Not applicable (entity) · unknown · unknown resource · en

• JSON Feed

  Creator not supplied; see original source.

  JSON Feed version 1.1 specification: JSON-based syndication format similar to RSS and Atom.

  Not applicable (entity) · unknown · unknown resource · und

• JSON-LD 1.1

  Creator not supplied; see original source.

  W3C JSON-LD 1.1 recommendation: JSON-based format for serialising linked data.

  Not applicable (entity) · unknown · unknown resource · en

• JSON Schema

  Creator not supplied; see original source.

  JSON Schema site: specification and tooling for describing and validating JSON data.

  Not applicable (entity) · unknown · unknown resource · en

• llms.txt proposal

  Creator not supplied; see original source.

  llms.txt proposal: a standard /llms.txt file giving agents concise information about a website.

  Not applicable (entity) · unknown · unknown resource · en

• OpenAI crawlers (GPTBot, OAI-SearchBot)

  Creator not supplied; see original source.

  OpenAI documentation on its web crawlers GPTBot and OAI-SearchBot and how to control them.

  Not applicable (entity) · unknown · unknown resource · en

• RFC 9309 robots.txt

  Creator not supplied; see original source.

  RFC 9309: the Robots Exclusion Protocol specification for robots.txt.

  Not applicable (entity) · unknown · unknown resource · en

• schema.org Dataset

  Creator not supplied; see original source.

  schema.org Dataset type: vocabulary for describing datasets in structured data.

  Not applicable (entity) · unknown · unknown resource · en

• schema.org DefinedTerm

  Creator not supplied; see original source.

  schema.org DefinedTerm type: vocabulary for terms with formal definitions in a glossary.

  Not applicable (entity) · unknown · unknown resource · en

• schema.org ItemList

  Creator not supplied; see original source.

  schema.org ItemList type: vocabulary for ordered or unordered lists of items.

  Not applicable (entity) · unknown · unknown resource · en

• schema.org LearningResource

  Creator not supplied; see original source.

  schema.org LearningResource type: vocabulary for describing educational resources.

  Not applicable (entity) · unknown · unknown resource · en

• Data Catalog Vocabulary (DCAT) Version 3

  Creator not supplied; see original source.

  W3C DCAT version 3: RDF vocabulary for describing data catalogues.

  Not applicable (entity) · unknown · unknown resource · en

Plain-text Markdown copy