agent-readable web standards: 18 curated resources

agent-readable web standards

18 resources · page 1 of 1

• Does Anthropic crawl data from the web, and how can site owners block the crawler?

  by Anthropic

  Anthropic support article on its web crawler and how site owners can block it.

  entity standard-policy West en

• Canonical URLs

  by Google

  Google Search Central page on specifying canonical URLs with rel="canonical" and other methods.

  entity standard-policy West en

• Google AI features and your website

  by Google

  Google Search Central page: how AI features such as AI Overviews use and show website content.

  entity standard-policy West en

• Google Dataset structured data

  by Google

  Google Search Central page: Dataset structured data markup for appearing in Dataset Search.

  entity standard-policy West en

• Google sitemaps overview

  by Google

  Google Search Central overview of sitemaps: what they are and when a site needs one.

  entity standard-policy West en

• Google structured data intro

  by Google

  Google Search Central introduction to structured data markup and how Search uses it.

  entity standard-policy West en

• IndexNow

  by Microsoft Bing

  Bing IndexNow page: protocol for notifying search engines of new and changed URLs.

  entity standard-policy West en

• JSON Feed

  by Manton Reece and Brent Simmons (JSON Feed)

  JSON Feed version 1.1 specification: JSON-based syndication format similar to RSS and Atom.

  entity standard-policy West en

• JSON-LD 1.1

  by W3C

  W3C JSON-LD 1.1 recommendation: JSON-based format for serialising linked data.

  entity standard-policy West en

• JSON Schema

  by JSON Schema project

  JSON Schema site: specification and tooling for describing and validating JSON data.

  entity standard-policy West en

• llms.txt proposal

  by Jeremy Howard (llms.txt proposal)

  llms.txt proposal: a standard /llms.txt file giving agents concise information about a website.

  entity standard-policy West en

• OpenAI crawlers (GPTBot, OAI-SearchBot)

  by OpenAI

  OpenAI documentation on its web crawlers GPTBot and OAI-SearchBot and how to control them.

  entity standard-policy West en

• RFC 9309 robots.txt

  by IETF (RFC Editor)

  RFC 9309: the Robots Exclusion Protocol specification for robots.txt.

  entity standard-policy West en

• schema.org Dataset

  by Schema.org

  schema.org Dataset type: vocabulary for describing datasets in structured data.

  entity standard-policy West en

• schema.org DefinedTerm

  by Schema.org

  schema.org DefinedTerm type: vocabulary for terms with formal definitions in a glossary.

  entity standard-policy West en

• schema.org ItemList

  by Schema.org

  schema.org ItemList type: vocabulary for ordered or unordered lists of items.

  entity standard-policy West en

• schema.org LearningResource

  by Schema.org

  schema.org LearningResource type: vocabulary for describing educational resources.

  entity standard-policy West en

• Data Catalog Vocabulary (DCAT) Version 3

  by W3C

  W3C DCAT version 3: RDF vocabulary for describing data catalogues.

  entity standard-policy West en

Plain-text Markdown copy