A Unique Signal in robots.txt to Reject AI Scraping

Four AI giants plan to align on a common signal in robots.txt to express opt-out from scraping. Anthropic is already using it, Google and Microsoft aim for next year, and OpenAI announces implementation in the coming months. Microsoft plans finer adjustments while maintaining visibility in Bing.

Staggered Deployment and Bing Visibility Maintained

Anthropic has already adopted the "policy token." Google and Microsoft plan to support it next year, and OpenAI indicates implementation in the coming months. Microsoft specifies that sites will remain visible in Bing search results, even if their scraping preferences are expressed via this signal.

Commitment to Restrict Crawlers and Specific Options at Microsoft

OpenAI, Google, Microsoft, and Anthropic announce that they will limit access of their AI crawlers to a site if the "policy token" is present. Additionally, Microsoft plans more detailed controls to allow publishers to opt-out either from model training or AI grounding, which involves linking an AI model to a verified information source to reduce hallucinations.

A Shared Token in robots.txt and an Already Defensive Landscape

Kevin Kelley, who serves as the technical standards lead at Anthropic, proposes a shared signal called the "policy token" within the IETF framework. This signal is intended to be added to the robots.txt file, which currently specifies which crawlers are allowed or prohibited from accessing a site for indexing, archiving, or training AI models. Today, publishers still need to express their preferences separately to each AI company, which this signal aims to centralize. This development comes as publishers face an increasing number of AI crawlers extracting their content to train models. Other solutions already exist: the Really Simple Licensing standard allows charging crawlers, and Cloudflare blocks AI crawlers by default.