Robots.txt: A Unique Signal for AI Training

Key Takeaways

1A common token "usage_ai-training" is proposed to control AI training via robots.txt

2Anthropic is already implementing it, while OpenAI, Microsoft, and Google plan to comply by 2027

3The signal only pertains to the use of content, not exploration or display in results

Why it matters : This unique signal simplifies the management of training preferences for publishers in relation to major AI providers.

A draft submitted to the IETF proposes a common identifier in the robots.txt file to accept or refuse the training of AI models. Anthropic is already implementing it, while OpenAI, Microsoft, and Google plan to follow suit according to different timelines. This signal only pertains to the use of content for training.

The signal targets usage, not crawling or display

The new signal in the robots.txt file concerns only the use of content for training AI models. The crawling rules for websites remain unchanged for each bot. Microsoft clarifies that pages will still appear in Bing search results, regardless of the choice expressed for training. The appearance of pages in generated responses, such as AI Overviews or Copilot, depends on other parameters. At Google, an expressed refusal with the common token will lead to the blocking of Google-Extended, unless stated otherwise. Existing blocks against specific bots, such as GPTBot or Google-Extended, will continue to operate separately. If the new line is not added, training remains allowed by default.

Staggered adoption: already active at Anthropic, 2027 for Bing and Google

Anthropic is already applying the signal and plans to gradually extend it to all its services. OpenAI intends to take it into account in the coming months. Microsoft indicates that Bing will respect this signal starting February 1, 2027. Google announces support in 2027, without a specific date. The four providers have announced their intention to comply with this common signal.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

How the "usage_ai-training" token works in robots.txt

The proposal introduces a common token, "usage_ai-training," to be placed in the User-Agent directive of the robots.txt file. A Disallow under this token indicates a refusal for training, an Allow marks an agreement, and both can be combined path by path to refine permissions, for example, by limiting the agreement to a specific folder. The syntax of the robots.txt remains unchanged. The definition of training adopted is that of the AI Preferences group of the IETF, focused on generative models. The text is currently an individual proposal, submitted for discussion by the working group. According to Google, this method aligns with the code of good practices of the AI Act, which considers robots.txt as a way to indicate preferences.

For publishers: a line to add, after years of targeting bot by bot

To benefit from this control, websites must add the new line to their robots.txt file. Until now, publishers had to target each provider separately, such as GPTBot for OpenAI or Google-Extended for Google, or go through intermediaries like Cloudflare.