OpenAI to Adopt a Common robots.txt Signal for Training

OpenAI announces that it will respect a common signal in robots.txt to express training preferences for generative models. The implementation is planned for the coming months and will not replace the controls already offered to websites, aligning with the ongoing work of the dedicated IETF group.

Timeline, Compatibility, and IETF Framework Announced by OpenAI

OpenAI plans to implement the common signal in the coming months and states that a detailed update regarding this implementation will be communicated later. This new signal will not replace the controls already provided via robots.txt, which will continue to apply and be respected. The initiative is part of the ongoing work of the "AI Preferences" working group of the Internet Engineering Task Force (IETF). OpenAI joins other members of the group to announce this shared signal and specifies that it will adapt its practices as discussions and context evolve, to ensure appropriate controls and fairness.

What the Signal Changes for Publishers and Website Operators

The common signal in robots.txt will allow websites to indicate their preferences regarding the use of their content for training foundational generative AIs. According to OpenAI, this mechanism will simplify the communication of the same preferences to different participating providers simultaneously. Until now, website operators were required to identify each crawler separately and manage rules specific to each provider. OpenAI believes that a shared signal among multiple providers alleviates this task. The company describes this initiative as a voluntary act.

OpenAI Invokes the Spirit of the Open Web and Calls on Other Stakeholders

OpenAI views this initiative as a practical step toward an open web, where everyone can publish and discover information while providing website owners with clearer and simpler means to communicate their choices regarding the use of their content. The company encourages other AI providers to respect this common signal and asserts that this announcement reinforces its ongoing participation in discussions about IETF standards.