AI training opt-outs converge on a single robots.txt signal

Summary

Until now, websites had to write separate robots.txt rules for each AI crawler. On October 30, OpenAI and Google each announced on their official blogs that a shared signal from the IETF AIPREF standards effort will let a site state its AI training preference to multiple providers at once. OpenAI plans to honor the signal within the coming months and Google aims for 2027, while existing company-specific crawler controls continue to work. These timelines are the companies' own commitments, not yet implemented.

Why it matters for our work

Publishers, companies, and research institutions can cut much of the repeated work of managing AI crawler policies. Teams handling content and internal documents will find it simpler to set a clear boundary for what AI training may use.

Translated from the Korean original. Summaries may be translated and edited. Commentary reflects our perspective; forecasts remain the source’s views.