OpenAI will honor a shared robots.txt signal for AI training preferences
OpenAI says it will honor a common signal in robots.txt, the file that lets websites express their preferences about the use of their content for foundational generative AI training. The shared signal lets publishers state one preference that multiple participating providers can follow, instead of maintaining separate rules for each crawler. OpenAI says it aims to implement the signal in the coming months and that its existing robots.txt crawler controls will still apply. The company is joining other participants in the IETF AI Preferences working group and encourages other AI providers to honor the signal.