The biggest names in AI are backing a new way to opt out of scraping

Introduction

Publishers have long struggled to control how their content is used by AI models. Each AI company traditionally required a separate request to signal that a site should not be scraped for training data. A new collaborative effort, spearheaded by leading AI players, introduces a single, standardized signal that can be placed in a site’s robots.txt file, allowing publishers to opt out of AI crawling across multiple providers at once.

A unified "policy token" for the web

The proposal, developed within the Internet Engineering Task Force (IETF), adds a policy token to the familiar robots.txt format. When an AI crawler encounters this token, it knows to respect the publisher’s preference and refrain from using the site’s content for model training. The token is designed to be simple to implement—publishers only need to add a line to an existing file that already governs search‑engine indexing and archiving.

Key points of the token:

• Cross‑company applicability – The same signal works for OpenAI, Google, Microsoft, Anthropic, and potentially others.
• Granular control – Companies like Microsoft plan to let sites specify whether they block only training data, only grounding (source verification), or both.
• No impact on search visibility – Sites remain searchable in Bing and Google regardless of their AI‑training preferences, preserving traffic while protecting content.

Industry backing and rollout timeline

OpenAI, Google, Microsoft, and Anthropic have publicly committed to honor the token. Anthropic has already begun honoring it, while Google and Microsoft aim to support it in the coming year. OpenAI expects implementation within the next few months. Meta’s communications head, Andy Stone, expressed support for the concept, though Meta has not yet confirmed whether its crawlers will respect the token.

The move follows earlier attempts to curb unauthorized scraping, such as the Really Simple Licensing (RSL) framework that lets publishers charge crawlers, and Cloudflare’s default blocking of AI bots. By providing a common, low‑friction signal, the new standard hopes to reduce the administrative burden on publishers and create a clearer expectation for AI developers.

Practical steps for publishers

1. Update robots.txt – Add the policy token line as specified by the IETF draft. The exact syntax will be published alongside the standard.
2. Test crawler behavior – Use tools to verify that major AI crawlers respect the token before relying on it fully.
3. Monitor compliance – Keep an eye on crawler logs and community reports to ensure that AI providers are honoring the signal.
4. Stay informed – Follow updates from the IETF and the participating AI companies, as implementation details may evolve.

Conclusion

The shared policy token marks a significant step toward a more transparent and publisher‑friendly web ecosystem. By consolidating opt‑out preferences into a single, widely recognized file, the initiative reduces friction for content creators while giving AI firms a clear, standardized way to respect those preferences. As the standard rolls out, publishers should adopt the new robots.txt directive promptly and monitor its effectiveness, helping shape a web where both human readers and AI systems coexist responsibly.