Four of the largest AI developers are lining up behind a single way for websites to say no to AI training. On October 30, OpenAI said it would respect a shared robots.txt signal about using web content to train foundation generative models, and Google published a matching plan the same day. Search Engine Land reported that Microsoft and Anthropic are part of the same effort, which grows out of the Internet Engineering Task Force's AI Preferences (AIPREF) working group.
Until now a publisher that wanted to keep its pages out of training sets had to know every relevant crawler by name and write separate rules for each one, such as GPTBot, ClaudeBot or Google's Google-Extended token. The new approach replaces that list with one shared entry. Instead of naming a bot, the site names a purpose: an Internet-Draft published on October 30 defines a policy token, __usage_ai-training, that goes in the User-Agent line of an ordinary robots.txt group. Disallow rules under that token express a preference against training on those paths, and Allow rules express consent, so a site could refuse training everywhere except a press-kit folder.
The draft is careful to separate this from crawling. Per its text, the token says nothing about whether a bot may fetch a page; existing crawler rules keep working exactly as before, a catch-all User-Agent: * group must not be read as covering the token, and private licensing agreements may override what the file says. It adds no new directives or syntax to the robots.txt standard, RFC 9309, so older parsers simply skip the new groups. The document lists Kevin Kelley of Anthropic as its author and is an individual submission, which the IETF datatracker notes has no formal standing in the standards process.
Each company described its own rollout. OpenAI plans to put the signal into effect over the next few months and says its current crawler controls stay in force. Google said that when a site opts out through the shared token and does not mention Google-Extended, it will treat the site as having blocked Google-Extended, while a site that explicitly allows Google-Extended stays eligible for grounding and clicks in Gemini Apps. Google expects to support the control alongside other providers in 2027 and argued that the approach fits the EU Code of Practice for general-purpose AI models under the EU AI Act, which treats robots.txt as a primary machine-readable opt-out.
Microsoft went further for its own services. According to Search Engine Land's summary of the Bing announcement, Microsoft will offer separate AI Training and AI Grounding preferences, extend the training choice to individual pages through a meta tag or HTTP header, and add a Limited Grounding option that permits use in its own products such as Copilot and Microsoft 365 but not in grounding services sold to third parties. Bing plans to begin honoring the new preferences on February 1, 2027 and will stop reading the older NoArchive and NoCache directives as training or grounding preferences after March 1, 2027.
For publishers, the practical gain is less maintenance and a choice that reaches every participating provider at once, including crawlers they may not have heard of, provided those providers adopt the token. Google said the proposal reflects pressure from publishers, civil society and regulators for an industry-wide control, and OpenAI urged other AI companies to honor it.
The limits are significant. Compliance is voluntary, no participant honors the token today, and the wider AIPREF vocabulary is still under discussion, so the token name and semantics could change before anything becomes an IETF standard. The signal also applies to training only: whether content can be used to ground live AI answers is handled separately, and so far only Microsoft has announced a dedicated control for that.