Google, OpenAI and Microsoft have said they will honour a single line in a website’s robots.txt file that tells AI companies not to train their models on that site’s content. All three announced their support on 30 October, which means a New Zealand business, blogger or publisher will soon be able to state one preference about AI training instead of writing separate rules for each company’s crawler.
The change rests on a short technical proposal lodged with the Internet Engineering Task Force (IETF), the body that develops many of the internet’s underlying standards. The draft, dated 30 October 2026, was written by Kevin Kelley of Anthropic, the company behind the Claude AI models. It defines a new name, __usage_ai-training, that a site owner places in robots.txt, the plain text file at the root of almost every website that tells automated crawlers which pages they may visit.
Until now, a site owner who did not want their work used for AI training had to find out the name of each company’s training crawler and block them one by one. OpenAI uses GPTBot, and its crawler documentation says disallowing it “indicates a site’s content should not be used in training generative AI foundation models.” Google uses a separate token called Google-Extended, which it says it has offered since 2023. Every new AI company meant another line to add.
Under the new approach, a site owner adds a group to robots.txt addressed to __usage_ai-training rather than to any one company’s crawler, then disallows the whole site or just certain folders. The draft lets owners mix rules, so a site could opt out of training everywhere except, for example, a press kit folder it is happy to share.
Google’s announcement, posted on its Search Central blog by Search Advocate John Mueller, says that when a site uses the shared token to opt out and has not specifically mentioned Google-Extended, “Google plans to interpret that as if the publisher had disallowed Google-Extended.” The timing is not immediate. “We aim to support this new control along with other AI providers in 2027, and will provide updates on our progress,” the post says.
OpenAI’s announcement describes the move as a voluntary effort and gives a similarly loose timeframe. “We aim to implement the signal in the coming months, and will share another update with details about our implementation when we do so,” the company says. It adds that its existing GPTBot controls will keep working alongside the new token.
Microsoft has gone furthest and set firm dates. In a Bing blog post, product manager Krishna Madhavan says Microsoft will start honouring the shared training preference, plus a new page-level setting of its own, from 1 February 2027 as its crawlers revisit sites. The new settings cover content used to train models and content used to “ground” answers, meaning web pages an AI assistant draws on when it writes a reply in Bing, Copilot or Microsoft 365. Microsoft says neither setting changes whether a page can appear in ordinary Bing search results.
The Microsoft change carries a deadline that some New Zealand sites will need to act on. Microsoft’s older tags, NoArchive and NoCache, are being retired as a way of opting out of training and grounding. Sites that rely on those tags have until 1 March 2027 to publish the new preferences, and Microsoft says that after that date the old tags “will no longer be read as a preference for training or grounding.” Sites using the old tags can record a temporary grounding preference in Bing Webmaster Tools from 15 December 2026.
The new token is about how content is used, not whether a crawler can visit. The draft is explicit that “This policy token is not a directive for crawling behavior.” A site that wants to stop a bot fetching its pages still has to block that bot by name.
It also does not opt anyone out by default. Under the draft, a robots.txt file with no __usage_ai-training group expresses no preference at all, and a general rule addressed to every crawler does not count. Microsoft’s guide says both its training and grounding settings default to allow when nothing is published. In practice, a New Zealand website that does nothing will be treated the same way tomorrow as it is today.
Nor does it bring payment. Microsoft’s post is blunt on that point. “Neither choice guarantees inclusion, traffic, or payment,” it says.
The proposal is still only a draft. Its IETF page states that it “is not endorsed by the IETF and has no formal standing in the IETF standards process,” and the document expires on 3 May 2027 unless it is updated. Its definition of AI training is borrowed from a separate vocabulary document the IETF’s AI Preferences working group has been developing, which covers changing the learned settings of a generative AI model and excludes models used only for tasks such as ranking or classification. Google notes the approach is consistent with the European Union’s code of practice for general purpose AI models, which recognises robots.txt as a main way for publishers to state machine-readable preferences.
Anthropic has not published a separate statement on its own newsroom, though the draft carries its employee’s name and lists contributors from Google, Microsoft, OpenAI and Anthropic among others.
For most New Zealand website owners the choice is less about training than it first appears. When Newswire logged every AI bot visiting this site for 11 days in September, training crawlers made just 16 per cent of the requests. OpenAI’s training crawler, GPTBot, made 36 requests in that period, while ChatGPT-User, which fetches a page when someone asks ChatGPT a question, made 2,823.
That is why the separation between training and everything else matters. A business that blocks AI crawlers broadly can drop out of the answers that send customers its way. OpenAI’s documentation, for instance, says sites that opt out of its search crawler, OAI-SearchBot, will not be shown in ChatGPT search answers. The shared token covers training only, and the draft keeps it separate from the rules that decide which bots may fetch a page. There is one wrinkle with Google. Its post says websites that specifically allow Google-Extended “are eligible to appear in responses and receive clicks via grounding in Gemini Apps,” so a site that opts out of training through the shared token may give up that eligibility too. Microsoft’s separate grounding setting gives owners a further choice on its own products.
Owners who want to opt out of training can add the shared token now, since parsers that do not yet recognise it will simply ignore it. Sites that have already blocked GPTBot or Google-Extended do not need to remove those rules, as both companies say their existing controls will continue to apply. The one firm date for New Zealand sites is Microsoft’s 1 March 2027 cut-off for anyone relying on NoArchive or NoCache.
Will you be opting your website out of AI training, or are you happy for your content to be used? Tell us in the comments below.
This article was written by AI, briefed to report the facts, hopefully without some of the bias people bring to the job 🙂

Leave a Reply