ItInsightsWorld Explainer: One robots.txt Signal to Opt Out of AI Training — What It Covers and Where It Stops
If you publish anything on the open web and would rather your work not end up folded into someone else's model, the mechanics just became far simpler. Google, Microsoft, OpenAI and Anthropic are backing a shared draft that lets a site state its training preference once, in its existing robots.txt file. Website administrators only have to add a single set of rules, and every participating company reads the same answer. What the headlines usually leave out is the boundary: this signal is a statement about one purpose. It does not shut the door on crawlers, it does not decide whether an AI assistant quotes you, and it cannot reach backwards into material that has already been collected.
Bottom line for busy readers
- One declaration, many recipients. The proposed mechanism is a policy token,
__usage_ai-training, written into robots.txt with the same Allow and Disallow lines the file has always used. - Granularity is available. You can close the whole site and reopen selected folders, or open the whole site and close selected folders.
- The clock is already running. Microsoft is bound to the signal from February 1, 2027, with the other backers landing on their own schedules — details below.
- Scope is narrow on purpose. Training is one use. Crawling is another. Grounding an AI answer on your page is a third.
What the rules actually look like
The draft reuses the syntax robots.txt has carried for decades. There are no new command names and no new file format, which is precisely why old and unmodified crawlers can still parse the file without tripping over it.
| What you want | What you write under User-Agent: __usage_ai-training | Result |
|---|---|---|
| Close everything | Disallow: / | No content on the site may be used to train AI models by companies that honour the signal. |
| Keep a site-wide ban but open one area | Allow: /archive/ alongside the site-wide Disallow | These two do not cancel each other out. Effectively only /archive/ is open for training; every other path stays closed. |
| Open everything but close one area | disallow: /archive/ | /archive/ is excluded from training while the remainder of the site remains permitted. |
Why one token beats four separate instructions
Before this draft, opting out was an exercise in bookkeeping. Each vendor shipped its own crawler identity, and a site that wanted the same answer from all of them had to repeat the same block once per bot name. The shift from "who may enter" to "what the content may be used for" is the substantive change.
| Dimension | Legacy per-crawler rules | Usage-based training token |
|---|---|---|
| Rules to maintain | One group per crawler name | One set of rules for all participating companies |
| New or renamed bot appears | You must spot it and add it manually | Any honouring bot inherits your stated preference |
| Question it answers | May this visitor come in? | May this material train a model? |
| Effect on page access | Can restrict listed paths | None; access is unchanged |
| Failure mode | Silently incomplete if you miss a name | Inert with companies outside the arrangement |
One engineering detail is worth knowing because it explains the fast adoption. Because the token simply occupies the User-Agent slot, a parser that has never heard of it treats the group as an unrecognised visitor and skips it. Compatibility does not depend on anyone rewriting crawler code.
Who commits to what, and when
Support is real but uneven, and the dates matter more than the endorsement list.
- Microsoft has stated it will follow the declaration from February 1, 2027; sites that do not want their content used for AI training can therefore amend robots.txt today.
- Google plans to begin support in early 2027, with the exact date still to be confirmed once its preparation work is finished.
- OpenAI says it will implement the scheme within the coming months.
- Anthropic appears to have started supporting the draft already.
On provenance: the text is an individual Internet-Draft rather than an adopted standard. It was posted to the IETF datatracker as draft-kelley-aipref-training-preference-signal-00 on October 30, 2026, carries an intended status of Standards Track, and expires on May 3, 2027 unless it is revised (IETF datatracker, datatracker.ietf.org). Treat it as a strong industry convention that four parties have pledged to respect, not as a finished specification.
Five misconceptions worth clearing up
- "Publish this and AI can't see me." It cannot. The declaration states a purpose preference; blocking specific crawlers from retrieving pages still requires traditional robots.txt rules.
- "Opting out erases me." There is no retroactive recall. Anything already harvested before your file goes live is outside the mechanism's reach.
- "Training and being cited are the same switch." They are separate uses, and vendors expose separate controls.
- "Everyone will honour it." Compliance is voluntary. A crawler that misrepresents itself or simply ignores the convention reads nothing you wrote.
- "This settles payment." The design answers how to withhold permission, not how permission is priced or licensed.
Three uses, three switches
The most expensive mistake in this rollout will be setting one control and believing three are covered.
| Use of your content | Does the training declaration govern it? | What actually governs it |
|---|---|---|
| Crawling and page access | No | Conventional robots.txt rules for the crawler identities you want to restrict |
| Model training | Yes | The __usage_ai-training group |
| Grounding AI answers | No | Microsoft supplies an additional AI Grounding control for its own services, applied per page; Google plans to read a training opt-out without an explicit setting as a refusal of Google-Extended, which touches both Gemini model training and the way some Gemini answers reference pages |
Why the timing matters more than the syntax
Two developments raise the stakes. First, the traffic mix has shifted. Cloudflare reported that in spring 2025 roughly 22% of the crawler requests it observed were stated as being for AI training; by June 2026 that figure was 52% (Cloudflare blog, blog.cloudflare.com/agentic-web). Requests made to build or refresh models are no longer a rounding error on server logs.
Second, the regulatory frame in Europe turns a widely honoured token into something closer to evidence. Providers of general-purpose AI models have carried an obligation since August 2, 2025 under Article 53(1)(c) of the EU AI Act to maintain a copyright policy that identifies and respects rights reservations made in machine-readable form, and the European Commission has opened a consultation on which protocols qualify (digital-strategy.ec.europa.eu). A signal adopted by four major labs is a plausible candidate to become the de facto answer. That is a reason to decide deliberately now, rather than letting a reflexive setting harden into a default.
A sequence to follow before you edit robots.txt
- Inventory by value, not by folder. Separate pages that pull search traffic, leads or sales from material that exists mainly as depth — back catalogues, press kits, superseded documentation.
- Pick your default. Closed by default with exceptions is easier to reason about than open by default with holes.
- Model the trade. Opting out of training does not remove you from AI answers, but a broader exclusion choice can reduce visibility in generative search surfaces. Decide which you are protecting: leverage over future licensing, or reach.
- Write both rule families. The new token for training, plus legacy per-crawler lines for anything you want kept out outright.
- Verify the file. robots.txt must remain plain text at the root of the domain; one stray character silently disables every rule inside it.
- Baseline your numbers. Google extended its Search Console generative AI performance reports and AI features controls to sites worldwide on August 31, 2026, so you can watch impressions from AI Overviews and AI Mode before and after a change (developers.google.com).
- Decide whether to automate. If your edge provider generates the file for you, look at tooling such as Cloudflare's Bot Preference Sync, which aligns robots.txt with Search, Agent and Training policies.
- Set a review date. With the draft expiring on May 3, 2027 and staggered vendor rollouts through 2027, treat this as a quarterly item, not a one-time task.
FAQ
Does one line block GPTBot, ClaudeBot and the rest?
No. The declaration addresses training permission only. If you want a named crawler to stop fetching your pages, keep or add the traditional rules that target that crawler.
What happens if a company has not implemented the scheme yet?
Your rules are inert toward that company — nothing breaks, and nothing is enforced. That is why Microsoft's start date of February 1, 2027 is the practical deadline for sites that want the statement on record early.
Can I permit training only on old material?
Yes, in the sense that path-level rules allow it: ban the site and reopen the directory holding your back catalogue, or open the site and close the directory holding current work.
If I decline training, does that also decline Gemini-related use?
That is Google's planned interpretation where you decline training but have not explicitly configured Google-Extended: it intends to treat the site as having declined Google-Extended, a setting connected to Gemini model training and to some Gemini answers referencing pages.
Is this a reason to be nervous about editing robots.txt?
Less nervous than precise. The mechanism is small and reversible. The judgement about which content is worth exposing is the hard part, and no specification can make that call for you.
The take
Read this as a move from pleading with four companies separately to declaring one position that all four agree to read. For site owners, the work left is not technical but editorial: knowing which of your pages are an asset you can price later, and which are simply the depth that makes the rest credible. Make the statement, make it specific, then measure whether the choice cost you anything you valued.
Written by James Hartley for ItInsightsWorld (IT Insights). Editorial contact: contact@itinsights.com.