Google's AI training opt-out will also cut Gemini's links

A line in robots.txt that says “do not train AI on my site” sounds like a privacy setting. At Google it will also be a traffic setting. On 30 October Google said it will honour a new shared robots.txt token for AI training opt-outs, and that it will read that token as a block on Google-Extended. Google-Extended does two jobs: it controls Gemini training, and it controls whether Gemini Apps can ground answers in your pages and link to them. Opt out of the first and you leave the second.

What Google said

The post on the Search Central blog backs a draft from the IETF’s AIPREF working group. The draft adds no new directive. A site writes a normal robots.txt group for a made-up user agent, the policy token __usage_ai-training:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

No crawler sends that name. Any AI provider that adopts the draft reads the group as the site’s training preference, path by path. Parsers that do not know the token skip it, as they skip any unknown user agent.

Google’s part is one rule. If a site disallows training with the shared token and does not name Google-Extended, Google treats it “as if the publisher had disallowed Google-Extended”. A Google-Extended group, where one exists, still wins. Google’s crawler documentation, updated the same day, says support comes in “early 2027”. The blog post says “2027”.

What the switch actually controls

Google’s own documentation lists what Google-Extended governs. The new token inherits all of it:

Google surface	Controlled by	Shared training token affects it?
Training future Gemini models	Google-Extended	Yes
Grounding and links in Gemini Apps	Google-Extended	Yes
Grounding with Google Search on Vertex AI	Google-Extended	Yes
Google Search ranking and inclusion	Googlebot	No
AI Overviews and AI Mode	Googlebot, plus nosnippet and max-snippet	No

The post is direct about the cost: sites that allow Google-Extended “are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” There is no Google token for “train no, cite yes”. A site that wants Gemini links and no training cannot express that to Google with either token.

The draft’s definition of training is also narrow: changing the parameters of a generative model. Models used only to classify, rank or score are outside it. The token is not a general “no AI” flag.

Who has already made this choice

We read the robots.txt files of 20 large US news publishers and 20 large US retail and brand sites on 31 October. Five returned 403 and one did not answer, which left 35 files. We counted a block only where the group for that user agent says Disallow: /.

	Files read	Block Google-Extended at /	Block GPTBot at /	Use __usage_
News publishers	18	7	9	0
Retail and brand sites	17	1 (amazon.com)	2	0

The seven publishers that block Google-Extended: nytimes.com, usatoday.com, nbcnews.com, bloomberg.com, theverge.com, wired.com and nypost.com. The other publishers read: cnn.com, foxnews.com, washingtonpost.com, wsj.com, cbsnews.com, apnews.com, reuters.com, theguardian.com, businessinsider.com, latimes.com and forbes.com. Not readable: npr.org and cnbc.com. Retailers read: amazon.com, walmart.com, target.com, ebay.com, bestbuy.com, homedepot.com, costco.com, etsy.com, wayfair.com, nike.com, kohls.com, chewy.com, ulta.com, nordstrom.com, apple.com, samsung.com and dell.com. Not readable: lowes.com, macys.com and sephora.com.

Two things stand out. Retailers almost never block Google-Extended. Best Buy, Ulta and Samsung name it only to keep it out of carts, wishlists and search pages. They want to be in Gemini’s answers. And five publishers (CNN, CBS News, AP, the LA Times and Forbes) block OpenAI’s GPTBot but leave Google-Extended open. They have already chosen to treat the two providers differently. If one of them adds a blanket __usage_ai-training line and no Google-Extended group, Google closes Gemini to it as well.

The read

The privacy decision and the referral decision become one line. The people who own robots.txt, usually SEO, legal or engineering, will see a standard training opt-out and a policy reason to add it. The people who report on traffic will see a channel change with no release note. Gemini sends its clicks as a referral, and GA4 files those under AI Assistant, the channel for ChatGPT, Gemini, Copilot and the rest. Google’s AI Overviews and AI Mode count as Organic Search. They stay untouched, so the drop shows up in one channel only and is easy to blame on the wrong cause.

For a brand, the trade is lopsided. A retailer’s product pages gain little from staying out of a training set, and grounding is how a Gemini answer points a shopper to them. A publisher sells its text, and the calculation goes the other way. Either way, the default after 2027 is no longer “nothing happens unless you name Google”. Any provider that adopts the draft reads the shared line, Google included.

What to do

1. Check your robots.txt for Google-Extended today. If you want Gemini citations, give it an explicit group. Google says a named Google-Extended group overrides the shared token. It also means Google may train on those pages: the one token covers both.
2. Put robots.txt changes in the analytics change log. A new __usage_ai-training group is a traffic event for the AI Assistant channel, not only a legal one.
3. Watch the AI Assistant channel by source (gemini.google.com apart from the others), so a Gemini-only drop is visible when it happens.

Watching that channel assumes it counts Gemini clicks properly. It does not count all of them: a tagged link drops out of it. GA4’s AI Assistant channel only counts the untagged clicks shows which arrivals land there and which ones GA4 files somewhere else.