Blocking GPTBot does not take you out of ChatGPT search

The common belief is that a User-agent: GPTBot / Disallow: / line keeps a site out of ChatGPT. It is the line most robots.txt files reach for when someone says “block AI”.

It does not. GPTBot only collects training data. Whether ChatGPT shows and links your pages in its search answers is a different bot, OAI-SearchBot, and OpenAI says the two settings are independent. Anthropic and Perplexity are built the same way: one bot for training, one for search, one that fetches a page live when a user asks. A block on one does nothing to the other two. We read 36 large sites’ robots.txt files on 14 November: 13 block GPTBot, and only 7 of those also block OpenAI’s search bot.

Three jobs, three bots

Read from each provider’s own crawler documentation on 14 November 2026. Every cell is in facts above with its source.

Provider	Training	Search answers and links	Live fetch when a user asks	Does robots.txt stop the live fetch?
	GPTBot	OAI-SearchBot	ChatGPT-User	“robots.txt rules may not apply”
	ClaudeBot	Claude-SearchBot	Claude-User	Yes. Anthropic says its bots honour robots.txt
	no training crawler documented	PerplexityBot	Perplexity-User	“generally ignores robots.txt rules”
	Google-Extended (a token, not a crawler)	Google-Extended for Gemini Apps; Googlebot for Search	user-triggered fetchers, Google-Agent	“generally ignore robots.txt rules”
	Applebot-Extended (a token)	Applebot	not documented	n/a

Two things in this table catch people out.

The training token is not the citation token, except at Google. OpenAI’s page says a site can allow OAI-SearchBot “to appear in search results while disallowing GPTBot”. Sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers”, and can still appear only as navigational links. Google is the exception: Google-Extended controls both Gemini training and grounding in Gemini Apps, so there is no “train no, cite yes” switch for Gemini. That is the trade our piece on Google’s AI training opt-out covers.

The live fetcher is the bot you can least control. When a user pastes your URL into ChatGPT or asks a question that needs a fresh page, the request comes from ChatGPT-User. OpenAI says robots.txt “may not apply” to it, Perplexity says Perplexity-User “generally ignores” it, and Google says the same of its user-triggered fetchers. Anthropic is the one provider of the four that says its user fetcher follows robots.txt.

Advertisers meet a fourth OpenAI bot. OAI-AdsBot visits the landing page of an ad submitted on ChatGPT, checks it against policy and may read it to decide when the ad is relevant. OpenAI says it only visits pages submitted as ads and that its data is not used for training. If you plan to run ads in ChatGPT, it is the bot that reads your landing page.

What 36 sites actually block

We fetched /robots.txt from 20 large US news publishers and 20 large US retail and brand sites, the same list as our 31 October count. 36 answered with a file. A site counts as blocking a bot when the group that applies to it, its own or the * group, says Disallow: / with no Allow: /.

Bot	Job	News (18)	Retail and brand (18)	Total blocking
GPTBot	OpenAI training	11	2	13
OAI-SearchBot	ChatGPT search	6	1	7
ChatGPT-User	ChatGPT live fetch	7	1	8
ClaudeBot	Anthropic training	16	2	18
Claude-SearchBot	Claude search	12	1	13
Claude-User	Claude live fetch	12	1	13
PerplexityBot	Perplexity search	14	2	16
Perplexity-User	Perplexity live fetch	9	1	10
Google-Extended	Gemini training and grounding	10	1	11
Applebot-Extended	Apple training	14	1	15

Seven sites took themselves out of ChatGPT search along with training: Amazon, Bloomberg, CNN, NBC News, NPR, The New York Times and USA Today block all three OpenAI bots. Each of those is a choice to give up ChatGPT’s citations and the clicks that come with them.

Six block GPTBot and leave OAI-SearchBot in. Three do it by name: Forbes allows OAI-SearchBot on /, eBay gives it its own group with path rules only, and Reuters names it in a path-rules group while GPTBot falls to a * group that disallows the whole site. AP, CBS News and the LA Times do not name OAI-SearchBot at all, so it reads them under their * rules. The result is the same, whether or not anyone decided it.

The Wall Street Journal inverts the default. Its * group disallows the entire site. ChatGPT-User, GPTBot and OAI-SearchBot are named in a group that says Allow: /. Every other AI bot in our list is shut out by default, and OpenAI’s three are let in.

The blocks on live fetchers are the weakest lines in these files. Eight sites block ChatGPT-User and ten block Perplexity-User, the two fetchers whose owners say robots.txt may not apply or is generally ignored. Thirteen block Claude-User, the one that says it complies. The same line is a rule for one provider and a request for the others.

Stale tokens are still doing the work. Fourteen files name anthropic-ai or Claude-Web, neither of which is on Anthropic’s current list of three bots. The Washington Post blocks anthropic-ai, Claude-Web and ClaudeBot, and names neither Claude-SearchBot nor Claude-User. So Anthropic’s training bot is out, and its search bot reads the site under the * rules. Business Insider has the same gap. Five sites in all block ClaudeBot and leave Claude-SearchBot open.

Most retailers have not chosen. Fourteen of the 18 retail and brand files, and Fox News, name none of the twelve tokens we checked. Walmart, Target, Apple, Home Depot and the rest are open to every AI bot under their ordinary * rules. Amazon is the exception the other way: it blocks every bot in the table except Applebot-Extended.

What this count misses

• Root-level blocks only. A group that disallows /checkout/ and /search/ counts as open. Several retailers keep AI bots out of carts and search pages, which is sensible and not graded here.
• A file read from one client on one day. Four of the 40 sites did not return a file to us (403, an HTML page, or a timeout). A bot may get a different answer, and files change without notice.
• robots.txt is a request. It only binds the providers that say they follow it, and the table above shows which live fetchers say they may not.
• The search answer is not the whole of AI reading. A page can be read by a live fetcher, summarised in an answer and never clicked. GA4 reports sessions, and an answer that is read and never clicked creates none. What reaches your analytics is the click, and only when the search bot was allowed to put your link in the answer.

What to do

1. Write one group per job, not one per company. Decide training, search and live fetch separately for each provider, and name each token. A brand that wants AI citations allows OAI-SearchBot, Claude-SearchBot and PerplexityBot by name, whatever it decides about GPTBot and ClaudeBot.
2. Replace stale tokens. If your file names anthropic-ai or Claude-Web, add ClaudeBot, Claude-SearchBot and Claude-User and decide each one. Old tokens give a false sense of a decision.
3. Log robots.txt changes next to traffic. OpenAI and Perplexity say a change takes about 24 hours. A drop in AI referrals two days after an edit to robots.txt is not an algorithm update.

Letting the search bot in earns the citation. The click it sends is a separate question, because GA4 has its own rules for what counts as AI traffic. GA4’s AI Assistant channel only counts the untagged clicks shows which of those arrivals land in that channel and which ones GA4 files somewhere else.