TechnicalNovember 7, 20269 min read

AI crawlers and robots.txt: which bots to allow

Which AI bots to allow in robots.txt: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more, and what blocking each does.

GDBy the GrowDesk team

If you want AI assistants to recommend your business, allow the search crawlers (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User). Blocking training crawlers like GPTBot, ClaudeBot or CCBot is a separate choice that doesn't remove you from AI answers. Then check that your CDN or firewall isn't blocking them anyway.

Key takeaways

AI bots come in three kinds: training crawlers, search crawlers and user-triggered fetchers. Blocking each kind has a different effect.
Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot keeps you out of those assistants' search answers. Blocking GPTBot or ClaudeBot only opts you out of model training.
Google-Extended does not affect Google Search, AI Overviews or AI Mode. It does control whether Gemini Apps can use your content for grounding.
User-triggered fetchers from OpenAI, Perplexity and Google may ignore robots.txt. Anthropic says Claude-User follows it.
Cloudflare and other firewalls can block AI bots no matter what your robots.txt says. Check both.

What's the difference between training, search and user-triggered bots?

Training crawlers collect pages to train future AI models (GPTBot, ClaudeBot, CCBot). Blocking them doesn't change whether an assistant can find or cite you today.
Search crawlers build the index an assistant searches when it answers a question (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot). Block these and you drop out of the answers.
User-triggered fetchers visit a page because a person asked about it right now (ChatGPT-User, Claude-User, Perplexity-User). This is the closest thing to a customer walking in the door.

For an HVAC company, a regional manufacturer or a dental clinic, search crawlers and user fetchers are how you get recommended. Training is a policy choice, and mostly a question for publishers.

Which AI user agents should you know?

We checked each entry against the vendor's own documentation in November 2026. The exception is Bytespider, which ByteDance documents only thinly in English.

User agent	Company	Type	What blocking it does	Follows robots.txt?
GPTBot	OpenAI	Training	Signals your content shouldn't train OpenAI's models. No effect on ChatGPT search.	Yes
OAI-SearchBot	OpenAI	Search	Your site won't be shown in ChatGPT search answers, except possibly as a plain navigational link.	Yes
ChatGPT-User	OpenAI	User-triggered	Visits pages when a ChatGPT user or Custom GPT asks.	Not always. OpenAI says robots.txt "may not apply."
ClaudeBot	Anthropic	Training	Excludes your future content from Anthropic's training data.	Yes
Claude-SearchBot	Anthropic	Search	Prevents indexing for Claude's search, which may reduce your visibility.	Yes
Claude-User	Anthropic	User-triggered	Prevents Claude from fetching your pages when users ask.	Yes
PerplexityBot	Perplexity	Search	Keeps you out of Perplexity's search results. Perplexity says it isn't used for training.	Yes
Perplexity-User	Perplexity	User-triggered	Fetches pages to answer a user's question.	Generally no
Googlebot	Google	Search	Removes you from Google Search, including AI Overviews and AI Mode.	Yes
Google-Extended	Google	Control token, no crawler of its own	Opts out of Gemini model training and grounding in Gemini Apps and Vertex AI. No effect on Search.	Yes (it's a robots.txt token)
Google-Agent	Google	User-triggered	Used by agents hosted on Google's servers.	Generally no
Bingbot	Microsoft	Search	Removes you from Bing, and from the Microsoft AI answers built on Bing.	Yes
Applebot	Apple	Search	Removes you from Siri, Spotlight and Safari search features.	Yes
Applebot-Extended	Apple	Control token	Opts out of Apple model training. You stay in Apple search.	Yes
Meta-ExternalAgent	Meta	Training and indexing	Opts out of Meta AI training.	Yes
Meta-ExternalFetcher	Meta	User-triggered	Fetches links users share with Meta AI.	May bypass it
CCBot	Common Crawl	Open web archive	Keeps your future pages out of a public dataset that many AI models have been trained on.	Yes
Bytespider	ByteDance	Crawler	Reports on its purpose and on whether it obeys robots.txt conflict.	Disputed

Two notes. xAI hasn't published crawler documentation we could verify for Grok, so there is no reliable Grok user agent to allow or block. And user agent strings can be faked. Google, Anthropic and Common Crawl all suggest checking IP ranges or reverse DNS before trusting a request.

Check your site in 20 seconds

See whether AI assistants can read you and AI agents can use you. Free, no account.

What does blocking Google-Extended actually do?

This one confuses a lot of people. Google's crawler documentation says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode are part of Search. They are controlled by Googlebot and normal snippet controls like nosnippet, as Google's AI features page explains.

But Google-Extended does more than training. Google says it also covers grounding in Gemini Apps, where the Gemini assistant pulls in live content to answer questions. If you block it, Gemini's own app may have less of your content to work with when someone asks it for a recommendation. If you want to be recommended in Gemini, leave it allowed.

Apple's Applebot-Extended works differently. Apple says it "does not crawl webpages" and that blocking it doesn't remove you from search results. It only covers training.

What robots.txt should a business use if it wants to be recommended?

Start simple. For most businesses, this is enough:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap.xml

That allows every search crawler, user fetcher and training crawler, and keeps bots out of your admin area. Change the paths to fit your site.

If you want to opt out of training but stay visible in AI answers, add a group for the training bots only:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

Don't add Google-Extended to that list unless you're fine with less visibility in Gemini Apps. And never block OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot or Bingbot if recommendations are the goal.

A bot follows only the most specific group that names it. If you give a bot its own group, it ignores your User-agent: * rules.

So if you add a User-agent: Googlebot group with Allow: /, Googlebot ignores the admin Disallow lines in your * group. Only name a bot when you mean to treat it differently. Changes can take up to about 24 hours to be picked up.

Is a shared "no AI training" rule coming?

Yes, but it isn't live yet. On October 30, 2026, an Internet-Draft proposed a shared robots.txt token, __usage_ai-training, that would let you state one training preference for every AI company that adopts it. Google says it intends to support it in early 2027. Microsoft says it will start honoring it on February 1, 2027. Anthropic says it will honor the signal, but a rule that names ClaudeBot still wins. It's still a draft, so keep your bot-specific rules for now.

Can your CDN or firewall block AI bots without you knowing?

Yes. robots.txt is a polite request. A CDN or firewall decides who actually gets through, and it can say no to bots your robots.txt welcomes.

Cloudflare is the big one, because it sits in front of a large share of small business websites:

In July 2025, Cloudflare began blocking known AI crawlers by default on new domains, through its "Block AI bots" setting.
In July 2026, it added separate controls for Search, Agent and Training, and on September 15, 2026, it changed the defaults. According to Cloudflare's own post, sites that had the old Block AI bots setting on were moved to Training: Disallow AI Training and Agent: Block on pages with ads. Cloudflare's July announcement says Agent includes fetchers like ChatGPT-User and browser agents.
New domains that carry ads are offered the same stricter preset. Cloudflare's current docs say other new domains start with all three categories allowed.
Choosing Block for Training now also applies to mixed-use crawlers like Googlebot, Bingbot and Applebot, which do search and training with one bot. That can take you out of regular search. Cloudflare recommends Disallow AI Training if you want to stop training but stay in search.

So a dental clinic that clicked "Block AI bots" in 2025 may now be blocking ChatGPT-User on any page with an ad, without anyone in the office knowing. Other hosts, security plugins and firewalls have similar AI bot switches.

How do you check what's blocked?

Read your live robots.txt. Open yoursite.com/robots.txt in a browser. Look for Disallow: / under any of the search or user agents above, and for lines you didn't write. Cloudflare's managed robots.txt can add its own rules and Content-Signal lines at the top.
Check your Cloudflare settings. Go to Security settings, then AI Crawler and Agent access policies. Search and Agent should be set to allow.
Look at the traffic. Cloudflare's AI Crawl Control is available on all plans and shows which AI crawlers visited and which were blocked. Without Cloudflare, search your server logs for the user agents in the table and look for 403 errors.
Test the response. Running curl -I -A "ChatGPT-User" https://www.example.com shows how your server treats that user agent. It's only a rough test, because firewalls that verify real bots by IP address or signature may treat your request differently.
Use the search engines' tools. Google Search Console has a robots.txt report, and Bing Webmaster Tools has a robots.txt tester.

GrowDesk's free check at growdesk.io flags robots.txt rules that block AI agents. For what else agents need, see is your website ready for AI agents. For the bigger picture, read AEO vs GEO vs SEO and our guide to Google AI Overviews and AI Mode.

FAQ

Does blocking GPTBot remove my business from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and live page visits use ChatGPT-User, so block GPTBot and you can still be cited.

Does Google-Extended affect AI Overviews?

No. Google says Google-Extended doesn't affect inclusion in Google Search, and AI Overviews and AI Mode are part of Search. It does affect training and grounding in Gemini Apps and Vertex AI.

Should I block all AI bots to protect my content?

For most businesses that want customers, no. Blocking search crawlers and user fetchers removes you from AI answers. If you object to training, block only the training crawlers.

Is robots.txt enough to block a bot?

No. robots.txt is a request that reputable crawlers honor. User-triggered fetchers may ignore it, and some crawlers ignore it entirely. Real blocking happens at your CDN or firewall.

How long do robots.txt changes take to work?

OpenAI, Perplexity and Meta all say up to about 24 hours. Google says it generally caches robots.txt for up to 24 hours too.