SEO & Technology

robots.txt and AI crawlers: GPTBot, OAI-SearchBot and more

robots.txt is a text file in the root directory of your website in which you ask crawlers to fetch, or not fetch, certain parts of it. AI providers use their own crawlers with their own names, such as GPTBot and OAI-SearchBot, which you can allow or block separately in that file. It remains a request, not a lock.

At a glance

robots.txt and AI crawlers: the short answer

Most AI providers work with more than one crawler. One collects content that can be used to train models, another builds the search index from which the system draws sources for its answers, and a third fetches a page when a user specifically asks for it.

That distinction matters, because you can control them separately. If you block all AI crawlers in one go, you also exclude yourself as a source in AI search results. If you only want to refuse training, you can do so without giving up your visibility.

Below, for each provider, you will find which names are officially documented, what they do and what robots.txt can and cannot do. The details were checked on 17 November 2026; providers adjust their crawlers regularly.

robots.txt – example that blocks a training crawler and allows a search crawler

Basics

How does robots.txt work?

The file is located at jouwdomein.be/robots.txt. A crawler that follows the convention reads it before fetching any pages.

The rules are laid down in a formal internet standard, the Robots Exclusion Protocol.

A simple robots.txt file
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml
  • User-agent specifies which crawler the rules apply to. An asterisk means they apply to all crawlers.
  • Disallow asks crawlers not to fetch a path. Allow makes an exception to that.
  • A dedicated block takes precedence. If there is a block for a specific crawler, that crawler follows that block and not the general rules.
  • Sitemap points to the list of your indexable pages.

Distinction

Three types of AI crawlers

The name of a crawler says little. What matters is what the fetched content is used for.

For the model

Training crawlers

They collect public content that can be used to train future models. Blocking them has no effect on your position in search results.

For the answers

Search crawlers

They build the index from which an AI system selects sources and shows links. If you block them, you will rarely, if ever, appear there as a source.

On request

User-initiated fetches

Someone pastes a link or asks the system to look at a page. That is not automated crawling, and providers treat it differently from their crawlers.

Reference

Which AI crawlers are officially documented?

This overview only includes names that the provider itself describes. Other bots also show up in your server logs; without official documentation, you cannot be sure who they belong to or what they do.

Officially documented AI crawlers and robots.txt tokens
Name in robots.txtProviderTypeWhat the provider says about it
GPTBotOpenAITrainingFetches content that may be used to train OpenAI’s models.
OAI-SearchBotOpenAISearchUsed for ChatGPT’s search feature. Separate from GPTBot: you can allow one and block the other.
ChatGPT-UserOpenAIAt a user’s requestVisits a page when a user asks it to. Not an automated crawler; robots.txt rules may not apply here.
OAI-AdsBotOpenAIAdvertisingChecks pages submitted as ads in ChatGPT. According to OpenAI, it only visits those pages; the data is not used to train models.
ClaudeBotAnthropicTrainingCollects public web content that may contribute to training models. Follows robots.txt.
Claude-SearchBotAnthropicSearchIndexes content to improve the quality of search results for users.
Claude-UserAnthropicAt a user’s requestFetches pages when a Claude user asks a question that requires it. According to Anthropic, it follows robots.txt.
PerplexityBotPerplexitySearchUsed to surface and link to websites in Perplexity’s search results; according to Perplexity, not used to train foundation models.
Perplexity-UserPerplexityAt a user’s requestVisits a page when a user’s question requires it; generally ignores robots.txt when doing so.
Google-ExtendedGoogleToken, not a crawlerControls whether content that Google already crawls may be used to train Gemini models and for grounding. Has no effect on Google Search.
Applebot-ExtendedAppleToken, not a crawlerControls whether content fetched by Applebot may be used to train Apple’s generative models. Content can still appear in search results.
CCBotCommon CrawlOpen web archiveBuilds an open, publicly accessible archive of web data. According to Common Crawl, it checks robots.txt first.

Last checked: 17 November 2026. Every name in the table was found in the provider’s own documentation on that date. The list is not exhaustive: providers also describe other, more specialised agents.

Sources: the documentation of OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl, accessed on 17 November 2026. If in doubt, always read the provider’s own page: it takes precedence.

Google

What about Google AI Overviews and AI Mode?

They are part of Google Search and use the regular Googlebot. There is no separate crawler you can block for AI Overviews. Google-Extended does not change that: that token concerns training Gemini models and grounding, not the search results.

If you do not want to appear in the AI features of Google Search, you are therefore left with the settings that also apply to standard search results, such as snippet restrictions. That directly affects how you are shown in the classic results. For most business websites, that is not a desirable trade-off. More on this in How do you appear in Google AI Overviews?.

Example

Allowing search and refusing training: what does that look like?

This is one possible choice, not a recommendation for every website. Each block applies to one crawler.

You can give Google-Extended and Applebot-Extended their own block in the same way. Note: according to Google, Google-Extended also controls grounding, meaning the use of your content as a source in Gemini.

Example: search crawlers allowed, training crawlers blocked
# Zoekcrawlers van AI-systemen: toegelaten
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Trainingscrawlers: geweigerd
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

If your file has no block for a particular crawler, that crawler follows the general rules under User-agent: *. So an AI crawler you do not mention anywhere may do everything other crawlers may do.

Deciding

Allow or block AI crawlers?

There is no right answer for everyone. It depends on what your website needs to do.

Allowing search crawlers makes sense when

  • you want to be found by people who ask an AI system for advice
  • your website presents your services or products
  • you want a system to use your own pages as a source, and not just what others write about you

Refusing training can make sense when

  • your content is itself the product, such as paid articles or courses
  • you do not want to contribute to training models on principle
  • you have legal or contractual reasons to restrict reuse

For a business website, blocking search crawlers is rarely to your advantage: you disappear as a source, while the system can still talk about you based on other websites. What you do with training crawlers is a choice you make yourself.

Limitations

What robots.txt cannot do

  • Enforce anything. The file is a request. Crawlers from well-known providers say they follow it; a bot with bad intentions does not read it.
  • Secure anything. Anything that must not be public belongs behind a login. A Disallow rule at most reveals where it is.
  • Stop user-initiated fetches. Several providers treat a visit requested by a user differently from automated crawling.
  • Undo anything. A rule added today does not remove content that has already been fetched.
  • Keep pages out of the index. Disallow prevents a URL from being fetched, not from being indexed. That is what noindex is for. See Website or page not indexed?.

If you want strict control, you need more than robots.txt: access control on the server, or firewall rules based on the IP ranges that providers publish.

Oversight

How do you check what your website allows?

Many websites block AI crawlers without the owner knowing, or allow everything without anyone ever having decided on it. Here is how to check.

01

Open your robots.txt

Type your domain followed by /robots.txt. Read the general rules first, then the blocks per crawler.

02

Look for an overly broad rule

A Disallow: / under User-agent: * shuts everyone out, search engines included. That happens more often than you would think after a website goes live.

03

Look beyond the file

Hosting, a CDN or a security plugin can block bots before they read your robots.txt. Check those settings too.

04

Check your server logs

There you can see which crawlers actually visit, and what response they receive.

05

Record your decision

Decide what you want for each type of crawler, and write down why. That way, the file does not become an accidental leftover.

06

Check it regularly

Providers add crawlers and change their behaviour. It can also take a while for a change to take effect everywhere.

Developments

What else is changing?

The landscape is not settled. In 2026, several providers announced that they intend to support a shared signal for AI training in robots.txt, so you no longer need to repeat your preference for each crawler. Adoption varies by provider, and the existing names continue to work.

There are also proposals such as llms.txt, which does not control access but provides an overview of your content. What it does and does not do is explained in llms.txt: what it does — and what it does not.

Robots.txt is, moreover, only one part of crawlability. Rendering, status codes, canonicals and internal links also determine whether a crawler can actually read your content. All of that falls under technical SEO.

FAQ

Frequently asked questions about robots.txt and AI crawlers

It is not mandatory. Without a file, crawlers assume everything may be fetched. A robots.txt is useful, though, for excluding areas such as an admin area or internal search results, and for pointing to your sitemap.

Always in the root directory: your domain name followed by /robots.txt. If it is anywhere else, crawlers will not read it.

WordPress creates a virtual robots.txt file itself if there is no real file. You can edit it with an SEO plugin that includes a file editor, or by placing your own file in the root directory. Then check in the browser what is actually being served.

Not through that rule alone. According to OpenAI, GPTBot is for training and OAI-SearchBot for ChatGPT’s search feature, and the settings are independent of each other. So you can block GPTBot and allow OAI-SearchBot.

Yes, for crawlers that respect it, and those include the documented crawlers of the major search engines and AI providers. It is not a technical barrier: anyone who wants to ignore the convention can do so.

Robots.txt is a technical convention, not a law. Whether ignoring it has legal consequences depends on the country, the terms of use and copyright. That is a question for a lawyer, not a developer.

More on this topic

Further reading

The related service

Technical SEO as a service

Not sure whether search engines and AI crawlers can fetch and read your website? The service page explains how I investigate and resolve crawlability, rendering and indexing issues.

Dries Nijs, Organic Web Specialist and author of this article

About the author

Dries Nijs, Organic Web Specialist

I am Dries Nijs, a developer and SEO consultant from Maasmechelen. I check crawlability in the code, the server settings and the logs, not just in a tool. MadeByDries is the personal brand under which I work.

Do you know which crawlers your website allows?

Tell me about your website. Then we can check what search engines and AI crawlers can read from it today.

Discuss your project

Tell me what you want to build or improve

You do not need a technical brief yet. Tell me where you are now and what you want to achieve, and I will help you find the right approach.

Choose at least one topic. You can select more than one.

I only use your details to respond to your enquiry.

Prefer to email me directly?

Location
Maasmechelen, Belgium