{"id":2319,"date":"2026-11-17T10:00:24","date_gmt":"2026-11-17T09:00:24","guid":{"rendered":"https:\/\/madebydries.be\/?p=2319"},"modified":"2026-11-19T14:32:15","modified_gmt":"2026-11-19T13:32:15","slug":"robots-txt-ai-crawlers","status":"publish","type":"post","link":"https:\/\/madebydries.be\/en\/knowledge-base\/robots-txt-ai-crawlers\/","title":{"rendered":"robots.txt and AI crawlers: GPTBot, OAI-SearchBot and more"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"2319\" class=\"elementor elementor-2319 elementor-754\" data-elementor-post-type=\"post\">\n\t\t\t\t<section class=\"elementor-element elementor-element-38459d5 e-con-boxed svc-hero mbd-section mbd-inset mbd-surface--dark mbd-bg-flow mbd-bg-flow--dark e-flex e-con e-parent\" data-id=\"38459d5\" data-element_type=\"container\" data-e-type=\"container\" id=\"top\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-8c498b0 e-con-full mbd-gap-l e-flex e-con e-child\" data-id=\"8c498b0\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-99ededf mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"99ededf\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">SEO &amp; Technology<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c540a3e svc-hero__title elementor-widget elementor-widget-heading\" data-id=\"c540a3e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h1 class=\"elementor-heading-title elementor-size-default\">robots.txt and AI crawlers: GPTBot, OAI-SearchBot and more<\/h1>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-84ee99c svc-hero__lead mbd-lead elementor-widget elementor-widget-text-editor\" data-id=\"84ee99c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>robots.txt is a text file in the root directory of your website in which you ask crawlers to fetch, or not fetch, certain parts of it. AI providers use their own crawlers with their own names, such as GPTBot and OAI-SearchBot, which you can allow or block separately in that file. It remains a request, not a lock.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-a49d308 e-con-full kb-byline mbd-gap-m e-flex e-con e-child\" data-id=\"a49d308\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-a23822a elementor-widget elementor-widget-heading\" data-id=\"a23822a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">By <a href=\"https:\/\/madebydries.be\/en\/about-dries-nijs\/\">Dries Nijs<\/a>, Organic Web Specialist<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cd2ab65 elementor-widget elementor-widget-heading\" data-id=\"cd2ab65\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Last updated: 19 November 2026<\/p>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-37a18a3 e-con-boxed mbd-section mbd-inset mbd-surface mbd-gap-xl e-flex e-con e-parent\" data-id=\"37a18a3\" data-element_type=\"container\" data-e-type=\"container\" id=\"in-het-kort\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-cc9ff6b e-grid e-con-full mbd-gap-2xl mbd-topic-grid e-con e-child\" data-id=\"cc9ff6b\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-e16fe90 e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"e16fe90\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-9561e2e mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"9561e2e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">At a glance<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8e743e3 elementor-widget elementor-widget-heading\" data-id=\"8e743e3\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">robots.txt and AI crawlers: the short answer<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-788d773 mbd-lead elementor-widget elementor-widget-text-editor\" data-id=\"788d773\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p><strong>Most AI providers work with more than one crawler.<\/strong> One collects content that can be used to train models, another builds the search index from which the system draws sources for its answers, and a third fetches a page when a user specifically asks for it.<\/p><p>That distinction matters, because you can control them separately. If you block all AI crawlers in one go, you also exclude yourself as a source in AI search results. If you only want to refuse training, you can do so without giving up your visibility.<\/p><p>Below, for each provider, you will find which names are officially documented, what they do and what <em>robots.txt<\/em> can and cannot do. The details were checked on 17 November 2026; providers adjust their crawlers regularly.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f72ae08 mbd-topic-cell elementor-widget elementor-widget-html\" data-id=\"f72ae08\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t\t<figure class=\"mbd-topic\"><img loading=\"lazy\" width=\"1440\" height=\"810\" src=\"https:\/\/madebydries.be\/wp-content\/uploads\/topic-visuals\/robots-txt-madebydries.webp\" class=\"mbd-topic__img\" alt=\"robots.txt \u2013 example that blocks a training crawler and allows a search crawler\" loading=\"lazy\" decoding=\"async\" sizes=\"auto, (min-width: 1025px) 440px, (min-width: 640px) 576px, calc(100vw - 2rem)\" srcset=\"https:\/\/madebydries.be\/wp-content\/uploads\/topic-visuals\/robots-txt-madebydries.webp 1440w, https:\/\/madebydries.be\/wp-content\/uploads\/topic-visuals\/robots-txt-madebydries-300x169.webp 300w, https:\/\/madebydries.be\/wp-content\/uploads\/topic-visuals\/robots-txt-madebydries-1024x576.webp 1024w, https:\/\/madebydries.be\/wp-content\/uploads\/topic-visuals\/robots-txt-madebydries-768x432.webp 768w\" \/><\/figure>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-da86d70 e-con-full svc-jump e-flex e-con e-child\" data-id=\"da86d70\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-805b636 elementor-widget elementor-widget-text-editor\" data-id=\"805b636\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<nav aria-label=\"On this page\"><p>On this page<\/p><ul><li><a href=\"#basis\">How robots.txt works<\/a><\/li><li><a href=\"#soorten\">Three types<\/a><\/li><li><a href=\"#overzicht\">Overview by provider<\/a><\/li><li><a href=\"#voorbeeld\">Example<\/a><\/li><li><a href=\"#kiezen\">Allow or block<\/a><\/li><li><a href=\"#grenzen\">Limitations<\/a><\/li><li><a href=\"#controleren\">Check<\/a><\/li><li><a href=\"#faq\">Frequently asked questions<\/a><\/li><\/ul><\/nav>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-50239a6 e-con-boxed mbd-section mbd-gap-xl e-flex e-con e-parent\" data-id=\"50239a6\" data-element_type=\"container\" data-e-type=\"container\" id=\"basis\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-569c62f e-grid e-con-full mbd-gap-2xl e-con e-child\" data-id=\"569c62f\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-e666573 e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"e666573\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-670f785 mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"670f785\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Basics<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1ffecb7 elementor-widget elementor-widget-heading\" data-id=\"1ffecb7\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">How does robots.txt work?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-52c54ef mbd-lead elementor-widget elementor-widget-text-editor\" data-id=\"52c54ef\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>The file is located at <code>jouwdomein.be\/robots.txt<\/code>. A crawler that follows the convention reads it before fetching any pages.<\/p><p>The rules are laid down in a formal internet standard, the Robots Exclusion Protocol.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5d4ed1e kb-prose elementor-widget elementor-widget-text-editor\" data-id=\"5d4ed1e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<figure class=\"kb-code\"><figcaption>A simple robots.txt file<\/figcaption><pre><code>User-agent: *\nDisallow: \/wp-admin\/\nAllow: \/wp-admin\/admin-ajax.php\n\nSitemap: https:\/\/example.com\/sitemap_index.xml<\/code><\/pre><\/figure><ul><li><strong>User-agent<\/strong> specifies which crawler the rules apply to. An asterisk means they apply to all crawlers.<\/li><li><strong>Disallow<\/strong> asks crawlers not to fetch a path. <strong>Allow<\/strong> makes an exception to that.<\/li><li><strong>A dedicated block takes precedence.<\/strong> If there is a block for a specific crawler, that crawler follows that block and not the general rules.<\/li><li><strong>Sitemap<\/strong> points to the list of your indexable pages.<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-71b63fc e-con-boxed mbd-section mbd-inset mbd-surface mbd-gap-2xl e-flex e-con e-parent\" data-id=\"71b63fc\" data-element_type=\"container\" data-e-type=\"container\" id=\"soorten\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-70c3d2e e-grid e-con-full mbd-gap-2xl e-con e-child\" data-id=\"70c3d2e\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-21d45b3 e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"21d45b3\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-2e3233b mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"2e3233b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Distinction<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9606efb elementor-widget elementor-widget-heading\" data-id=\"9606efb\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Three types of AI crawlers<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-dd6ac2f mbd-lead elementor-widget elementor-widget-text-editor\" data-id=\"dd6ac2f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>The name of a crawler says little. What matters is what the fetched content is used for.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-761012e e-grid e-con-full mbd-gap-xl e-con e-child\" data-id=\"761012e\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-58425d1 e-con-full svc-duo__col mbd-gap-s e-flex e-con e-child\" data-id=\"58425d1\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-276fd80 home-label elementor-widget elementor-widget-heading\" data-id=\"276fd80\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">For the model<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-31131bf elementor-widget elementor-widget-heading\" data-id=\"31131bf\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Training crawlers<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-da2efa1 elementor-widget elementor-widget-text-editor\" data-id=\"da2efa1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>They collect public content that can be used to train future models. Blocking them has no effect on your position in search results.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-c40ba04 e-con-full svc-duo__col mbd-gap-s e-flex e-con e-child\" data-id=\"c40ba04\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-877e931 home-label elementor-widget elementor-widget-heading\" data-id=\"877e931\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">For the answers<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-441d97e elementor-widget elementor-widget-heading\" data-id=\"441d97e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Search crawlers<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-52de744 elementor-widget elementor-widget-text-editor\" data-id=\"52de744\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>They build the index from which an AI system selects sources and shows links. If you block them, you will rarely, if ever, appear there as a source.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-da30afa e-con-full svc-duo__col mbd-gap-s e-flex e-con e-child\" data-id=\"da30afa\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-3b3b7fd home-label elementor-widget elementor-widget-heading\" data-id=\"3b3b7fd\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">On request<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9ce1329 elementor-widget elementor-widget-heading\" data-id=\"9ce1329\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">User-initiated fetches<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d070855 elementor-widget elementor-widget-text-editor\" data-id=\"d070855\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Someone pastes a link or asks the system to look at a page. That is not automated crawling, and providers treat it differently from their crawlers.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-160446f e-con-boxed mbd-section mbd-gap-xl e-flex e-con e-parent\" data-id=\"160446f\" data-element_type=\"container\" data-e-type=\"container\" id=\"overzicht\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-90aeb75 e-grid e-con-full mbd-gap-2xl e-con e-child\" data-id=\"90aeb75\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-fabf97b e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"fabf97b\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-31003b9 mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"31003b9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Reference<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8f13fbb elementor-widget elementor-widget-heading\" data-id=\"8f13fbb\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Which AI crawlers are officially documented?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c748913 mbd-lead elementor-widget elementor-widget-text-editor\" data-id=\"c748913\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>This overview only includes names that the provider itself describes. Other bots also show up in your server logs; without official documentation, you cannot be sure who they belong to or what they do.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f794862 kb-table kb-table--compare elementor-widget elementor-widget-text-editor\" data-id=\"f794862\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<table><caption>Officially documented AI crawlers and robots.txt tokens<\/caption><thead><tr><th scope=\"col\">Name in robots.txt<\/th><th scope=\"col\">Provider<\/th><th scope=\"col\">Type<\/th><th scope=\"col\">What the provider says about it<\/th><\/tr><\/thead><tbody><tr><th scope=\"row\"><code>GPTBot<\/code><\/th><td data-label=\"Provider\">OpenAI<\/td><td data-label=\"Type\">Training<\/td><td data-label=\"What the provider says about it\">Fetches content that may be used to train OpenAI\u2019s models.<\/td><\/tr><tr><th scope=\"row\"><code>OAI-SearchBot<\/code><\/th><td data-label=\"Provider\">OpenAI<\/td><td data-label=\"Type\">Search<\/td><td data-label=\"What the provider says about it\">Used for ChatGPT\u2019s search feature. Separate from GPTBot: you can allow one and block the other.<\/td><\/tr><tr><th scope=\"row\"><code>ChatGPT-User<\/code><\/th><td data-label=\"Provider\">OpenAI<\/td><td data-label=\"Type\">At a user\u2019s request<\/td><td data-label=\"What the provider says about it\">Visits a page when a user asks it to. Not an automated crawler; robots.txt rules may not apply here.<\/td><\/tr><tr><th scope=\"row\"><code>OAI-AdsBot<\/code><\/th><td data-label=\"Provider\">OpenAI<\/td><td data-label=\"Type\">Advertising<\/td><td data-label=\"What the provider says about it\">Checks pages submitted as ads in ChatGPT. According to OpenAI, it only visits those pages; the data is not used to train models.<\/td><\/tr><tr><th scope=\"row\"><code>ClaudeBot<\/code><\/th><td data-label=\"Provider\">Anthropic<\/td><td data-label=\"Type\">Training<\/td><td data-label=\"What the provider says about it\">Collects public web content that may contribute to training models. Follows robots.txt.<\/td><\/tr><tr><th scope=\"row\"><code>Claude-SearchBot<\/code><\/th><td data-label=\"Provider\">Anthropic<\/td><td data-label=\"Type\">Search<\/td><td data-label=\"What the provider says about it\">Indexes content to improve the quality of search results for users.<\/td><\/tr><tr><th scope=\"row\"><code>Claude-User<\/code><\/th><td data-label=\"Provider\">Anthropic<\/td><td data-label=\"Type\">At a user\u2019s request<\/td><td data-label=\"What the provider says about it\">Fetches pages when a Claude user asks a question that requires it. According to Anthropic, it follows robots.txt.<\/td><\/tr><tr><th scope=\"row\"><code>PerplexityBot<\/code><\/th><td data-label=\"Provider\">Perplexity<\/td><td data-label=\"Type\">Search<\/td><td data-label=\"What the provider says about it\">Used to surface and link to websites in Perplexity\u2019s search results; according to Perplexity, not used to train foundation models.<\/td><\/tr><tr><th scope=\"row\"><code>Perplexity-User<\/code><\/th><td data-label=\"Provider\">Perplexity<\/td><td data-label=\"Type\">At a user\u2019s request<\/td><td data-label=\"What the provider says about it\">Visits a page when a user\u2019s question requires it; generally ignores robots.txt when doing so.<\/td><\/tr><tr><th scope=\"row\"><code>Google-Extended<\/code><\/th><td data-label=\"Provider\">Google<\/td><td data-label=\"Type\">Token, not a crawler<\/td><td data-label=\"What the provider says about it\">Controls whether content that Google already crawls may be used to train Gemini models and for grounding. Has no effect on Google Search.<\/td><\/tr><tr><th scope=\"row\"><code>Applebot-Extended<\/code><\/th><td data-label=\"Provider\">Apple<\/td><td data-label=\"Type\">Token, not a crawler<\/td><td data-label=\"What the provider says about it\">Controls whether content fetched by Applebot may be used to train Apple\u2019s generative models. Content can still appear in search results.<\/td><\/tr><tr><th scope=\"row\"><code>CCBot<\/code><\/th><td data-label=\"Provider\">Common Crawl<\/td><td data-label=\"Type\">Open web archive<\/td><td data-label=\"What the provider says about it\">Builds an open, publicly accessible archive of web data. According to Common Crawl, it checks robots.txt first.<\/td><\/tr><\/tbody><\/table>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-27f4a4d mbd-measure elementor-widget elementor-widget-text-editor\" data-id=\"27f4a4d\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p><strong>Last checked: 17 November 2026.<\/strong> Every name in the table was found in the provider\u2019s own documentation on that date. The list is not exhaustive: providers also describe other, more specialised agents.<\/p><p>Sources: the documentation of <a href=\"https:\/\/developers.openai.com\/api\/docs\/bots\" target=\"_blank\" rel=\"noopener\">OpenAI<\/a>, <a href=\"https:\/\/support.claude.com\/en\/articles\/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler\" target=\"_blank\" rel=\"noopener\">Anthropic<\/a>, <a href=\"https:\/\/docs.perplexity.ai\/docs\/resources\/perplexity-crawlers\" target=\"_blank\" rel=\"noopener\">Perplexity<\/a>, <a href=\"https:\/\/developers.google.com\/crawling\/docs\/crawlers-fetchers\/google-common-crawlers\" target=\"_blank\" rel=\"noopener\">Google<\/a>, <a href=\"https:\/\/support.apple.com\/en-us\/119829\" target=\"_blank\" rel=\"noopener\">Apple<\/a> and <a href=\"https:\/\/commoncrawl.org\/ccbot\" target=\"_blank\" rel=\"noopener\">Common Crawl<\/a>, accessed on 17 November 2026. If in doubt, always read the provider\u2019s own page: it takes precedence.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-684ec0c e-con-boxed mbd-section mbd-inset mbd-surface mbd-gap-xl e-flex e-con e-parent\" data-id=\"684ec0c\" data-element_type=\"container\" data-e-type=\"container\" id=\"google\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-c4dae76 e-grid e-con-full mbd-gap-2xl e-con e-child\" data-id=\"c4dae76\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-173e3a6 e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"173e3a6\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-2a7a686 mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"2a7a686\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Google<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8a10dc0 elementor-widget elementor-widget-heading\" data-id=\"8a10dc0\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">What about Google AI Overviews and AI Mode?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d693148 kb-prose elementor-widget elementor-widget-text-editor\" data-id=\"d693148\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>They are part of Google Search and use the regular Googlebot. There is no separate crawler you can block for AI Overviews. Google-Extended does not change that: that token concerns training Gemini models and grounding, not the search results.<\/p><p>If you do not want to appear in the AI features of Google Search, you are therefore left with the settings that also apply to standard search results, such as snippet restrictions. That directly affects how you are shown in the classic results. For most business websites, that is not a desirable trade-off. More on this in <a href=\"https:\/\/madebydries.be\/en\/knowledge-base\/google-ai-overviews\/\">How do you appear in Google AI Overviews?<\/a>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-6cc452b e-con-boxed mbd-section mbd-gap-xl e-flex e-con e-parent\" data-id=\"6cc452b\" data-element_type=\"container\" data-e-type=\"container\" id=\"voorbeeld\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-6cbbc85 e-grid e-con-full mbd-gap-2xl e-con e-child\" data-id=\"6cbbc85\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-c0954e1 e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"c0954e1\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-3719ac8 mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"3719ac8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Example<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0a16e2c elementor-widget elementor-widget-heading\" data-id=\"0a16e2c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Allowing search and refusing training: what does that look like?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f98f552 mbd-lead elementor-widget elementor-widget-text-editor\" data-id=\"f98f552\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>This is one possible choice, not a recommendation for every website. Each block applies to one crawler.<\/p><p>You can give Google-Extended and Applebot-Extended their own block in the same way. Note: according to Google, Google-Extended also controls grounding, meaning the use of your content as a source in Gemini.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1b2c686 kb-prose elementor-widget elementor-widget-text-editor\" data-id=\"1b2c686\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<figure class=\"kb-code\"><figcaption>Example: search crawlers allowed, training crawlers blocked<\/figcaption><pre><code># Zoekcrawlers van AI-systemen: toegelaten\nUser-agent: OAI-SearchBot\nAllow: \/\n\nUser-agent: Claude-SearchBot\nAllow: \/\n\nUser-agent: PerplexityBot\nAllow: \/\n\n# Trainingscrawlers: geweigerd\nUser-agent: GPTBot\nDisallow: \/\n\nUser-agent: ClaudeBot\nDisallow: \/\n\nUser-agent: CCBot\nDisallow: \/<\/code><\/pre><\/figure><p>If your file has no block for a particular crawler, that crawler follows the general rules under <code>User-agent: *<\/code>. So an AI crawler you do not mention anywhere may do everything other crawlers may do.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-4192ec3 e-con-boxed mbd-section mbd-inset mbd-surface mbd-gap-2xl e-flex e-con e-parent\" data-id=\"4192ec3\" data-element_type=\"container\" data-e-type=\"container\" id=\"kiezen\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-6f98cff e-grid e-con-full mbd-gap-2xl e-con e-child\" data-id=\"6f98cff\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-37ceae1 e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"37ceae1\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-7858957 mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"7858957\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Deciding<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3fd5049 elementor-widget elementor-widget-heading\" data-id=\"3fd5049\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Allow or block AI crawlers?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-952bdc8 mbd-lead elementor-widget elementor-widget-text-editor\" data-id=\"952bdc8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>There is no right answer for everyone. It depends on what your website needs to do.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-4ea2930 e-grid e-con-full mbd-gap-xl e-con e-child\" data-id=\"4ea2930\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-e0772b0 e-con-full svc-duo__col mbd-gap-s e-flex e-con e-child\" data-id=\"e0772b0\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-5377086 elementor-widget elementor-widget-heading\" data-id=\"5377086\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Allowing search crawlers makes sense when<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8bee255 kb-list elementor-widget elementor-widget-text-editor\" data-id=\"8bee255\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<ul><li>you want to be found by people who ask an AI system for advice<\/li><li>your website presents your services or products<\/li><li>you want a system to use your own pages as a source, and not just what others write about you<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-a847f17 e-con-full svc-duo__col mbd-gap-s e-flex e-con e-child\" data-id=\"a847f17\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-8adeb50 elementor-widget elementor-widget-heading\" data-id=\"8adeb50\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Refusing training can make sense when<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-56fcb91 kb-list elementor-widget elementor-widget-text-editor\" data-id=\"56fcb91\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<ul><li>your content is itself the product, such as paid articles or courses<\/li><li>you do not want to contribute to training models on principle<\/li><li>you have legal or contractual reasons to restrict reuse<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e49329a mbd-measure elementor-widget elementor-widget-text-editor\" data-id=\"e49329a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>For a business website, blocking search crawlers is rarely to your advantage: you disappear as a source, while the system can still talk about you based on other websites. What you do with training crawlers is a choice you make yourself.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-da2e407 e-con-boxed mbd-section mbd-inset mbd-surface--tint mbd-gap-xl e-flex e-con e-parent\" data-id=\"da2e407\" data-element_type=\"container\" data-e-type=\"container\" id=\"grenzen\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-17bbbcd e-grid e-con-full mbd-gap-2xl e-con e-child\" data-id=\"17bbbcd\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-495fc86 e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"495fc86\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-b59b1ea mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"b59b1ea\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Limitations<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3cc3624 elementor-widget elementor-widget-heading\" data-id=\"3cc3624\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">What robots.txt cannot do<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-1fbf826 e-con-full mbd-gap-m e-flex e-con e-child\" data-id=\"1fbf826\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-f5a4e7d svc-list elementor-widget elementor-widget-text-editor\" data-id=\"f5a4e7d\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<ul><li><strong>Enforce anything.<\/strong> The file is a request. Crawlers from well-known providers say they follow it; a bot with bad intentions does not read it.<\/li><li><strong>Secure anything.<\/strong> Anything that must not be public belongs behind a login. A Disallow rule at most reveals where it is.<\/li><li><strong>Stop user-initiated fetches.<\/strong> Several providers treat a visit requested by a user differently from automated crawling.<\/li><li><strong>Undo anything.<\/strong> A rule added today does not remove content that has already been fetched.<\/li><li><strong>Keep pages out of the index.<\/strong> Disallow prevents a URL from being fetched, not from being indexed. That is what noindex is for. See <a href=\"https:\/\/madebydries.be\/en\/knowledge-base\/website-not-indexed\/\">Website or page not indexed?<\/a>.<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c2fbba9 elementor-widget elementor-widget-text-editor\" data-id=\"c2fbba9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>If you want strict control, you need more than robots.txt: access control on the server, or firewall rules based on the IP ranges that providers publish.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-9fe43dd e-con-boxed mbd-section mbd-gap-xl e-flex e-con e-parent\" data-id=\"9fe43dd\" data-element_type=\"container\" data-e-type=\"container\" id=\"controleren\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-1e7678d e-grid e-con-full mbd-gap-2xl e-con e-child\" data-id=\"1e7678d\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-c01bdec e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"c01bdec\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-9a5a16d mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"9a5a16d\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Oversight<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-256cd42 elementor-widget elementor-widget-heading\" data-id=\"256cd42\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">How do you check what your website allows?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-97ff85e mbd-lead elementor-widget elementor-widget-text-editor\" data-id=\"97ff85e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Many websites block AI crawlers without the owner knowing, or allow everything without anyone ever having decided on it. Here is how to check.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-8869952 e-con-full mbd-gap-0 e-flex e-con e-child\" data-id=\"8869952\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-ea4f63b e-grid e-con-full svc-row mbd-gap-m svc-row--compact e-con e-child\" data-id=\"ea4f63b\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-ada8224 home-label home-label--accent elementor-widget elementor-widget-heading\" data-id=\"ada8224\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">01<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3053f3c svc-row__title elementor-widget elementor-widget-heading\" data-id=\"3053f3c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Open your robots.txt<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-906e660 svc-row__desc elementor-widget elementor-widget-text-editor\" data-id=\"906e660\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Type your domain followed by \/robots.txt. Read the general rules first, then the blocks per crawler.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-3650738 e-grid e-con-full svc-row mbd-gap-m svc-row--compact e-con e-child\" data-id=\"3650738\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-6a68c33 home-label home-label--accent elementor-widget elementor-widget-heading\" data-id=\"6a68c33\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">02<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7eeaa93 svc-row__title elementor-widget elementor-widget-heading\" data-id=\"7eeaa93\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Look for an overly broad rule<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-37cb5ce svc-row__desc elementor-widget elementor-widget-text-editor\" data-id=\"37cb5ce\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>A <code>Disallow: \/<\/code> under <code>User-agent: *<\/code> shuts everyone out, search engines included. That happens more often than you would think after a website goes live.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-2b2972c e-grid e-con-full svc-row mbd-gap-m svc-row--compact e-con e-child\" data-id=\"2b2972c\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-42039e8 home-label home-label--accent elementor-widget elementor-widget-heading\" data-id=\"42039e8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">03<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5809a6c svc-row__title elementor-widget elementor-widget-heading\" data-id=\"5809a6c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Look beyond the file<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8e9f8a7 svc-row__desc elementor-widget elementor-widget-text-editor\" data-id=\"8e9f8a7\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Hosting, a CDN or a security plugin can block bots before they read your robots.txt. Check those settings too.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-095fd13 e-grid e-con-full svc-row mbd-gap-m svc-row--compact e-con e-child\" data-id=\"095fd13\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-d1bcea4 home-label home-label--accent elementor-widget elementor-widget-heading\" data-id=\"d1bcea4\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">04<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f3939b9 svc-row__title elementor-widget elementor-widget-heading\" data-id=\"f3939b9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Check your server logs<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fa384f2 svc-row__desc elementor-widget elementor-widget-text-editor\" data-id=\"fa384f2\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>There you can see which crawlers actually visit, and what response they receive.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-cf4f78d e-grid e-con-full svc-row mbd-gap-m svc-row--compact e-con e-child\" data-id=\"cf4f78d\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-4298c4b home-label home-label--accent elementor-widget elementor-widget-heading\" data-id=\"4298c4b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">05<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bb09199 svc-row__title elementor-widget elementor-widget-heading\" data-id=\"bb09199\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Record your decision<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-87c4f8c svc-row__desc elementor-widget elementor-widget-text-editor\" data-id=\"87c4f8c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Decide what you want for each type of crawler, and write down why. That way, the file does not become an accidental leftover.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-d08680d e-grid e-con-full svc-row mbd-gap-m svc-row--compact e-con e-child\" data-id=\"d08680d\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-6d717f2 home-label home-label--accent elementor-widget elementor-widget-heading\" data-id=\"6d717f2\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">06<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cf082a1 svc-row__title elementor-widget elementor-widget-heading\" data-id=\"cf082a1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Check it regularly<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e2fc187 svc-row__desc elementor-widget elementor-widget-text-editor\" data-id=\"e2fc187\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Providers add crawlers and change their behaviour. It can also take a while for a change to take effect everywhere.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-a3e6af1 e-con-boxed mbd-section mbd-inset mbd-surface mbd-gap-xl e-flex e-con e-parent\" data-id=\"a3e6af1\" data-element_type=\"container\" data-e-type=\"container\" id=\"evolutie\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-a2fbfa5 e-grid e-con-full mbd-gap-2xl e-con e-child\" data-id=\"a2fbfa5\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-0f431f8 e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"0f431f8\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-a2628a6 mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"a2628a6\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Developments<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-481b83c elementor-widget elementor-widget-heading\" data-id=\"481b83c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">What else is changing?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-23a46b6 kb-prose elementor-widget elementor-widget-text-editor\" data-id=\"23a46b6\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>The landscape is not settled. In 2026, several providers announced that they intend to support a shared signal for AI training in robots.txt, so you no longer need to repeat your preference for each crawler. Adoption varies by provider, and the existing names continue to work.<\/p><p>There are also proposals such as llms.txt, which does not control access but provides an overview of your content. What it does and does not do is explained in <a href=\"https:\/\/madebydries.be\/en\/knowledge-base\/llms-txt\/\">llms.txt: what it does \u2014 and what it does not<\/a>.<\/p><p>Robots.txt is, moreover, only one part of crawlability. Rendering, status codes, canonicals and internal links also determine whether a crawler can actually read your content. All of that falls under <a href=\"https:\/\/madebydries.be\/en\/seo-consultant\/technical-seo\/\">technical SEO<\/a>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-e809c69 e-con-boxed mbd-section mbd-inset mbd-surface mbd-gap-xl e-flex e-con e-parent\" data-id=\"e809c69\" data-element_type=\"container\" data-e-type=\"container\" id=\"faq\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-470f8cd e-grid e-con-full mbd-gap-2xl e-con e-child\" data-id=\"470f8cd\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-4bd2f0f e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"4bd2f0f\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-7ae676f mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"7ae676f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">FAQ<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4fa0b76 elementor-widget elementor-widget-heading\" data-id=\"4fa0b76\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Frequently asked questions about robots.txt and AI crawlers<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2c93839 home-faq elementor-widget elementor-widget-n-accordion\" data-id=\"2c93839\" data-element_type=\"widget\" data-e-type=\"widget\" data-settings=\"{&quot;default_state&quot;:&quot;all_collapsed&quot;,&quot;max_items_expended&quot;:&quot;one&quot;,&quot;n_accordion_animation_duration&quot;:{&quot;unit&quot;:&quot;ms&quot;,&quot;size&quot;:400,&quot;sizes&quot;:[]}}\" data-widget_type=\"nested-accordion.default\">\n\t\t\t\t\t\t\t<div class=\"e-n-accordion\" aria-label=\"Accordion. Open links with Enter or Space, close with Escape, and navigate with Arrow Keys\">\n\t\t\t\t\t\t<details id=\"e-n-accordion-item-4670\" class=\"e-n-accordion-item\" >\n\t\t\t\t<summary class=\"e-n-accordion-item-title\" data-accordion-index=\"1\" tabindex=\"0\" aria-expanded=\"false\" aria-controls=\"e-n-accordion-item-4670\" >\n\t\t\t\t\t<span class='e-n-accordion-item-title-header'><h3 class=\"e-n-accordion-item-title-text\"> Do you need a robots.txt file? <\/h3><\/span>\n\t\t\t\t\t\t\t<span class='e-n-accordion-item-title-icon'>\n\t\t\t<span class='e-opened' ><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-minus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h384c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t\t<span class='e-closed'><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-plus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H272V64c0-17.67-14.33-32-32-32h-32c-17.67 0-32 14.33-32 32v144H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h144v144c0 17.67 14.33 32 32 32h32c17.67 0 32-14.33 32-32V304h144c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t<\/span>\n\n\t\t\t\t\t\t<\/summary>\n\t\t\t\t<div role=\"region\" aria-labelledby=\"e-n-accordion-item-4670\" class=\"elementor-element elementor-element-b067be9 e-con-full mbd-gap-s e-flex e-con e-child\" data-id=\"b067be9\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-2db4986 elementor-widget elementor-widget-text-editor\" data-id=\"2db4986\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>It is not mandatory. Without a file, crawlers assume everything may be fetched. A robots.txt is useful, though, for excluding areas such as an admin area or internal search results, and for pointing to your sitemap.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/details>\n\t\t\t\t\t\t<details id=\"e-n-accordion-item-4671\" class=\"e-n-accordion-item\" >\n\t\t\t\t<summary class=\"e-n-accordion-item-title\" data-accordion-index=\"2\" tabindex=\"-1\" aria-expanded=\"false\" aria-controls=\"e-n-accordion-item-4671\" >\n\t\t\t\t\t<span class='e-n-accordion-item-title-header'><h3 class=\"e-n-accordion-item-title-text\"> Where can I find my website\u2019s robots.txt file? <\/h3><\/span>\n\t\t\t\t\t\t\t<span class='e-n-accordion-item-title-icon'>\n\t\t\t<span class='e-opened' ><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-minus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h384c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t\t<span class='e-closed'><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-plus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H272V64c0-17.67-14.33-32-32-32h-32c-17.67 0-32 14.33-32 32v144H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h144v144c0 17.67 14.33 32 32 32h32c17.67 0 32-14.33 32-32V304h144c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t<\/span>\n\n\t\t\t\t\t\t<\/summary>\n\t\t\t\t<div role=\"region\" aria-labelledby=\"e-n-accordion-item-4671\" class=\"elementor-element elementor-element-e1e25d9 e-con-full mbd-gap-s e-flex e-con e-child\" data-id=\"e1e25d9\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-8b644e4 elementor-widget elementor-widget-text-editor\" data-id=\"8b644e4\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Always in the root directory: your domain name followed by \/robots.txt. If it is anywhere else, crawlers will not read it.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/details>\n\t\t\t\t\t\t<details id=\"e-n-accordion-item-4672\" class=\"e-n-accordion-item\" >\n\t\t\t\t<summary class=\"e-n-accordion-item-title\" data-accordion-index=\"3\" tabindex=\"-1\" aria-expanded=\"false\" aria-controls=\"e-n-accordion-item-4672\" >\n\t\t\t\t\t<span class='e-n-accordion-item-title-header'><h3 class=\"e-n-accordion-item-title-text\"> How do I edit robots.txt in WordPress? <\/h3><\/span>\n\t\t\t\t\t\t\t<span class='e-n-accordion-item-title-icon'>\n\t\t\t<span class='e-opened' ><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-minus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h384c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t\t<span class='e-closed'><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-plus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H272V64c0-17.67-14.33-32-32-32h-32c-17.67 0-32 14.33-32 32v144H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h144v144c0 17.67 14.33 32 32 32h32c17.67 0 32-14.33 32-32V304h144c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t<\/span>\n\n\t\t\t\t\t\t<\/summary>\n\t\t\t\t<div role=\"region\" aria-labelledby=\"e-n-accordion-item-4672\" class=\"elementor-element elementor-element-46d163d e-con-full mbd-gap-s e-flex e-con e-child\" data-id=\"46d163d\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-186d92c elementor-widget elementor-widget-text-editor\" data-id=\"186d92c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>WordPress creates a virtual robots.txt file itself if there is no real file. You can edit it with an SEO plugin that includes a file editor, or by placing your own file in the root directory. Then check in the browser what is actually being served.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/details>\n\t\t\t\t\t\t<details id=\"e-n-accordion-item-4673\" class=\"e-n-accordion-item\" >\n\t\t\t\t<summary class=\"e-n-accordion-item-title\" data-accordion-index=\"4\" tabindex=\"-1\" aria-expanded=\"false\" aria-controls=\"e-n-accordion-item-4673\" >\n\t\t\t\t\t<span class='e-n-accordion-item-title-header'><h3 class=\"e-n-accordion-item-title-text\"> If I block GPTBot, will I disappear from ChatGPT? <\/h3><\/span>\n\t\t\t\t\t\t\t<span class='e-n-accordion-item-title-icon'>\n\t\t\t<span class='e-opened' ><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-minus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h384c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t\t<span class='e-closed'><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-plus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H272V64c0-17.67-14.33-32-32-32h-32c-17.67 0-32 14.33-32 32v144H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h144v144c0 17.67 14.33 32 32 32h32c17.67 0 32-14.33 32-32V304h144c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t<\/span>\n\n\t\t\t\t\t\t<\/summary>\n\t\t\t\t<div role=\"region\" aria-labelledby=\"e-n-accordion-item-4673\" class=\"elementor-element elementor-element-2634e4a e-con-full mbd-gap-s e-flex e-con e-child\" data-id=\"2634e4a\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-376db02 elementor-widget elementor-widget-text-editor\" data-id=\"376db02\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Not through that rule alone. According to OpenAI, GPTBot is for training and OAI-SearchBot for ChatGPT\u2019s search feature, and the settings are independent of each other. So you can block GPTBot and allow OAI-SearchBot.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/details>\n\t\t\t\t\t\t<details id=\"e-n-accordion-item-4674\" class=\"e-n-accordion-item\" >\n\t\t\t\t<summary class=\"e-n-accordion-item-title\" data-accordion-index=\"5\" tabindex=\"-1\" aria-expanded=\"false\" aria-controls=\"e-n-accordion-item-4674\" >\n\t\t\t\t\t<span class='e-n-accordion-item-title-header'><h3 class=\"e-n-accordion-item-title-text\"> Does robots.txt still work? <\/h3><\/span>\n\t\t\t\t\t\t\t<span class='e-n-accordion-item-title-icon'>\n\t\t\t<span class='e-opened' ><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-minus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h384c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t\t<span class='e-closed'><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-plus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H272V64c0-17.67-14.33-32-32-32h-32c-17.67 0-32 14.33-32 32v144H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h144v144c0 17.67 14.33 32 32 32h32c17.67 0 32-14.33 32-32V304h144c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t<\/span>\n\n\t\t\t\t\t\t<\/summary>\n\t\t\t\t<div role=\"region\" aria-labelledby=\"e-n-accordion-item-4674\" class=\"elementor-element elementor-element-6078624 e-con-full mbd-gap-s e-flex e-con e-child\" data-id=\"6078624\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-d3cac6c elementor-widget elementor-widget-text-editor\" data-id=\"d3cac6c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Yes, for crawlers that respect it, and those include the documented crawlers of the major search engines and AI providers. It is not a technical barrier: anyone who wants to ignore the convention can do so.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/details>\n\t\t\t\t\t\t<details id=\"e-n-accordion-item-4675\" class=\"e-n-accordion-item\" >\n\t\t\t\t<summary class=\"e-n-accordion-item-title\" data-accordion-index=\"6\" tabindex=\"-1\" aria-expanded=\"false\" aria-controls=\"e-n-accordion-item-4675\" >\n\t\t\t\t\t<span class='e-n-accordion-item-title-header'><h3 class=\"e-n-accordion-item-title-text\"> Is ignoring robots.txt illegal? <\/h3><\/span>\n\t\t\t\t\t\t\t<span class='e-n-accordion-item-title-icon'>\n\t\t\t<span class='e-opened' ><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-minus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h384c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t\t<span class='e-closed'><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-plus\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M416 208H272V64c0-17.67-14.33-32-32-32h-32c-17.67 0-32 14.33-32 32v144H32c-17.67 0-32 14.33-32 32v32c0 17.67 14.33 32 32 32h144v144c0 17.67 14.33 32 32 32h32c17.67 0 32-14.33 32-32V304h144c17.67 0 32-14.33 32-32v-32c0-17.67-14.33-32-32-32z\"><\/path><\/svg><\/span>\n\t\t<\/span>\n\n\t\t\t\t\t\t<\/summary>\n\t\t\t\t<div role=\"region\" aria-labelledby=\"e-n-accordion-item-4675\" class=\"elementor-element elementor-element-2a9a691 e-con-full mbd-gap-s e-flex e-con e-child\" data-id=\"2a9a691\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-9a87176 elementor-widget elementor-widget-text-editor\" data-id=\"9a87176\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Robots.txt is a technical convention, not a law. Whether ignoring it has legal consequences depends on the country, the terms of use and copyright. That is a question for a lawyer, not a developer.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/details>\n\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-e2f25f8 e-con-boxed mbd-section mbd-inset mbd-surface--tint mbd-gap-xl e-flex e-con e-parent\" data-id=\"e2f25f8\" data-element_type=\"container\" data-e-type=\"container\" id=\"verder-lezen\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-8c816f5 e-con-full home-head mbd-gap-m e-flex e-con e-child\" data-id=\"8c816f5\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-f698709 mbd-eyebrow elementor-widget elementor-widget-heading\" data-id=\"f698709\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">More on this topic<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2dfdb37 elementor-widget elementor-widget-heading\" data-id=\"2dfdb37\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Further reading<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-2e5307a e-grid e-con-full mbd-gap-xl e-con e-child\" data-id=\"2e5307a\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t<div class=\"elementor-element elementor-element-41225f4 e-con-full svc-duo__col mbd-gap-s e-flex e-con e-child\" data-id=\"41225f4\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-ecb984a home-label elementor-widget elementor-widget-heading\" data-id=\"ecb984a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">The related service<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3abee03 elementor-widget elementor-widget-heading\" data-id=\"3abee03\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Technical SEO as a service<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-df272ff elementor-widget elementor-widget-text-editor\" data-id=\"df272ff\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Not sure whether search engines and AI crawlers can fetch and read your website? The service page explains how I investigate and resolve crawlability, rendering and indexing issues.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5385c57 mbd-btn--secondary elementor-widget elementor-widget-button\" data-id=\"5385c57\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm\" href=\"https:\/\/madebydries.be\/en\/seo-consultant\/technical-seo\/\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t\t\t\t<span class=\"elementor-button-text\">Explore technical SEO<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-77b2a35 e-con-full svc-duo__col mbd-gap-s e-flex e-con e-child\" data-id=\"77b2a35\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-0890825 home-label elementor-widget elementor-widget-heading\" data-id=\"0890825\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">Related articles<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a949355 kb-list kb-related elementor-widget elementor-widget-text-editor\" data-id=\"a949355\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<ul><li><a href=\"https:\/\/madebydries.be\/en\/knowledge-base\/llms-txt\/\">llms.txt: what it does \u2014 and what it does not<\/a><\/li><li><a href=\"https:\/\/madebydries.be\/en\/knowledge-base\/website-not-indexed\/\">Website or page not indexed? All the causes explained<\/a><\/li><li><a href=\"https:\/\/madebydries.be\/en\/knowledge-base\/visibility-in-chatgpt\/\">Becoming visible in ChatGPT: what can you influence?<\/a><\/li><li><a href=\"https:\/\/madebydries.be\/en\/knowledge-base\/what-is-geo\/\">What is GEO (Generative Engine Optimization)?<\/a><\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-c89d02e e-con-boxed mbd-section mbd-gap-xl e-flex e-con e-parent\" data-id=\"c89d02e\" data-element_type=\"container\" data-e-type=\"container\" id=\"auteur\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-46dbec3 e-grid e-con-full svc-author mbd-card mbd-card--tint mbd-gap-xl e-con e-child\" data-id=\"46dbec3\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-e88ce44 svc-portrait elementor-widget elementor-widget-html\" data-id=\"e88ce44\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t\t<img loading=\"lazy\" width=\"800\" height=\"1000\" src=\"https:\/\/madebydries.be\/wp-content\/uploads\/2026\/11\/Dries-Nijs-beste-seo-geo-webdesigner-van-Limburg-819x1024.webp\" class=\"attachment-large size-large\" alt=\"Dries Nijs, Organic Web Specialist and author of this article\" loading=\"lazy\" sizes=\"auto, (min-width: 1025px) 300px, (min-width: 768px) 30vw, 224px\" decoding=\"async\" srcset=\"https:\/\/madebydries.be\/wp-content\/uploads\/2026\/11\/Dries-Nijs-beste-seo-geo-webdesigner-van-Limburg-819x1024.webp 819w, https:\/\/madebydries.be\/wp-content\/uploads\/2026\/11\/Dries-Nijs-beste-seo-geo-webdesigner-van-Limburg-240x300.webp 240w, https:\/\/madebydries.be\/wp-content\/uploads\/2026\/11\/Dries-Nijs-beste-seo-geo-webdesigner-van-Limburg-768x960.webp 768w, https:\/\/madebydries.be\/wp-content\/uploads\/2026\/11\/Dries-Nijs-beste-seo-geo-webdesigner-van-Limburg.webp 1122w\" \/>\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-639716a e-con-full mbd-gap-m e-flex e-con e-child\" data-id=\"639716a\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-fab2356 home-label elementor-widget elementor-widget-heading\" data-id=\"fab2356\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<p class=\"elementor-heading-title elementor-size-default\">About the author<\/p>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e105f5d svc-author__name elementor-widget elementor-widget-heading\" data-id=\"e105f5d\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Dries Nijs, Organic Web Specialist<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-49b2b71 mbd-lead elementor-widget elementor-widget-text-editor\" data-id=\"49b2b71\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>I am Dries Nijs, a developer and SEO consultant from Maasmechelen. I check crawlability in the code, the server settings and the logs, not just in a tool. MadeByDries is the personal brand under which I work.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e4e4d80 mbd-btn--secondary elementor-widget elementor-widget-button\" data-id=\"e4e4d80\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm\" href=\"https:\/\/madebydries.be\/en\/about-dries-nijs\/\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t\t\t\t<span class=\"elementor-button-text\">More about Dries<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t<section class=\"elementor-element elementor-element-c0abf72 e-con-boxed home-cta mbd-section mbd-section--lg mbd-inset mbd-surface mbd-gap-l mbd-bg-flow mbd-bg-flow--light mbd-bg-flow--both mbd-bg-flow--strong e-flex e-con e-parent\" data-id=\"c0abf72\" data-element_type=\"container\" data-e-type=\"container\" id=\"contact\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-8a96aff home-cta__title elementor-widget elementor-widget-heading\" data-id=\"8a96aff\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Do you know which crawlers your website allows?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5980ffc mbd-lead elementor-widget elementor-widget-text-editor\" data-id=\"5980ffc\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>Tell me about your website. Then we can check what search engines and AI crawlers can read from it today.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-4cf31b7 e-con-full mbd-gap-s e-flex e-con e-child\" data-id=\"4cf31b7\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-0f5152b mbd-btn--icon mbd-btn--lg elementor-widget elementor-widget-button\" data-id=\"0f5152b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm\" href=\"https:\/\/madebydries.be\/en\/contact\/\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-icon\">\n\t\t\t\t<svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-arrow-right\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M190.5 66.9l22.2-22.2c9.4-9.4 24.6-9.4 33.9 0L441 239c9.4 9.4 9.4 24.6 0 33.9L246.6 467.3c-9.4 9.4-24.6 9.4-33.9 0l-22.2-22.2c-9.5-9.5-9.3-25 .4-34.3L311.4 296H24c-13.3 0-24-10.7-24-24v-32c0-13.3 10.7-24 24-24h287.4L190.9 101.2c-9.8-9.3-10-24.8-.4-34.3z\"><\/path><\/svg>\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t<span class=\"elementor-button-text\">Discuss your project<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-94bc5a7 mbd-btn--secondary mbd-btn--lg elementor-widget elementor-widget-button\" data-id=\"94bc5a7\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm\" href=\"https:\/\/madebydries.be\/en\/seo-consultant\/technical-seo\/\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t\t\t\t<span class=\"elementor-button-text\">Technical SEO<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>robots.txt and AI crawlers explained: which crawlers are officially documented, what they do and what you can allow or block.<\/p>\n","protected":false},"author":1,"featured_media":1366,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[103],"tags":[],"class_list":["post-2319","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo-technology"],"ase":null,"_links":{"self":[{"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/posts\/2319","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/comments?post=2319"}],"version-history":[{"count":10,"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/posts\/2319\/revisions"}],"predecessor-version":[{"id":9046,"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/posts\/2319\/revisions\/9046"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/media\/1366"}],"wp:attachment":[{"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/media?parent=2319"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/categories?post=2319"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/madebydries.be\/en\/wp-json\/wp\/v2\/tags?post=2319"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}