--- language: en title: "Robots.txt for AI: which bots to allow and how to configure it" description: "Separate search, training, and user-requested access when configuring robots.txt for OpenAI, Perplexity, Claude, and Google. Includes file templates and validation." canonical: "https://tropk.ai/en/blog/robots-txt-for-ai-which-bots-to-allow-and-how-to-configure-it" last_modified: "2026-10-12T16:05:25.925Z" --- # Robots.txt for AI: which bots to allow and how to configure it Separate search, training, and user-requested access when configuring robots.txt for OpenAI, Perplexity, Claude, and Google. Includes file templates and validation. [Luiz Gardelin]() Published on 2026-04-14T12:07:36.000Z Categorias: How to, Best practices Configuring [robots.txt](/en/resources/glossary/robots-txt) for AI starts with distinguishing three activities: [crawling](/en/resources/glossary/crawling) for search, data collection for training, and access to a page requested by a person. They may use different agents from the same company, with different controls and consequences. Blocking [GPTBot](/en/resources/glossary/gptbot) is not the same as blocking [ChatGPT](/en/resources/glossary/chatgpt) Search. Nor is it correct to treat [ChatGPT-User](/en/resources/glossary/chatgpt-user) as the [crawler](/en/resources/glossary/crawler) that determines inclusion in search. This guide explains the differences, presents three configuration templates, and shows how to check robots.txt, your CDN, and logs together. The agents' roles were reviewed on **October 6, 2026**, using the platforms' own documentation. The examples are text files that need to be adapted to your domain and your site's access policy. ## Key points - robots.txt states crawling preferences; privacy requires authentication and authorization. - OAI-SearchBot controls crawling for ChatGPT search, while GPTBot is for training. - ChatGPT-User and Perplexity-User make user-requested accesses and are handled differently by robots.txt. - Google-Extended covers training and grounding in certain Gemini/Vertex products without controlling Google Search. - Check the specific groups, paths, CDN, and logs before concluding that a crawler has access. ## What is robots.txt robots.txt is a text file at the root of an origin, such as `https://example.com/robots.txt`, that declares which paths compatible robots may crawl. It is part of the Robots Exclusion Protocol, formalized in [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309). The protocol is a crawling instruction, not a security barrier. Private files need authentication and authorization. Listing a directory in `Disallow` still makes the directory name public to anyone who reads the file. A robots.txt file on one host does not automatically control another subdomain. If `www.example.com` and `docs.example.com` have different policies, each origin needs to publish and validate its own configuration. ### Crawling and indexing are different A URL blocked by robots.txt may still be known to a search engine through links or previously available information. Blocking prevents a crawler that follows the rule from accessing the content; it is not a universal instruction to remove the page from search results. To request that a page be excluded from Google Search, use appropriate [indexing](/en/resources/glossary/indexing) mechanisms, such as `noindex`, and allow [Googlebot](/en/resources/glossary/googlebot) to read the instruction. Blocking access and adding `noindex` at the same time may prevent the exclusion instruction itself from being read. ## How the syntax works A group starts with one or more `User-agent` lines and is followed by path rules. The most common directives are: - `User-agent`: identifies a family of robots or uses `*` as a generic group. - `Disallow`: declares paths that should not be crawled. - `Allow`: declares paths that are allowed, including exceptions within a broader directory. - `Sitemap`: provides the location of a [sitemap](/en/resources/glossary/sitemap); it is a widely used extension, not an access permission. A minimal file allowing access could be: ```text User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` ### The rule is not simply “the last line wins” Matching depends on the agent and the path. More specific path rules take precedence over broader rules; when there is an equivalent match between `Allow` and `Disallow`, the allow rule takes precedence. When there is a group specific to the agent, do not assume it inherits all restrictions from the `*`. group.`Allow: /`For example, a specific group with may allow that crawler to access a path blocked by the generic group. Repeat necessary restrictions in specific groups and test the relevant public and private URLs.`*`Characters such as `$` and ### are supported by the protocol for path patterns. Paths should match the URLs actually used; details such as capitalization, slashes, and encoding can change the match. ```text User-agent: * Disallow: /area-interna/ Allow: /area-interna/apresentacao-publica/ Sitemap: https://example.com/sitemap.xml ``` Partial allow example ## The example documents that a public presentation within a directory is allowed. It does not make the rest of the internal area secure: the server must require authorization before providing private information. | Which bots matter for search, training, and user actions | Agent or token | Company | Documented function | | --- | --- | --- | --- | | `OAI-SearchBot` | How to interpret the control | OpenAI | Crawling for ChatGPT search features | | `GPTBot` | This is the search crawling and exclusion control described by OpenAI | OpenAI | Content that may be used to train generative models | | `ChatGPT-User` | Can be blocked independently of OAI-SearchBot | OpenAI | Access associated with actions by ChatGPT users and GPTs | | `OAI-AdsBot` | Does not determine inclusion in search; robots.txt rules may not apply | OpenAI | Validation and analysis of pages submitted as ad destinations | | `PerplexityBot` | This is an advertising function, separate from search and training | Perplexity | Presenting and linking to sites in search results | | `Perplexity-User` | The platform recommends allowing the bot and its official IPs | Perplexity | Fetching a page at a user's request | | `ClaudeBot` | The documentation says it generally ignores robots.txt in this mode | Anthropic | Collecting content that may contribute to training | | `Claude-SearchBot` | It has a different function from Claude-SearchBot and Claude-User | Anthropic | Crawling content to improve search results | | `Claude-User` | Restrictions may reduce visibility in Claude results | Anthropic | Retrieving pages at a user's request | | `Googlebot` | Anthropic says its bots respect robots.txt | Google | Crawling for Google Search and its features | | `Google-Extended` | Relevant to content access for AI Overviews and AI Mode | Google | Training and grounding control token for certain Gemini and Vertex products | It is not a separate crawler and does not affect inclusion or ranking in Google Search[Sources: ](https://developers.openai.com/api/docs/bots), [OpenAI](https://docs.perplexity.ai/guides/bots), [Perplexity](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)Anthropic[ and ](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers). ### Google OpenAI: allow search without allowing training`OAI-SearchBot`OpenAI documents that `GPTBot` and have independent settings. You can allow the former for search and block the latter to signal that the content should not be used for training. `ChatGPT-User`The documentation states that sites that exclude [OAI-SearchBot](/en/resources/glossary/oai-searchbot) do not appear in ChatGPT search answers, although they may still appear as navigation links. This distinction avoids turning a crawling restriction into a promise that a site will disappear entirely from all mentions. `OAI-AdsBot` is used for actions initiated by people. It is not an automated crawler and does not determine whether a page can appear in search. OpenAI says robots.txt may not apply to these accesses. ### visits pages submitted as ads, evaluates safety, and may help establish ad relevance. Allowing this agent does not buy an organic citation; restricting it requires considering how destination pages submitted to ChatGPT Ads work. `PerplexityBot`[Perplexity](/en/resources/glossary/perplexity): search bot and user-requested access`Perplexity-User` has a search function and is not described as collecting data to train foundation models. retrieves pages in response to an individual request. The documentation distinguishes the two modes and says user-requested access generally ignores robots.txt. ### In addition to the file, Perplexity recommends checking firewall rules and verifying the agent against official IP ranges. Access allowed in robots.txt may still be blocked by infrastructure. Anthropic: three separate agents`ClaudeBot`, `Claude-SearchBot` and `Claude-User`. Therefore, it is not accurate to claim that blocking [ClaudeBot](/en/resources/glossary/claudebot) alone removes a site from every [Claude](/en/resources/glossary/claude) response. Anthropic says its bots respect robots.txt. It also documents support for `Crawl-delay`, a non-standardized extension that should not be treated as a universal control for other search engines. ### Google-Extended does not block AI Overviews [Google-Extended](/en/resources/glossary/google-extended) is a token for controlling whether content crawled by Google can be used to train future generations of [Gemini](/en/resources/glossary/gemini) and for [grounding](/en/resources/glossary/grounding) in products specified in the documentation, such as Gemini Apps and certain Vertex AI features. Blocking this token does not affect inclusion or ranking in Google Search. Generative search features use the Search ecosystem, with Googlebot and content presentation controls. The [official guide to AI features](https://developers.google.com/search/docs/appearance/ai-features) explains that a page must be indexed and eligible to appear with a snippet. Do not describe Google-Extended as “training only”: the current control also covers grounding uses outside Google Search. ## Three robots.txt templates to adapt Before choosing, define the policy: is the goal to make public content available to search, restrict collection for training by certain companies, or allow only one family of crawlers? There is no ideal template for every site. ### Template 1: allow crawling of public content ```text # Modelo ilustrativo: permite rastreamento público. User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` This file expresses an open policy. It does not guarantee that bots will visit or cite the site. Paths that require login remain protected by the application; the file should not replace that protection. ### Template 2: allow search and restrict training by OpenAI and Anthropic ```text # Grupo genérico: conteúdo público pode ser rastreado. User-agent: * Allow: / # Treinamento OpenAI. User-agent: GPTBot Disallow: / # Treinamento Anthropic. User-agent: ClaudeBot Disallow: / Sitemap: https://example.com/sitemap.xml ``` The template allows OAI-SearchBot, [PerplexityBot](/en/resources/glossary/perplexitybot), and [Claude-SearchBot](/en/resources/glossary/claude-searchbot) through the generic group. It restricts the two listed training agents. It does not block all companies, all datasets, or all AI use. If the policy should also restrict the Gemini/Vertex uses covered by Google-Extended, add the group below after reviewing the effect on grounding in those products. This control does not affect Google Search: ```text User-agent: Google-Extended Disallow: / ``` Review the list of agents from relevant companies periodically. Old or undocumented names should not be treated as proof of a complete opt-out policy. ### Template 3: allow Googlebot and restrict other compatible crawlers ```text # Restrição geral de rastreamento automático. User-agent: * Disallow: / # Exceção para Googlebot. User-agent: Googlebot Allow: / Sitemap: https://example.com/sitemap.xml ``` This template is restrictive. It may prevent crawling by other search engines and agents relevant to discovery. It does not guarantee that content will appear only on Google or prevent access by people or agents that do not follow the file. You should also evaluate specific Google agents, such as those for images and inspection tools, based on your configuration and the products you use. Choose restrictions based on a verifiable business decision, not as an attempt to “protect everything” without considering the effects. ## How to test the configuration 1. Open the robots.txt file for each relevant domain and subdomain, and check the HTTP response and the content served. 2. Check paths with the specific agents, especially when there are exceptions beyond the `*`. 3. Test representative public pages, recently changed URLs, and paths that require authentication. 4. Inspect server and CDN logs to identify requests, blocks, errors, and [HTTP status](/en/resources/glossary/http-status) codes. 5. Compare the reported agent against official origin verification methods; the User-Agent text can be imitated. 6. Check indexing with search engine tools and monitor subsequent crawls. A 404 response for robots.txt normally permits crawling under the protocol; network errors or server failures are handled differently. Do not conclude that “everything is allowed” just because your browser did not show the file. ### CDNs and firewalls can block access before content is served JavaScript challenges, CAPTCHAs, country-based blocks, and request limits can prevent a legitimate crawler from accessing a site. Permission in robots.txt does not automatically create an exception in the CDN. Use each company’s official IP sources and recommended verification methods. Lists can change. Avoid copying old addresses into a permanent exception, and do not rely solely on a name declared in the header. Even when the infrastructure allows access and the crawler visits the document, discovery, indexing, and source selection still need to be assessed. A visit in the logs confirms the observed request, not a future [citation](/en/resources/glossary/ai-citation). ## Are robots.txt and llms.txt complementary? [llms.txt](/en/blog/llms-txt-what-it-is-how-to-implement-it-and-what-it-changes-about-ai-visibility) provides context and curated links to agents that use the format. It does not grant access denied by the server or impose training rules. A document may be listed in the index and still be unavailable to the consumer. [Terms of use](/en/terms) and licensing are another dimension. No crawling file replaces a decision about authorization, usage rights, or confidentiality. If the goal is to keep content private, the solution must prevent delivery without credentials. ## How to track the results After correcting the controls, keep a record of the date, the rules changed, and the agents affected. Confirm access in the logs and indexing in the available tools, and track citations for comparable questions across platforms. Do not automatically attribute a change in visibility to robots.txt. Content updates, competitors, consulted sources, and system behavior can also affect responses. The correct configuration removes access barriers; whether a page is selected depends on additional criteria. ## Sources for further reading - [RFC 9309: Robots Exclusion Protocol]() - [OpenAI: crawlers and agents]() - [Perplexity: crawlers]() - [Anthropic: crawler functions and controls]() - [Google: common crawlers and Google-Extended]() - [Google: AI features and websites]() ## Frequently asked questions ### What is the difference between GPTBot, OAI-SearchBot, and ChatGPT-User? GPTBot crawls content that may be used to train OpenAI models. OAI-SearchBot crawls for ChatGPT search features and is the relevant agent for managing that access. ChatGPT-User retrieves pages during certain user-initiated actions, does not determine inclusion in search, and may not apply robots.txt to these accesses. ### Can you block OpenAI training without blocking ChatGPT Search? Yes. OpenAI documents separate controls for GPTBot and OAI-SearchBot. A site can restrict GPTBot in robots.txt and allow OAI-SearchBot to crawl for search. You should also check the CDN and firewall; allowing crawling does not guarantee that a page will be cited. ### Does blocking Google-Extended remove a site from AI Overviews or Google Search? No. Google-Extended controls training and grounding uses in certain Gemini and Vertex AI products, without affecting inclusion or ranking in Google Search. Crawling and content presentation in Search AI features are managed by controls such as Googlebot and indexing and snippet directives. ### What is the difference between ClaudeBot and Claude-SearchBot? According to Anthropic, ClaudeBot collects content that may contribute to model training, while Claude-SearchBot crawls to improve search results. Claude-User is a third agent that retrieves pages at users’ request. The agents should be treated separately when setting robots.txt permissions. ### Does robots.txt prevent access to private content and remove pages from search engines? No. robots.txt states preferences for compliant robots and does not protect private information. Confidential content requires authentication and authorization. A blocked URL may remain known to a search engine; instructions such as noindex must be accessible to the crawler to be processed. ### How can you check whether AI bots can access a site after changing robots.txt? Check the robots.txt file for each origin, test agent and path matching, and examine server and CDN logs. Check for firewall blocks, challenges, and HTTP errors, using official documentation to validate bots’ origins. Then track indexing and citations separately, because observed access does not prove selection as a source. ## About the author [Luiz Gardelin]() Luiz Gardelin is the founder of Tropk.ai and works in GEO/AEO, product vision, strategy, and growth, helping brands understand and improve their presence in AI responses. --- [All pages in Markdown]()