--- language: en title: "ClaudeBot" description: "ClaudeBot collects content that may contribute to training Anthropic's models. See robots.txt controls and the limits of identifying requests." canonical: "https://tropk.ai/en/resources/glossary/claudebot" last_modified: "2026-10-12T16:11:29.311Z" --- # ClaudeBot Reviewed on 2026-10-06. ## What it means ClaudeBot is Anthropic's web [crawler](/en/resources/glossary/crawler), which collects content from the web that may be useful for training its [generative AI](/en/resources/glossary/generative-ai) models. According to the documentation, restricting its access signals that future site materials should be excluded from these training datasets. The company distinguishes it from agents used for search and to retrieve pages at users' request. ## Why it matters for marketing For marketing and editorial management, this distinction makes it possible to discuss use of the archive without conflating all forms of [Claude](/en/resources/glossary/claude) access. Publishing rights and interest in discovery may lead to different decisions. Visit frequency also matters: an access policy may need adjustment when [crawling](/en/resources/glossary/crawling) affects the service offered to visitors. ## Example in practice Imagine a cultural archive that allows ClaudeBot to access its public catalog, but notices several requests arriving close together while users are consulting documents. The infrastructure team proposes testing [Crawl-delay](/en/resources/glossary/crawl-delay) in this agent's rule and comparing request times and catalog availability after the change. The decision concerns the frequency of collection for training, without confusing this control with a complete denial of access. ## How to apply or verify Anthropic states that it respects [robots.txt](/en/resources/glossary/robots-txt) and supports the non-standard Crawl-delay extension. Check the file in the root directory and on each relevant subdomain, then monitor the logs. Compare the source with Anthropic's official IP list. The company warns that blocking only IPs may prevent robots.txt from being read and may not ensure a persistent denial. ## Important distinctions ClaudeBot is not [Claude-SearchBot](/en/resources/glossary/claude-searchbot) or [Claude-User](/en/resources/glossary/claude-user). The training rule should not be described as refusing all searches or all user requests. The document limits the exclusion to future materials; the control does not prove that past [training data](/en/resources/glossary/training-data) will be deleted. Allowing collection also does not guarantee that information will appear accurately or be cited in responses. ## Sources and references - [Does Anthropic crawl data from the web, and how can site owners block the crawler?]() — Anthropic ## Continue a leitura - [Claude-SearchBot]() - [Training data]() - [robots.txt]() ## About the author [Vinicius Pacheco]() Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content. --- [All pages in Markdown]()