--- language: en title: "CCBot" description: "CCBot collects pages for the Common Crawl open repository. Learn about access rules, source verification, and limits on data reuse." canonical: "https://tropk.ai/en/resources/glossary/ccbot" last_modified: "2026-10-12T16:11:12.344Z" --- # CCBot Reviewed on 2026-10-06. ## What it means CCBot is the [crawler](/en/resources/glossary/crawler) for Common Crawl, a foundation that maintains an open repository of data collected from the web. Its stated goal is to broaden access to this material for analysis and research. It identifies itself in the User-Agent of its requests. Since the data is available for reuse, its collection should not automatically be attributed to a single assistant or model. ## Why it matters for marketing For marketing, CCBot calls for a decision about participating in an open archive, in addition to assessing traffic on your own site. A visit does not prove commercial interest or presence in an [AI response](/en/resources/glossary/ai-response). The team needs to understand that access for collection and subsequent use by third parties are different stages, with evidence and responsibilities that do not appear fully in server logs. ## Example in practice Suppose a school publishes teaching materials and decides to allow its lessons to be collected, while excluding a drafts directory. The [robots.txt](/en/resources/glossary/robots-txt) file contains a rule for CCBot that allows the entire site, including the drafts. The team requests that the directory be restricted while preserving access to the lessons it wants to make available in the Common Crawl open archive. ## How to apply or verify The technical lead should check the CCBot rule and the drafts path in the robots.txt file served by the host. To investigate actual visits, they can consult the official IP ranges or check reverse DNS and forward-confirmed reverse DNS, as described in the documentation. Common Crawl warns about agents that imitate its name and notes that reverse DNS is not yet available for IPv6. ## Important distinctions CCBot is not a search agent for a specific AI product. Allowing access does not guarantee inclusion in the archive, [indexing](/en/resources/glossary/indexing) by search engines, or [citation](/en/resources/glossary/ai-citation) by models that reuse data. The documentation explains how to block the CCBot token in robots.txt; a future restriction does not prove that earlier captures have been excluded. It also does not make it possible to determine which project used each collected document. ## Sources and references - [Common Crawl - CCBot]() — Common Crawl ## Continue a leitura - [AI2Bot]() - [User agent]() - [robots.txt]() ## About the author [Vinicius Pacheco]() Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content. --- [All pages in Markdown]()