---
language: en
title: "Mixed-use crawler"
description: "A mixed-use crawler serves more than one documented purpose. Learn how to review uses and controls without assuming that each request reveals its ultimate destination."
canonical: "https://tropk.ai/en/resources/glossary/mixed-use-crawler"
last_modified: "2026-10-12T16:11:33.609Z"
---

# Mixed-use crawler

Reviewed on 2026-10-06.

## What it means

A mixed-use [crawler](/en/resources/glossary/crawler) is a collection client whose activity serves more than one documented purpose, such as search [indexing](/en/resources/glossary/indexing) and providing content to AI systems. The term describes a combination of uses without defining a protocol of its own. Cloudflare recognizes that a bot can exhibit multiple behaviors, avoiding classifications that are necessarily exclusive.

## Why it matters for marketing

For marketing, combined uses make decisions based solely on an AI bot label insufficient. Blocking a client can affect a purpose the team wants to preserve. Before requesting changes, the responsible person needs to understand which uses the operator declares and whether separate controls exist for collection, training, [retrieval](/en/resources/glossary/information-retrieval), or other functions that are actually documented.

## Example in practice

Imagine a portal that wants to allow discovery in Google Search while restricting the uses covered by [Google-Extended](/en/resources/glossary/google-extended). On reviewing the documentation, the team sees that this token controls training and [grounding](/en/resources/glossary/grounding) in [Gemini](/en/resources/glossary/gemini) products, using collections with existing identifiers. It requests a rule targeting the documented token, without inventing a separate HTTP bot called Google-Extended.

## How to apply or verify

To verify a mixed-use situation, map the client's documentation to the products affected by each control. Record the chosen policy and examine requests and responses before and after any change. For Google-Extended, the documentation states that the control does not affect inclusion or ranking in Search; this does not justify generalizing that distinction to other operators.

## Important distinctions

An operator may maintain several specialized bots without each bot being mixed-use. Conversely, the same access may be involved in different uses without revealing its subsequent destination in the log. Do not infer training, indexing, or [citation](/en/resources/glossary/ai-citation) from a single request. Control tokens, HTTP clients, and products must be distinguished to correctly interpret the policy applied.

## Sources and references

- [Bots](<https://developers.cloudflare.com/bots/concepts/bot/>) — Cloudflare
- [List of Google's common crawlers](<https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers>) — Google

## Continue a leitura

- [Crawler purpose category](</en/resources/glossary/crawler-purpose-category>)
- [Training data](</en/resources/glossary/training-data>)
- [Google-Extended](</en/resources/glossary/google-extended>)

## About the author

[Vinicius Pacheco](</en/team/vinicius-pacheco>)

Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content.

---

[All pages in Markdown](<https://tropk.ai/sitemap.md>)