---
language: en
title: "Crawl-delay"
description: "Crawl-delay is a robots.txt extension for requesting a slower crawl rate. Support depends on the operator; Google does not interpret this field."
canonical: "https://tropk.ai/en/resources/glossary/crawl-delay"
last_modified: "2026-10-12T16:11:05.792Z"
---

# Crawl-delay

Reviewed on 2026-10-06.

## What it means

Crawl-delay is a non-standard [robots.txt](/en/resources/glossary/robots-txt) extension used to request that a [crawler](/en/resources/glossary/crawler) reduce its request rate. It is not part of the rules defined by RFC 9309, and its effect depends on the operator. Anthropic states that it supports the extension to limit activity; Google’s documentation says this field is not supported.

## Why it matters for marketing

For marketing, this matters when bot access increases the load on pages that need to remain available. The team can request rate control without assuming that all services will interpret the same instruction. The goal should be defined with infrastructure: monitor pressure on the server and preserve permitted access to relevant content.

## Example in practice

Suppose a recipe collection receives visits identified as [ClaudeBot](/en/resources/glossary/claudebot) at short intervals, along with increased latency. After confirming the source, infrastructure adds Crawl-delay to that bot’s group in robots.txt. The team then monitors request intervals and latency on recipe pages again to decide whether the measure meets the collection’s needs.

## How to apply or verify

For the recipe collection, check that robots.txt is accessible on the correct host and that the instruction is in the intended crawler’s group. Read the operator’s current support statement. Compare request intervals and latency before and after the change over defined periods, recording other load changes that could explain the results.

## Important distinctions

Crawl-delay does not block paths, prevent [indexing](/en/resources/glossary/indexing), or impose a server-side limit on its own. Appropriate infrastructure controls are needed to enforce traffic restrictions. Anthropic’s stated support does not automatically apply to other bots; adding the extension to Google’s group will not make its crawlers respect a requested interval.

## Sources and references

- [RFC 9309: Robots Exclusion Protocol](<https://www.rfc-editor.org/rfc/rfc9309.html>) — RFC Editor
- [How Google interprets the robots.txt specification](<https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec>) — Google
- [Does Anthropic crawl data from the web, and how can site owners block the crawler?](<https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler>) — Anthropic

## Continue a leitura

- [ClaudeBot](</en/resources/glossary/claudebot>)
- [Crawler](</en/resources/glossary/crawler>)
- [robots.txt](</en/resources/glossary/robots-txt>)

## About the author

[Vinicius Pacheco](</en/team/vinicius-pacheco>)

Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content.

---

[All pages in Markdown](<https://tropk.ai/sitemap.md>)