---
language: en
title: "robots.txt"
description: "robots.txt provides access rules for crawlers. Understand its limitations, how it differs from noindex, and how to distinguish search bots from training bots."
canonical: "https://tropk.ai/en/resources/glossary/robots-txt"
last_modified: "2026-10-12T16:08:33.086Z"
---

# robots.txt

Reviewed on 2026-10-06.

## What it means

robots.txt is a file that publishes access rules for crawlers that follow the protocol. It is located at /robots.txt on the corresponding host and lets you indicate which paths an agent may crawl. Rules are targeted by user-agent. The file manages automated access; it is not authentication, privacy control, or a guarantee that results will be removed.

## Why it matters for marketing

For marketing and [SEO](/en/resources/glossary/seo), an incorrect rule can prevent access to relevant pages. In AI projects, it is also necessary to distinguish between bots’ functions. OpenAI separates [OAI-SearchBot](/en/resources/glossary/oai-searchbot), used for search, from [GPTBot](/en/resources/glossary/gptbot), associated with content that may be used for training. The settings are independent and should reflect a conscious decision about each purpose.

## Example in practice

A site retains Disallow: / for all agents after leaving the test environment. People can open the pages normally, but crawlers that respect the rule cannot access them. In another case, blocking GPTBot is not equivalent to blocking OAI-SearchBot. Copying a list of names without understanding these differences can undermine the intended purpose.

## How to apply or check

Read the file on the actual host and check the [HTTP status](/en/resources/glossary/http-status), groups, and paths. Test important URLs against the effective rules, and also investigate the firewall and CDN. If there is a separate policy for search and training, document the agents involved and consult their current documentation. Check the result after the change, since a file saved locally does not confirm the public response.

## Important distinctions

robots.txt and [noindex](/en/resources/glossary/noindex) serve different purposes. A blocked URL may still appear in results if it is known through links, even if its content is not crawled. For Google to read a noindex rule, it needs to access the page. Restricted content requires appropriate protection; a path written in robots.txt remains a publicly discoverable address.

## Sources and references

- [Introduction to robots.txt](<https://developers.google.com/search/docs/crawling-indexing/robots/intro>) — Google Search Central
- [Overview of OpenAI crawlers](<https://openai.com/searchbot>) — OpenAI

## Continue a leitura

- [Crawling](</en/resources/glossary/crawling>)
- [llms.txt](</en/resources/glossary/llms-txt>)
- [noindex](</en/resources/glossary/noindex>)

## About the author

[Vinicius Pacheco](</en/team/vinicius-pacheco>)

Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content.

---

[All pages in Markdown](<https://tropk.ai/sitemap.md>)