---
language: en
title: "Crawler"
description: "A crawler is a program that visits pages to collect content. Its purpose and the server’s responses guide the assessment of access for search or AI."
canonical: "https://tropk.ai/en/resources/glossary/crawler"
last_modified: "2026-10-12T16:13:19.326Z"
---

# Crawler

Reviewed on 2026-10-06.

## What it means

A crawler is a program that requests pages and other files to collect content automatically. It can follow URLs found in links and sitemaps. Its purpose depends on its operator: building a [search index](/en/resources/glossary/search-index) and gathering material that may be used for training are different uses. Visiting a page does not prove that either of these end results occurred.

## Why it matters for marketing

The marketing team needs to identify which crawler is finding its materials before diagnosing an access issue. A page available to people may be blocked to an agent by the server or hosting infrastructure. Allowing access requires a decision about its purpose; allowing search and allowing collection for training are not necessarily equivalent authorizations.

## Example in practice

In a hypothetical case, a manufacturer publishes a coffee maker’s product sheet. People receive HTTP 200, but the logs show HTTP 403 for visits by the search crawler identified by the technical team. The [SEO](/en/resources/glossary/seo) lead requests a review of the corresponding blocking rule and a new test before attributing the absence to problems with the text.

## How to apply or verify

Check the operator’s documentation, the agent identifier, and the available official methods for verifying origin. Cross-check the time, URL, [HTTP status](/en/resources/glossary/http-status) code, and content delivered in the server logs. Review [robots.txt](/en/resources/glossary/robots-txt) and hosting rules. If rendering is involved, check that the main content was accessible at that stage using the method supported by the service being evaluated.

## Important distinctions

Crawler names the agent; [crawling](/en/resources/glossary/crawling) describes the activity. [Indexing](/en/resources/glossary/indexing) is processing that may occur after collection. Not all automated access is continuous crawling: OpenAI distinguishes user-initiated visits from its automated bots. A recorded visit also does not demonstrate [citation](/en/resources/glossary/ai-citation), recommendation, an update to a model’s parameters, or exposure to real people.

## Sources and references

- [In-depth guide to how Google Search works](<https://developers.google.com/search/docs/fundamentals/how-search-works>) — Google Search Central
- [OpenAI crawlers](<https://platform.openai.com/api/docs/bots>) — OpenAI

## Continue a leitura

- [Googlebot](</en/resources/glossary/googlebot>)
- [Crawling](</en/resources/glossary/crawling>)
- [robots.txt](</en/resources/glossary/robots-txt>)

## About the author

[Vinicius Pacheco](</en/team/vinicius-pacheco>)

Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content.

---

[All pages in Markdown](<https://tropk.ai/sitemap.md>)