---
language: en
title: "Crawling"
description: "Crawling is automated access to pages by a bot. Learn how to check for blocks and server responses, and distinguish access, indexing, and citation."
canonical: "https://tropk.ai/en/resources/glossary/crawling"
last_modified: "2026-10-12T16:14:51.920Z"
---

# Crawling

Reviewed on 2026-10-06.

## What it means

Crawling is the process by which an automated agent accesses URLs and retrieves their resources. In search engines, it is used to discover and examine new or updated pages. The agent may follow links and use lists such as sitemaps. A visit does not mean that the page will be included in the index, shown in a result, or cited in an [AI response](/en/resources/glossary/ai-response).

## Why it matters for marketing

For marketing, crawling is a read-access dependency for many discovery systems. Published, polished content can remain inaccessible due to a server error, firewall, authentication, or [robots.txt](/en/resources/glossary/robots-txt). Checking this step helps avoid investing in rewriting when the real problem is access. The investigation should consider the specific agent and destination.

## Example in practice

A page responds normally in a person’s browser, but the server returns 403 to the [crawler](/en/resources/glossary/crawler). A human test alone does not prove automated access. In another case, access returns 200, but the main text depends on a blocked resource. The page was retrieved, but the information that matters may still be unavailable to the reading process.

## How to apply or verify

Check status, redirects, access rules, and resource availability. Use logs with agent identification and appropriate validation where possible; a declared user-agent can be imitated. For Google, inspection tools help observe access and rendering. Record the evidence by URL and date, distinguishing a current test from a historical bot visit.

## Important distinctions

Crawling and [indexing](/en/resources/glossary/indexing) are different stages. You should also distinguish search, training, and user-initiated bots: their goals and rules vary. Allowing access does not guarantee a recommendation, and blocking one agent does not describe the behavior of all agents. The technical report should specify which access was tested and what result was observed.

## Sources and references

- [In-depth guide to how Google Search works](<https://developers.google.com/search/docs/fundamentals/how-search-works>) — Google Search Central
- [Overview of OpenAI crawlers](<https://openai.com/searchbot>) — OpenAI

## Continue a leitura

- [Indexing](</en/resources/glossary/indexing>)
- [HTTP Status](</en/resources/glossary/http-status>)
- [robots.txt](</en/resources/glossary/robots-txt>)

## About the author

[Vinicius Pacheco](</en/team/vinicius-pacheco>)

Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content.

---

[All pages in Markdown](<https://tropk.ai/sitemap.md>)