--- language: en title: "meta-webindexer" description: "meta-webindexer crawls pages to improve Meta AI search. Learn about access controls and why visits do not prove citations in responses." canonical: "https://tropk.ai/en/resources/glossary/meta-webindexer" last_modified: "2026-10-12T16:11:14.319Z" --- # meta-webindexer Reviewed on 2026-10-06. ## What it means meta-webindexer identifies the [crawler](/en/resources/glossary/crawler) Meta uses to improve the quality of [Meta AI](/en/resources/glossary/meta-ai) search results. Meta describes analyzing online content to increase relevance and accuracy and says that allowing this agent in [robots.txt](/en/resources/glossary/robots-txt) helps the system cite and link to pages in responses. This describes a possible use, not a guarantee of selection. ## Why it matters for marketing For marketing, this distinction makes it possible to assess access related to Meta [AI search](/en/resources/glossary/ai-search) without automatically grouping it with data collection for training. A team that wants its pages to be discoverable needs to review this agent’s rules. However, measuring visits from meta-webindexer does not reveal how many responses cited the company or how many people reached the site through those responses. ## Example in practice Consider a manufacturer that wants to make its public assembly manual available for searches in Meta AI. When reviewing robots.txt, the team finds a restriction for meta-webindexer that covers the manual. It requests a change to that rule for the document, in line with its access policy, and keeps checking for citations as a separate, later assessment. ## How to apply or verify To verify the proposed change, the technical lead checks the meta-webindexer token, the manual’s path, and the robots.txt actually served by the host. They allow for the up to 24-hour period Meta gives for cached changes to take effect. Then they monitor requests and HTTP responses for the manual; these records verify access, while citations require observing Meta [AI responses](/en/resources/glossary/ai-response). ## Important distinctions The stated search purpose of meta-webindexer differs from the training and direct [indexing](/en/resources/glossary/indexing) uses described for [meta-externalagent](/en/resources/glossary/meta-externalagent). Do not infer a complete content-use policy from the agent’s name alone. The User-Agent does not authenticate its origin. Allowing [crawling](/en/resources/glossary/crawling) does not guarantee indexing or ranking, and blocking future access does not prove that previous copies have been deleted. ## Sources and references - [Meta web crawlers]() — Meta ## Continue a leitura - [Indexing]() - [meta-externalagent]() - [meta-externalfetcher]() ## About the author [Vinicius Pacheco]() Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content. --- [All pages in Markdown]()