--- language: en title: "Vinicius Pacheco | Author at Tropk" description: "Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content." canonical: "https://tropk.ai/en/team/vinicius-pacheco" last_modified: "2026-10-12T16:03:31.108Z" --- # Vinicius Pacheco [Back to the blog]() Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content. Crawling · Indexing · Structured data · Agent access [LinkedIn]() ## Resources by this author - [AI2Bot](): AI2Bot collects web content to train Ai2’s open models. See its published identification, traffic filtering, and verification limitations. - [Accessible, selected, and extractable](): Accessible, selected, and extractable is an editorial model for separating access, observable signals of selection, and the use of excerpts, without revealing internal decisions. - [Agent Payments Protocol](): AP2 uses verifiable mandates to authorize checkout and payment by agents. Learn how they link to the transaction and about the limits of specification v0.2. - [Agent2Agent](): Agent2Agent standardizes communication between AI agents: capability discovery, messages, and tasks, with access defined by each implementation. - [AI agent](): An AI agent performs a task by choosing steps and tools based on feedback from its environment, within defined permissions, completion criteria, and limits. - [Agents with verifiable signatures](): Agents with verifiable signatures sign HTTP requests. Validation identifies a key and its association, but does not by itself authorize access or actions. - [Agentic Commerce Protocol](): ACP defines commercial exchanges between agents and sellers, including checkout. Implementing the protocol and participating in a commerce experience are distinct steps. - [Aggregation by page and domain](): Choose the unit used to group citations: individual URLs or domains, with explicit rules for subdomains, duplicates, and traceability. - [Prompt clustering](): Prompt clustering brings together semantically similar questions. Learn how to review clusters to track topics without erasing differences in intent and context. - [AI hallucination](): An AI hallucination is fabricated or unsupported information presented as fact. Learn how to detect errors about brands and review claims and sources. - [Amazonbot](): Amazonbot collects content to improve Amazon products and services, with possible use in training. Learn the rules and how it differs from the search agent. - [Amzn-SearchBot](): Amzn-SearchBot collects content for Amazon search experiences, such as Alexa, without collecting it for generative training. See access rules and limitations. - [Amzn-User](): Amzn-User accesses the web in response to Amazon user actions, such as questions asked to Alexa. Learn about its purpose, identification, and the limits of robots.txt. - [Apple Intelligence](): Apple Intelligence brings together AI features built into Apple devices and apps. Each feature has device, software version, and language requirements. - [Applebot](): Applebot crawls content for Apple search experiences and other stated uses. Learn about access rules, identification, and separate controls. - [Applebot-Extended](): Applebot-Extended controls the use of content to train Apple’s models. Learn why blocking this token differs from preventing crawling or search. - [Machine learning](): Machine learning trains models on data to predict, classify, or generate content. Learn how to evaluate examples, performance, and limitations in marketing. - [AI assistant](): An AI assistant is a product that brings together an interface, models, and functions to help with tasks. Learn how context and features affect responses. - [Vector database](): A vector database stores and queries vectors by proximity. Learn how to evaluate filters, updates, and retrieval accuracy in your application. - [Bi-encoder](): A bi-encoder represents queries and documents as separate vectors. Learn how it is used for retrieval and how to test the selected content. - [Brave Leo](): Brave Leo is the assistant built into the Brave browser for chatting and working with web pages and documents, with model options and support from Brave Search. - [Agentic search](): Agentic search delegates research to software that consults sources and adapts its steps. Understand the criteria, evidence, and limits when evaluating offers. - [Semantic search](): Semantic search considers meaning and context beyond exact terms. Understand how it relates to intent, synonyms, and useful content without resorting to artificial repetition. - [CCBot](): CCBot collects pages for the Common Crawl open repository. Learn about access rules, source verification, and limits on data reuse. - [CLS: Cumulative Layout Shift](): CLS measures unexpected shifts of visible elements on a page. Learn about the visual stability score and how to investigate changes during use. - [Cache](): Cache reuses stored data or responses to handle new requests. Understand its effect on your website, content updates, and performance evaluation. - [Canonical](): Canonical indicates a preferred URL among identical or very similar pages. Learn about the role of rel=canonical and how to align links, the sitemap, and redirects. - [Crawler purpose category](): Crawler purpose category describes the documented use of access, such as search, training, or user-triggered activity. It guides rules without inferring function from the name. - [ChatGPT](): ChatGPT is OpenAI’s product for conversations, analyzing materials, and performing tasks. Its search can consult the web, with its own sources and access conditions. - [ChatGPT Atlas](): ChatGPT Atlas was OpenAI’s browser with ChatGPT built in. Documentation records its discontinuation and the transition to other browsing experiences. - [ChatGPT-User](): ChatGPT-User identifies certain accesses initiated by users of ChatGPT and custom GPTs. Its role differs from automated crawling and search. - [AI readiness checklist](): 46 checks covering access, content, and operations to investigate how AI agents and systems find and read your site. Separate search fundamentals from optional protocols. - [Technical SEO Checklist](): 51 checks to investigate indexing, crawling, performance, structured data, and mobile usability. Editable spreadsheet with methods, acceptance criteria, and fields for evidence. - [Chunking](): Chunking divides documents into sections for processing and retrieval. The size, boundaries, and overlap affect how much context each section preserves. - [Semantic chunking](): Semantic chunking aims to split text based on relationships in meaning. See how to check chunk boundaries and test the strategy in Portuguese. - [Claude-SearchBot](): Claude-SearchBot analyzes pages to improve Claude search results. Learn how to check access and distinguish indexing from training and citations. - [Claude-User](): Claude-User: page retrieval to answer requests in Claude, with a purpose and access controls distinct from Anthropic's other bots. - [ClaudeBot](): ClaudeBot collects content that may contribute to training Anthropic's models. See robots.txt controls and the limits of identifying requests. - [Cloudflare AI Crawl Control](): Cloudflare AI Crawl Control shows AI crawler activity and lets you manage access. Learn about the available actions and the limits of this visibility. - [Model collapse](): Model collapse is the degradation studied in recursive training with generated data. It is not equivalent to an isolated error or to every use of synthetic data. - [Comet](): Comet is Perplexity’s browser with AI assistance. See how tab context affects an analysis and which access permissions need to be checked. - [Candidate set](): A candidate set brings together passages selected before reranking. Learn how to inspect this stage to find gaps in retrieval. - [Implied query](): An implied query is a question inferred from incomplete context. Check referents and constraints to avoid searching for evidence based on the wrong interpretation. - [Synthetic query](): A synthetic query is a search formulated by the system based on the request it receives. Its wording may change, so results must be evaluated and constraints preserved. - [Position-Adjusted Word Count](): A GEO research metric that weights words in sentences citing a source according to their position in the response, without measuring human attention or brand presence. - [Content Signals](): Cloudflare framework for declaring content-use preferences for search, AI input, and training, separate from technical access controls. - [Machine-readable content](): Machine-readable content presents text and structure in a representation that a program can parse, without guaranteeing access, understanding, or citation. - [AI Copilot](): An AI copilot, in the context of agents, prepares proposals and submits actions for human decision. Assess permissions, review, and execution in the specific workflow. - [Core Web Vitals](): Core Web Vitals measure loading, response to interactions, and visual stability. Understand LCP, INP, and CLS, field data, and thresholds for SEO decisions. - [Custom corpus](): A custom corpus is a curated collection of searchable documents. Selection, updates, and permissions determine which evidence a response can retrieve. - [Training data cutoff](): Training data cutoff is the stated time reference for a model’s data. The date does not guarantee complete coverage of earlier facts. - [Effective knowledge cutoff](): Effective knowledge cutoff estimates the version of a source that is most temporally aligned with the model. It may differ from the stated cutoff and vary by resource. - [Crawl budget](): Crawl budget describes a site's crawling capacity and demand. Learn when it deserves attention and why it isn't a priority for every SEO project. - [Crawl-delay](): Crawl-delay is a robots.txt extension for requesting a slower crawl rate. Support depends on the operator; Google does not interpret this field. - [Cross-encoder](): A cross-encoder evaluates a query and passage together to score relevance. Learn how it is used in reranking, what to test, and its time and input limits. - [Training data](): Training data consists of examples used to adjust a model’s parameters. Their selection, origin, and composition influence the behavior learned. - [Structured data](): Structured data describes information in an explicit format for machines. Learn how it is used on pages, how it relates to Schema.org, and what to validate before publishing. - [Synthetic data](): Synthetic data are artificially generated examples for training, testing, or analysis. Learn how to assess their usefulness, origin, and limits of representativeness. - [DeepSeek](): DeepSeek offers chat and API access to its models. Identifying the service, interface, and version avoids comparing responses obtained under different conditions. - [Entity disambiguation](): Entity disambiguation distinguishes people, brands, or products with similar names using identifiers, attributes, and references to the correct identity. - [Knowledge staleness](): Knowledge staleness occurs when answers retain facts that are no longer valid. Checking dates and sources helps distinguish outdated information from fabrication. - [Knowledge distillation](): Knowledge distillation trains a student model to imitate a teacher. Learn how to assess cost, speed, and quality loss by task. - [Bot directory](): Bot directories organize identifiers and categories for automated services. Use their records alongside operator sources and observed traffic. - [Inline domain](): Inline domain identifies the domain of a reference inserted into the text of an AI response. How the link is displayed does not replace identifying the cited page. - [Embeddings](): Embeddings represent texts as vectors to compare relationships in meaning. Learn about their role in search and what to watch for when evaluating similar results. - [Prompt engineering](): Prompt engineering formulates and tests instructions for a task. Learn how to use success criteria and examples without mistaking improvement for a guarantee. - [Entity](): An entity is something identifiable, such as a brand, person, or product. See how names, context, and consistent data help distinguish your company from others with the same name. - [Retrieval poisoning](): Retrieval poisoning introduces content to distort answers when retrieved. Checking provenance and claims helps investigate the source of the deviation. - [Entity establishment](): Entity establishment organizes names, identifiers, and verifiable attributes to present a coherent identity across pages and public sources. - [Content extractability](): Content extractability is the ability to extract passages with their context from processed material, without confusing extraction with citation in AI. - [FCP: First Contentful Paint](): FCP measures when the first content appears on screen after navigation. Learn about the 1.8-second benchmark and how it differs from LCP and full page load. - [Citation faithfulness](): Citation faithfulness checks whether a source supports what an answer claims. Learn to check conditions, numbers, and limits before using the information. - [Fine-tuning](): Fine-tuning adapts a previously trained model to a task or domain. Learn when to test this technique and how to evaluate its results. - [AI answer source](): An AI answer source is material retrieved to formulate a response. Learn how to distinguish a search, a visible citation, and factual support. - [GPTBot](): GPTBot is OpenAI’s crawler for content that may contribute to model training. Learn how to distinguish this collection from access for ChatGPT search. - [Gemini](): Gemini names Google's assistant and a family of models. Evaluation should distinguish the app, the API, and the features enabled for each account. - [Gemini Notebook](): Gemini Notebook is Google's research tool based on sources selected by the user. Learn about its scope, the name change, and how to review responses. - [Google-Agent](): Google-Agent identifies visits by agents hosted by Google that browse and act at the user's request. Learn how to verify their origin and define permissions. - [Google-Extended](): Google-Extended controls the use of content for training and grounding Gemini systems. It is a robots.txt token separate from Search controls. - [Google-GeminiNotebook](): Google-GeminiNotebook identifies the fetcher for URLs provided as sources in Gemini Notebook. Understand user-requested access and the previous token. - [GoogleOther](): GoogleOther is Google's generic crawler for public content. Learn about its identification, permissions, and the limits of interpreting its visits. - [Googlebot](): Googlebot crawls pages for Google Search. Learn how to verify its identity and distinguish access, robots.txt blocking, and indexing control. - [Knowledge graph](): A knowledge graph represents entities and relationships using a defined structure and vocabulary. Learn how to check identity, connections, and data sources. - [Grounding](): Grounding connects an AI response to reference information. Learn how to check sources and claims, and why an answer with links still needs review. - [HTML: the web’s markup language](): HTML defines the structure and meaning of a page’s content. Learn about elements, links, and headings, and what to check beyond a site’s visual appearance. - [HTTP 402](): HTTP 402 is the Payment Required code, reserved in RFC 9110. Learn about its uses in automated billing and why the response does not confirm payment. - [Headings: HTML section titles](): Headings are section titles marked with h1 to h6 in HTML. Learn how to organize content hierarchy without confusing visual size, SEO titles, and structure. - [Multimodal AI](): Multimodal AI works with more than one type of data, such as text and images. See how to test whether an analysis really depends on visual information. - [INP: Interaction to Next Paint](): INP assesses the visual response to clicks, taps, and keyboard input during a visit. Learn how to investigate interaction delays and interpret the 200-millisecond benchmark. - [Indexing](): Indexing is the analysis and organization of pages in a search index. Learn why an accessible URL may be left out of the index and how to investigate. - [Inference](): Inference is the execution of a trained model to produce a prediction or response. Learn what to look for when evaluating generation, latency, and production use. - [AI HTML Inspector](): Review metadata, titles, declared indexing, authorship, dates, and JSON-LD in the supplied HTML. Extract text to check how the content is organized. - [Artificial intelligence](): Artificial intelligence brings together systems that perform tasks such as forecasting, language processing, and decision-making. Learn what to assess in marketing applications and understand their limitations. - [Artificial general intelligence](): AGI refers to artificial intelligence with broad capabilities across different tasks. Learn about the competing criteria and how to assess claims of generality. - [JSON-LD](): JSON-LD is a format for linked data used in page markup. See how it relates to Schema.org and how to check syntax and editorial consistency. - [Context window](): A context window is the token space available for a model’s inputs and response. Its limit affects the documents, history, and instructions used. - [LCP: Largest Contentful Paint](): LCP measures when the largest eligible content appears in the page’s visible area. Learn about loading, the 2.5-second benchmark, and field and lab analysis. - [LLM](): An LLM is a language model used to interpret and generate text. Learn the difference between a model and an assistant, and how it affects brand monitoring. - [LLM as a judge](): LLM-as-a-judge uses a language model to evaluate responses against defined criteria. Learn about biases and calibration with human review. - [Late chunking](): Late chunking creates embeddings for chunks after processing a larger context. The technique aims to preserve references between parts of the same document. - [Agent readability](): Agent readability describes the clarity of content and interfaces in the representations an agent uses to recognize information, relationships, and actions. - [Data cleaning](): Documented handling of errors, formats, and duplicates before analysis, while preserving original data, exclusion rules, and the basis for metrics. - [Author markup](): Author markup identifies authors in structured data, with names and references that match the page. Declaring a person does not prove their expertise. - [Attention mechanism](): The attention mechanism combines input representations with weights calculated by the model. Learn how to interpret its limitations when evaluating AI responses. - [Answer engine](): An answer engine provides a direct explanation in response to a question. Learn how to assess sources, alternatives, and differences between generative engines. - [Generative engine](): A generative engine combines information retrieval and answer generation. Learn about the role of sources and attribution, and the limits of observing third-party systems. - [Non-parametric memory](): Non-parametric memory is information kept outside a model’s parameters and accessed through retrieval, such as a document index queried in RAG. - [Parametric memory](): Parametric memory is knowledge represented in a model’s learned parameters. Updating a page does not directly modify these parameters. - [MistralAI-User](): MistralAI-User identifies page accesses requested by Vibe users. Learn its purpose, how to verify its origin, and what site controls apply. - [Mixture of Experts](): Mixture of Experts uses routing to activate parts of a network. Learn why total and active parameters help evaluate different models. - [Model Context Protocol](): MCP is an open communication protocol between AI applications and external services, which can expose data, prompts, and tools with defined capabilities and access. - [Frontier model](): A frontier model refers to capabilities near the state of the art at a given time. Learn why the label requires criteria, a date, and task evaluation. - [Foundation model](): A foundation model is a base trained on broad data and adaptable to various tasks. Learn to identify dependencies and evaluate the application that uses it. - [NLWeb](): NLWeb brings together open protocols and tools for querying websites in natural language, with structured responses and MCP integration in the reference implementation. - [AI browser](): An AI browser incorporates browsing assistance, such as search and tab analysis. Actions on websites depend on the product’s features, access, and permissions. - [Agentic browsing](): Agentic browsing occurs when an AI navigates and interacts with websites to complete a user’s task, subject to permissions and confirmation of results. - [Non-determinism in LLMs](): Non-determinism in LLMs is the variation in outputs under seemingly identical inputs. Learn how to compare responses and distinguish variation from factual errors. - [OAI-AdsBot](): OAI-AdsBot visits pages submitted as ads in ChatGPT to assess safety. OpenAI may also use their content to determine display relevance. - [OAI-SearchBot](): OAI-SearchBot crawls pages for ChatGPT search features. Learn about access controls and why allowing the bot does not guarantee a citation. - [Generative features opt-out](): A generative features opt-out lets you exclude content from AI experiences defined by the operator, with a scope distinct from data collection and training. - [Chunk optimization](): Chunk optimization adjusts segments used for information retrieval. Evaluate boundaries, context, and evidence for each task rather than adopting a universal size. - [Perplexity-User](): Perplexity-User retrieves pages in response to user actions. See the caveat about robots.txt and how to distinguish access from use in the response. - [PerplexityBot](): PerplexityBot is Perplexity’s search crawler. Learn its purpose, how to check permissions, and why access does not prove citation. - [AI Usage Preferences](): AI usage preferences express how content may be processed. Learn about the IETF’s work, its limits, and how it differs from blocking access. - [Prompt injection](): Prompt injection attempts to change a model’s behavior through inputs treated as instructions. Access limits and output checks reduce the impact. - [Indirect prompt injection](): Indirect prompt injection arrives through webpages, files, or other external sources. Retrieved content must remain separate from the application’s instructions and permissions. - [Agent readiness](): Agent readiness assesses whether a website or service enables defined tasks to be completed with access, understanding, controls, and verifiable evidence. - [Query fan-out](): Query fan-out expands a question into related searches. Learn its role in AI responses and how to analyze observed queries without treating them as measured demand. - [RAG](): RAG combines information retrieval and response generation. Learn why current sources can be used in AI searches and what limitations remain. - [RLHF](): RLHF uses human evaluations to guide model training. Learn about the role of preferences and the limitations of evaluating AI responses. - [Crawlability for AI](): Crawlability for AI checks whether an identified agent can reach a URL and receive its content, taking purpose, permissions, and the actual response into account. - [Crawler](): A crawler is a program that visits pages to collect content. Its purpose and the server’s responses guide the assessment of access for search or AI. - [Mixed-use crawler](): A mixed-use crawler serves more than one documented purpose. Learn how to review uses and controls without assuming that each request reveals its ultimate destination. - [Crawling](): Crawling is automated access to pages by a bot. Learn how to check for blocks and server responses, and distinguish access, indexing, and citation. - [Crawl-to-referral ratio](): The ratio of crawler requests to referral requests for a platform, calculated using the same data set, period and log criteria. - [Bot-to-human visit ratio](): The bot-to-human ratio compares visits classified within the same scope. Requests, sessions, and people are different units and require compatible datasets. - [Really Simple Licensing](): Really Simple Licensing expresses content licenses in XML. Learn about the scope of the RSL 1.0 specification and the difference between declaring and enforcing a license. - [Citation recall](): Citation recall measures the proportion of verifiable claims in a response that are fully supported by the cited sources. Learn how to define your baseline. - [Information retrieval](): Information retrieval selects documents or passages relevant to a query. Learn how to check the retrieved material before evaluating the answer. - [Passage retrieval](): Passage retrieval selects relevant excerpts for a question. Learn how to evaluate the context retrieved before judging an AI response. - [Referrer](): Referrer indicates a navigation source when available. Policies may omit or reduce this data; its absence does not prove a direct visit. - [Reindexing](): Reindexing is the reprocessing of a page in the index after changes. Compare the published version with the indexed data before considering the update complete. - [JavaScript rendering](): JavaScript rendering produces content from a page’s scripts. See what to check so search engines can read titles, text, and links. - [Server-side rendering](): Server-side rendering generates HTML before delivering it to the client. Checking it distinguishes received content, hydration, and subsequent JavaScript processing. - [Reranking](): Reranking reorders results retrieved for a query. Learn how to compare the lists, assess relevance, and recognize the limits of this search stage. - [AI response](): An AI response is the output a system presents for a prompt. Defining its boundaries makes it possible to measure mentions and citations without confusing responses with sessions. - [Entity salience](): Entity salience describes how central an entity is within a text or under a defined method. It is not equivalent to brand popularity, frequency, or recommendation. - [Schema FAQPage](): Schema FAQPage represents pages of frequently asked questions. The type remains in Schema.org, but FAQ rich results no longer appear in Google Search. - [Schema Organization](): Schema Organization describes an organization using Schema.org properties. Identity and contact details must be accurate; valid markup does not guarantee results. - [Schema Product](): Schema Product describes products or services with structured attributes and related offers. Accurate, current data matters more than filling in unsupported fields. - [Schema.org](): Schema.org is a vocabulary of types and properties for describing content. Learn how it relates to JSON-LD, structured data, and validation, without promises of citations. - [Citation hijacking](): Attributing content or data to another source in AI responses. Learn how to check authorship, references, and limitations before concluding where the misattribution originated. - [Adversarial text sequence](): An adversarial text sequence is inserted into a source to influence a model’s recommendation. Controlled research does not prove an effect across all search engines. - [MCP server](): An MCP server provides tools, resources, and prompts to compatible clients. Its usefulness depends on the functions it exposes and the permissions granted. - [Cosine similarity](): Cosine similarity compares vectors by direction. Learn about its role in semantic search and why a score does not prove relevance or truth. - [Sitemap](): A sitemap informs search engines about URLs and updates. Learn how to select pages, keep dates accurate, and check the file without treating it as a guarantee of indexing. - [Agent Skills](): Agent Skills bring together reusable instructions and resources for specific tasks. The Agent Skills format organizes this material into folders containing SKILL.md. - [Slopsquatting](): Slopsquatting exploits names invented by models to register malicious packages. A package’s presence in a repository does not prove that a dependency is trustworthy. - [HTTP Status](): HTTP status indicates the result of a request. Learn how 200 responses, redirects, and errors affect page access, and why technical success does not guarantee indexing. - [Response synthesis](): Response synthesis turns a question and passages into an explanation. Learn how to verify the support for each claim and preserve the sources’ caveats. - [TBT: Total Blocking Time](): TBT adds up the time by which long tasks exceed the threshold for blocking the browser’s main thread. Learn about lab diagnosis and how it differs from INP. - [TTFB: Time to First Byte](): TTFB measures the wait until the first byte of the response. Understand network, redirects, server, and cache without confusing this measure with content visible on screen. - [Temperature in LLMs](): Temperature in LLMs adjusts randomness during generation. Learn how to compare text variations and why the parameter does not guarantee factual accuracy. - [Token](): A token is a unit that represents part of the text for a language model. The count depends on the tokenizer and is not equivalent to the number of words. - [Discontinued crawler tokens](): Discontinued tokens are identifiers whose support has ended, according to a statement from the operator. Check the date, the replacement, and the limits of this evidence. - [Transformer](): Transformer is a neural network architecture based on attention mechanisms. Learn what it does and why it does not prove a tool’s quality. - [Universal Commerce Protocol](): UCP organizes commerce capabilities between platforms and sellers. Discovery, checkout, and payment authorization have distinct roles and checks. - [User agent](): A user agent identifies the client software; in HTTP, the header declares information about it. It helps analyze traffic but does not, by itself, authenticate the visitor. - [JSON-LD Validator](): Find syntax, context, entity type and date issues in structured data. Review the results and copy the formatted JSON. - [Variability in retrieved sources](): Variability in retrieved sources compares documents obtained across repeated runs of the same question, subject to limits on access to the logs. - [Response variability](): Variability is the difference between responses to equivalent questions. Learn why repetition, sample size, and collection conditions matter in AI monitoring. - [AI robots.txt checker](): Check which crawling rules apply to a page for search and training bots. Paste your file and see the decisive rule for each agent. - [Position bias](): Position bias describes how the location of information in the context can change how a model uses it. Learn about the test and the limits of this evidence. - [Web Bot Auth](): Web Bot Auth uses HTTP signatures to identify automated traffic. See the protocol status, key verification, and the limits of authentication. - [WebMCP](): WebMCP proposes structured tools on web pages for browser agents. Learn about its experimental status and how it differs from a remote MCP server. - [Wikidata](): Wikidata organizes entities, properties, and references in a collaborative, multilingual database. An item is not the same as a Wikipedia article and does not guarantee use by AI. - [llms-full.txt](): A supplementary file that brings together expanded documentation content for AI tools, distinct from the llms.txt index and with no guarantee of being read or affecting rankings. - [llms.txt](): llms.txt is a proposed file containing context and links for agents. Learn its purpose and why it does not replace robots.txt, a sitemap, or the fundamentals of SEO. - [max-snippet](): A directive that limits the search text snippet by character count, with effects documented by Google on content input to its AI features. - [meta-externalagent](): meta-externalagent identifies Meta’s crawler for training and direct indexing. Learn how to check specific rules without confusing it with other agents. - [meta-externalfetcher](): meta-externalfetcher fetches links requested by users and supports Meta’s agentic capabilities. Learn about the possible exception to robots.txt rules. - [meta-webindexer](): meta-webindexer crawls pages to improve Meta AI search. Learn about access controls and why visits do not prove citations in responses. - [noai](): A preference signal against AI uses, adopted in contexts such as Fab, whose meaning and effect depend on the platform and do not guarantee technical blocking. - [noarchive](): A directive historically associated with page caching, now ignored by Google and still documented by Bing with specific effects for Chat, Copilot, and training. - [noindex](): noindex instructs search engines not to index a page. Learn how to check the directive and why blocking crawling can prevent Google from reading it. - [nosnippet](): A preview-control directive recognized by Google and Bing, with distinct effects in each search engine on snippets and content use in AI features. - [robots.txt](): robots.txt provides access rules for crawlers. Understand its limitations, how it differs from noindex, and how to distinguish search bots from training bots. - [sameAs](): sameAs links an item described with Schema.org to pages that unambiguously indicate its identity, without making every external link a valid equivalence. - [Search index](): A search index is a searchable structure that organizes information about documents. Learn how it relates to crawling, retrieval, and presence in search results. --- [All pages in Markdown]()