--- language: en title: "Technical guide: Optimize your site and content for AI citations" description: "Implement clear content, structured data, robots.txt, and sitemaps with code examples. Learn what each setting does and which promises to avoid." canonical: "https://tropk.ai/en/blog/technical-guide-how-to-optimize-your-website-and-content-to-be-cited-by-ai" last_modified: "2026-10-12T16:05:29.412Z" --- # Technical guide: Optimize your site and content for AI citations Implement clear content, structured data, robots.txt, and sitemaps with code examples. Learn what each setting does and which promises to avoid. [Luiz Gardelin]() Published on 2026-04-13T16:51:46.000Z Categorias: How to Optimizing a website to be consulted and cited by AI systems requires useful content, technical access, and verifiable information. No markup guarantees a [citation](/en/resources/glossary/ai-citation): systems select sources based on the question, the search product, and the information available. This guide organizes the work into three layers: understandable content, consistent [structured data](/en/resources/glossary/structured-data), and access and discovery settings. The examples show what to implement and check, without turning research correlations into performance promises. The technical guidance was reviewed on **October 6, 2026**. Implementation should improve the reading experience and the quality of published information, while measurement checks actual results by platform. ## Key points - Organize content, sources, and authorship to answer real questions with sufficient context. - Structured data must match visible content; there is no required schema for Google's generative search. - Use OAI-SearchBot for ChatGPT Search access and decide on training separately. - Validate HTTP responses, CDN, robots.txt, and sitemaps before attributing a lack of citations to the content. - Measure mentions, citations, traffic, and conversions by platform; observational studies do not guarantee gains. ## Why SEO and AI visibility need to be monitored together [SEO](/en/blog/what-is-seo-search-engine-optimization-the-complete-guide) remains important for making pages discoverable, crawlable, and understandable. Presence in [AI responses](/en/resources/glossary/ai-response) adds other questions: Was the brand mentioned? Was a page cited? Did the response accurately represent the product? The [Google guide to AI features](https://developers.google.com/search/docs/appearance/ai-features) states that [AI Overviews](/en/resources/glossary/ai-overviews) and [AI Mode](/en/resources/glossary/ai-mode) use Search fundamentals. To appear as a supporting link, a page needs to be indexed and eligible to appear with a snippet. There is no additional set of technical requirements. That does not make all search engines the same. [ChatGPT](/en/resources/glossary/chatgpt) Search, [Perplexity](/en/resources/glossary/perplexity), and Google products have their own controls and behaviors. A URL being indexed or ranking well in one search engine does not prove it will be selected in every AI response. Use [SEO](/en/resources/glossary/seo) as the foundation and [GEO](/en/blog/what-is-geo-generative-engine-optimization-complete-2026-guide) as an area for monitoring and improving discovery in generative responses. Evaluate results using defined questions, languages, and platforms. ## Layer 1: content that answers questions and can be understood on its own ### Write direct answers with enough context A section should begin with the answer to the question in its heading. Then add conditions, an example, and a source when needed. The main benefit is for the reader who needs to make a decision; any benefit to AI [retrieval](/en/resources/glossary/information-retrieval) should be observed, not assumed. A self-contained answer identifies the subject without relying on “as we saw above.” For example, when explaining crawlers, say which agent performs which activity. Don’t just say “this bot” if the paragraph might be read outside the section. Example of a useful explanation: > [OAI-SearchBot](/en/resources/glossary/oai-searchbot) is OpenAI’s [crawler](/en/resources/glossary/crawler) for ChatGPT search features. Allowing this agent helps remove a [crawling](/en/resources/glossary/crawling) barrier, but does not guarantee that a page will be selected as a source. [GPTBot](/en/resources/glossary/gptbot) has a different purpose: collecting content that may contribute to training. A promise like “configure these bots and appear in AI systems” doesn’t explain the agents’ roles or the limits of the configuration. ### Organize headings, tables, and lists around the reader’s problem Use one H1 for the article and organize H2 and H3 [headings](/en/resources/glossary/headings-html-section-titles) to separate topics and steps. Tables work when criteria can be compared; lists help readers follow sequences or requirements. Don’t break text into dozens of subheadings just to try to influence a search engine. Each heading should preview the section’s content. If the heading promises a procedure, include actionable steps and conditions. If it promises a comparison, use the same criteria for all alternatives. The [Google’s official AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) recommends prioritizing useful [content and SEO](/en/solutions/content-and-seo-teams), and cautions against treating “[chunking](/en/resources/glossary/chunking)” or special files as mandatory shortcuts. Clear editorial structure is a readability choice, not a secret ranking protocol. ### What observational research can help us understand The [Semrush content study](https://www.semrush.com/blog/content-optimization-ai-search-study/), published in January 2026, compared characteristics of cited pages with similar pages that appeared on Google. It found differences associated with clarity, experience signals, and organization. | Characteristic measured by Semrush | Difference reported in the study | | --- | --- | | Clarity and summarization | +32,83% | | E-E-A-T signals | +30,64% | | Question-and-answer format | +25,45% | | Section structure | +22,91% | | Structured data elements | +21,60% | These figures describe associations in the analyzed dataset. They do not mean that adding an FAQ increases citations for any article by 25.45%. Variables such as topic, quality, authorship, and sample selection can affect the results. The study also discusses a negative association for its non-promotional tone variable and the possibility of confounding with the quality of professional writing. Therefore, it is incorrect to turn this result into a universal 26.19% penalty for promotional writing. The [GEO academic paper, presented at KDD 2024](https://arxiv.org/html/2311.09735v3), investigated reformulation strategies in a benchmark and reported gains of up to 40% under certain conditions. The result is relevant to research, but is not a forecast of traffic or citations in current products. ### Keep sources, authorship, and updates accurate Use data you can verify, including the period, sample, and metric definition. A percentage without a source or an experimental result presented as a general rule reduces information quality. Identify the author and editorial contact using real information. Distinguish the publication date from the update date. A substantial revision may justify a new update date; changing the date without revising the content doesn’t make the information more current. In fast-changing fields, check primary documentation before repeating a recommendation. For this guide, crawler roles, the use of [llms.txt](/en/resources/glossary/llms-txt), and the limitations of structured data were reviewed against official sources. ## Layer 2: structured data that matches the content [Schema markup](/en/blog/schema-markup-for-ai-visibility-the-7-types-that-matter-and-how-to-implement-each) lets you declare information such as content type, author, organization, navigation, and frequently asked questions. [JSON-LD](/en/resources/glossary/json-ld) is a common way to publish this information. Google states that no special schema is required for generative search. Use structured data as part of your website’s organization and SEO, keeping the markup consistent with the visible text. Don’t invent reviews, credentials, or capabilities to fill in fields. ### What a study of 5,499 sites shows—and what it doesn’t The [study originally published by Spotlight](https://www.get-spotlight.com/articles/empirical-study-which-schema-markup-types-appear-in-ai-cited-websites/) observed 5,499 sites cited by AI models. In the blog post category, it found the following types: | Schema type | Occurrences reported in blog posts | What it describes | | --- | --- | --- | | Person | 364 | Information about people, such as authors | | SiteNavigationElement | 274 | Navigation elements | | ListItem | 152 | Structured list items | | ImageObject | 134 | Information about images | | Organization | 94 | Organization information | | Blog | 92 | Blog structure | | Article | 49 | Article type | | BreadcrumbList | 49 | Navigation path | The table shows how frequently these types occurred on the observed sites. It does not measure a causal effect of implementation or prove that Person generates seven times more citations than Article. A site can have both, and the source of the markup may vary by CMS. A useful application is to check that the author, publisher, article type, and navigation are described correctly. Priorities should reflect the actual content and the technical issue identified. ### Article, Person, and Organization: a consistent example An article page can connect the author and publisher using identifiers. This example is illustrative; names, URLs, and dates need to reflect the site’s public information. ```json { "@context": "https://schema.org", "@graph": [ { "@type": "Organization", "@id": "https://example.com/#organization", "name": "Exemplo de Empresa", "url": "https://example.com/" }, { "@type": "Person", "@id": "https://example.com/autores/ana/#person", "name": "Ana Exemplo", "url": "https://example.com/autores/ana/" }, { "@type": "Article", "@id": "https://example.com/blog/guia/#article", "headline": "Guia de configuração do produto", "datePublished": "2026-04-13", "dateModified": "2026-10-06", "author": { "@id": "https://example.com/autores/ana/#person" }, "publisher": { "@id": "https://example.com/#organization" }, "mainEntityOfPage": "https://example.com/blog/guia/" } ] } ``` Add `sameAs` only for real, verifiable profiles belonging to the correct [entity](/en/resources/glossary/entity). Don’t use this field to associate an author with publications or credentials they don’t have. ### FAQPage: visible questions and answers, with no promise of a rich result When an article includes an FAQ with editorial questions and answers, `FAQPage` describes these pairs. Each question and answer should make sense on its own and match the displayed content. An answer hidden in an accessible accordion can still be available to readers; content that is never displayed should not be added to the schema just for that purpose. ```json { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "O que é robots.txt e qual é sua função em um site?", "acceptedAnswer": { "@type": "Answer", "text": "robots.txt é um arquivo que declara preferências de rastreamento para robôs compatíveis. Ele não protege informações privadas e não garante remoção de páginas dos resultados de busca." } } ] } ``` The JSON needs to be published in an appropriate element. The [HTML](/en/resources/glossary/html-the-web-s-markup-language) structure is: ```html ``` This HTML example shows only the wrapper; replace the object with complete markup for the visible pairs. Don’t publish an empty FAQ or duplicate incompatible markup generated by plugins. **FAQPage does not guarantee rich results on Google.** The [official update log](https://developers.google.com/search/updates) states that the FAQ feature in search results stopped appearing on May 7, 2026; the specific documentation was removed in June. The type remains in the [Schema.org](https://schema.org/FAQPage) vocabulary, but should not be sold as a promise of a rich result. ### HowTo for procedures `HowTo` can describe a real procedure with steps. Don’t use tutorial schema for an opinion article or add steps that aren’t in the text. ```json { "@context": "https://schema.org", "@type": "HowTo", "name": "Como revisar o acesso público de uma página", "step": [ { "@type": "HowToStep", "name": "Conferir a resposta HTTP", "text": "Abra a URL pública e verifique se ela entrega o conteúdo sem exigir login." }, { "@type": "HowToStep", "name": "Conferir regras de rastreamento", "text": "Revise a correspondência entre o agente, a URL e as regras do robots.txt." }, { "@type": "HowToStep", "name": "Inspecionar logs", "text": "Verifique bloqueios e erros HTTP no servidor e na CDN." } ] } ``` The vocabulary helps describe the page; support for special presentations depends on each consumer. Adding HowTo does not guarantee a citation. ### Speakable is not a universal marker of priority for AI The [Google documentation on speakable](https://developers.google.com/search/docs/appearance/structured-data/speakable) covers passages suitable for voice playback with TTS, in a beta news feature with specific language and market requirements. This documentation does not support claiming that marking up H2s, tables, and FAQs with speakable makes generative search systems prioritize those sections. Evaluate the feature if there is a real, compatible audio use case; do not treat it as a requirement for citations in Portuguese. ### How to validate markup 1. Check that every JSON example is syntactically valid. 2. Ensure entities, URLs, authors, dates, and answers match the visible page. 3. Inspect the served HTML and DOM to confirm that the markup is published. 4. Use validators relevant to the data type and consumer, distinguishing [Schema.org](/en/resources/glossary/schema-org) validation from eligibility for a Google feature. 5. Review the page after content changes to avoid conflicting answers and dates. ## Layer 3: access, crawling, and discovery ### robots.txt: distinguish search, training, and user actions A technical block can prevent a system from retrieving a page, but the consequences depend on the agent. The [OpenAI documentation](https://developers.openai.com/api/docs/bots) describes OAI-SearchBot for search, GPTBot for collection that may be used for training, and [ChatGPT-User](/en/resources/glossary/chatgpt-user) for certain user-requested actions. The [Perplexity](https://docs.perplexity.ai/guides/bots) also distinguishes [PerplexityBot](/en/resources/glossary/perplexitybot) from [Perplexity-User](/en/resources/glossary/perplexity-user). [Anthropic](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) distinguishes [ClaudeBot](/en/resources/glossary/claudebot), [Claude-SearchBot](/en/resources/glossary/claude-searchbot), and [Claude-User](/en/resources/glossary/claude-user). | Agent or control | Primary documented function | Configuration decision | | --- | --- | --- | | OAI-SearchBot | ChatGPT Search | Allow when the policy includes search crawling | | GPTBot | Collection for possible OpenAI training | Decide separately from search | | PerplexityBot | Perplexity search | Allow when the policy includes this search engine | | ClaudeBot | Collection for possible Anthropic training | Decide separately from Claude-SearchBot | | Claude-SearchBot | Crawling for Claude Search | Allow when the policy includes Claude Search | | Googlebot | Google Search and its features | Check crawling and indexing eligibility | | Google-Extended | Training and grounding in certain Gemini/Vertex products | Do not confuse this with Google Search controls | This model restricts collection paths for training while allowing public crawling through the generic group. Adapt it to the site's policy and existing rules: ```text # Rastreamento público permitido para agentes sem grupo específico. User-agent: * Allow: / # Restrição de coleta OpenAI para os caminhos indicados. User-agent: GPTBot Disallow: /conteudo-premium/ Disallow: /relatorios-restritos/ # Restrição de coleta Anthropic para o caminho indicado. User-agent: ClaudeBot Disallow: /conteudo-premium/ Sitemap: https://example.com/sitemap.xml ``` `ChatGPT-User` does not determine inclusion in search, and OpenAI says that [robots.txt](/en/resources/glossary/robots-txt) may not apply to its user-initiated actions. Perplexity documents a similar distinction for Perplexity-User. Therefore, do not use these agents as substitutes for search crawler controls. Premium content and private reports must remain protected by authentication. The example documents a crawling preference; it does not establish confidentiality. See the [complete robots.txt guide for AI](/en/blog/robots-txt-for-ai-which-bots-to-allow-and-how-to-configure-it) for specific rules, templates, and infrastructure checks. ### CDN, firewall, and HTML must deliver useful information A site may allow a bot in robots.txt and block it at another layer. Check for JavaScript challenges, CAPTCHAs, 403 or 429 responses, redirects, and authentication. Verify agents' origins using official methods before creating exceptions. The page must provide its main content in an accessible form. Do not hide all information in an image or an interaction the consumer cannot perform. Use readable text, real links, and clear procedures. Correct structured data cannot fix an empty page, a server error, or a URL that redirects to login. Prioritize failures that prevent access before adding optional conventions. ### llms.txt: optional index for compatible consumers [llms.txt](/en/blog/llms-txt-what-it-is-how-to-implement-it-and-what-it-changes-about-ai-visibility) provides context and links in Markdown. The current proposal requires an H1 with the site name and recommends a summary and link lists. It can be published at the root or in a subdirectory. ```markdown # Exemplo de Empresa > Documentação pública e informações oficiais sobre o produto. ## Documentação - [Primeiros passos](https://example.com/docs/quickstart.md): Configuração inicial. - [Referência](https://example.com/docs/reference.md): Parâmetros e limites. ## Informações institucionais - [Sobre a empresa](https://example.com/sobre): Identidade e canais oficiais. ``` Do not assume every AI reads this file before accessing the site. Google says Search and its generative features do not use it as a special file. Its use should have a defined consumer and task, such as helping an agent navigate documentation. ### XML sitemap and update dates A [sitemap](/en/resources/glossary/sitemap) helps provide URLs for discovery. Use [canonical](/en/resources/glossary/canonical), accessible URLs with dates that reflect real changes. [Google advises that lastmod be consistent and verifiably accurate](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap). ```xml https://example.com/blog/guia/ 2026-10-06 ``` The date is illustrative. Do not update lastmod in bulk unless the pages have changed. Google ignores the values `priority` and `changefreq`, so do not present these fields as a priority control or a guarantee of citation. Submitting sitemaps in search engine tools can help with discovery and diagnostics. It does not guarantee [indexing](/en/resources/glossary/indexing), citation, or use of a URL by a specific AI platform. ## Differences between platforms and fan-out AI Overviews and AI Mode may use different models and techniques. Google documents that both can perform [**query fan-out**](/en/resources/glossary/query-fan-out), issuing searches related to subtopics to build an answer. This expands potential sources, but does not mean a brand should publish pages for every variation of a question. ChatGPT Search and Perplexity have their own retrieval processes and access rules. Monitor each product by its correct name and note when search is enabled. An answer without search should not automatically be compared with one based on web sources. Editorial strategy should cover real problems, useful comparisons, and original information. Creating dozens of repetitive pages for possible fan-outs can produce low-value content; Google warns against scaled content intended to manipulate results. ## Where to start: prioritize based on observed issues | Timing | Task | Evidence of completion | | --- | --- | --- | | First | Fix HTTP errors, unintended login prompts, blocks, and broken links | Public page is accessible and crawling is allowed according to policy | | Next | Review the introduction, headings, examples, authorship, dates, and sources | Complete, understandable, and verifiable content | | Then | Align Article, entities, navigation, and FAQPage | Valid JSON consistent with the visible page | | As needed | Review the sitemap and publish additional formats for agents | Correct URLs and identified consumer | | Ongoing | Compare answers and sources across platforms | Record of questions, dates, mentions, citations, and changes | Timelines depend on site size, crawl frequency, and the changes made. There is no universal four-to-eight-week window for schema to generate citations. Establish a baseline, document what changed, and wait for systems to update before interpreting the results. For measurement, use the same set of questions, language, platforms, and classification criteria. Separate brand mentions, linked citations, traffic, and conversions. Improvement in one measure does not imply improvement in the others. See the [visibility score guide](/en/blog/what-is-visibility-score-in-ai-responses) and the [share of voice guide](/en/blog/what-is-share-of-voice-in-ai-responses) to define comparable measures. Technical setup removes obstacles and describes the content more clearly; results need to be tracked in real answers. ## Sources for further reading - [Google: AI features and websites]() - [Google: AI optimization guide]() - [Semrush: content and citations study]() - [GEO: Generative Engine Optimization, KDD 2024]() - [Spotlight/rocketblue: schema types on cited websites]() - [Schema.org: FAQPage]() - [Google: official documentation update log]() - [Google: Speakable]() - [Google: create and submit a sitemap]() - [OpenAI: crawlers]() - [Perplexity: crawlers]() - [Anthropic: crawler functions and controls]() - [llms.txt proposal]() ## Frequently asked questions ### What technical changes help a website get crawled by AI-powered search systems? Prioritize accessible public pages, correct HTTP responses, crawling permitted under the relevant policy, useful internal links, and readable main content. Review sources, authorship, and freshness; keep structured data consistent and maintain a sitemap with correct URLs. These measures remove barriers and improve how content is described, but do not guarantee selection as a source. ### Is schema markup required for a site to appear in AI Overviews or AI Mode? No. Google states that there is no required special schema or additional technical requirements for its generative features. A page must be indexed and eligible to appear in Google Search with a snippet. Structured data can be part of a site's SEO, provided it accurately represents the visible content. ### How can you publish FAQPage schema without relying on guarantees of enhanced results? Publish an editorial FAQ with self-contained questions and answers, and represent the same pairs in JSON-LD using FAQPage, Question, and Answer. Validate the JSON and check that it matches the page. The type remains in Schema.org, but Google stopped showing the FAQ feature in results in May 2026; the markup does not guarantee rich results or AI citations. ### Which crawler should be allowed for ChatGPT search crawling? OAI-SearchBot is the crawler OpenAI documents for ChatGPT search features. Its permission is independent of GPTBot, which is associated with potential training. ChatGPT-User handles certain user-initiated actions and does not determine inclusion in search. You also need to check your CDN and firewall to ensure access allowed by robots.txt works. ### Are llms.txt and Speakable necessary for an article to be cited by AI? There is no universal requirement for llms.txt or Speakable for AI citations. llms.txt is a Markdown index for compatible consumers, and Google Search does not use it as a special file. Speakable identifies excerpts for playback by voice in specific uses; Google's documentation does not present it as a general priority signal for generative search. ### How can you check whether content and access optimizations have improved AI visibility? Record a baseline using the same questions, platforms, language, and evaluation criteria; document the changes and compare new responses and sources after the systems are updated. Measure mentions, linked citations, traffic, and conversions separately. There is no universal timeframe or guaranteed percentage gain from changing schema, robots.txt, or llms.txt. ## About the author [Luiz Gardelin]() Luiz Gardelin is the founder of Tropk.ai and works in GEO/AEO, product vision, strategy, and growth, helping brands understand and improve their presence in AI responses. --- [All pages in Markdown]()