How AI Search Engines Choose Which Sources to Cite

Anyone who has to do with getting featured for organic traffic, would like to know how to get there. And, to do that, they need to know how pages are cited by these AI tools. The question of how AI search engines choose sources has a short documented answer and a long undocumented one. The platforms publish eligibility rules and crawler controls. They do not publish their ranking logic.

This guide helps by providing what is documented and what is only observed.

The basic pattern: search, retrieve, write

It is inevitable for the Answer engines to make a search. Google says AI Overviews and AI Mode may issue several related searches across subtopics and data sources, a technique it calls query fan-out, and that further supporting pages can be identified while the response is being generated. The links shown can therefore be wider and more varied than in classic results.

Anthropic's documentation for the Claude web search tool describes the same sequence in API terms. Claude decides whether a request needs current information, the API runs the searches (which can repeat within one request), and the final answer carries cited sources. Each citation records the page URL, its title and up to 150 characters of the cited text.

For a site owner this has one practical consequence: a page can be retrieved for a sub-question that the user never typed. A page that answers a specific question clearly can be more useful to these systems than a broad page that mentions the topic in passing. That is an inference from how the systems are described, and no platform publishes a rule that says so.

What each platform documents

Google AI Overviews and AI Mode: a page can appear as a supporting link if it is indexed and eligible to appear in Search with a snippet, and it meets the usual technical requirements. As per Google, you are good if you have done all the tagging right for SEO purposes and bot crawlability. The controls are robots.txt for Googlebot, the nosnippet, data-nosnippet and max-snippet directives, and noindex. Google-Extended is a separate control that applies to some other Google AI uses.

ChatGPT search: OpenAI uses OAI-SearchBot to surface websites in its search features, GPTBot for training models and ChatGPT-User to handle user actions. Sites that opt out will not appear in ChatGPT search answers, although they can still show up as navigational links. Disallowing GPTBot does not affect search appearance. ChatGPT-User  is not used to decide search appearance, and may not follow robots.txt rules because a user triggered the visit.

Perplexity: PerplexityBot is the crawler that helps surface and link sites in Perplexity results, and Perplexity says it is not used to crawl content for foundation models. Perplexity recommends allowing it in robots.txt and allowing its published IP ranges. Perplexity-User fetches a page when a person asks a question, and because the user triggers that fetch it generally ignores robots.txt.

Claude web search (API): the developer who builds the application can restrict results with allowed or blocked domain lists, and an organisation administrator can restrict domains in the Claude Console. So in an application built on this tool, the pool of possible sources can be narrowed by whoever configured it, before any ranking happens.

It is important to note that each platform treats its crawlers as independent of the other. A site can allow a search crawler and disallow a training crawler, and OpenAI and Perplexity both say a robots.txt change can take around 24 hours to take effect.

What is not documented

None of the documentation I read explains why one eligible page is cited and another is not. There is no published list of citation factors, no weighting and no guarantee of placement. That gap has been filled by a large amount of third-party commentary, and the pieces I reviewed disagree with one another about which search index feeds which assistant. Treat claims of that kind as unverified unless the platform itself states them.

There is some academic evidence. A paper on GEO by Aggarwal and colleagues, accepted at KDD 2024, proposes a framework for measuring and improving a page's visibility in generative engine responses. The authors report improvements of up to 40 per cent in their tests and note that effectiveness varies across domains. It is a controlled experiment on a research benchmark, so it shows that visibility can be influenced. It does not show that any particular tactic will work on a live product today.

Practical checks you can make this week

Confirm that your key pages are indexed and snippet-eligible in Google. Use the URL Inspection tool to check the HTML Googlebot received, and look for accidental noindex, nosnippet or max-snippet settings.

Read your robots.txt line by line and look for each crawler by name. A broad Disallow rule can block OAI-SearchBot and PerplexityBot without anyone intending it. Decide deliberately whether to allow search crawlers and, separately, training crawlers.

Ask whoever manages your CDN or firewall whether bot protection blocks the crawlers above. OpenAI and Perplexity both recommend allowing their published IP ranges.

Keep the important content in text, as Google advises, and make sure structured data matches what is visible on the page.

Check that each important page answers its core question near the top, in plain language, and includes information a reader could not get from a competitor's page, such as your own data, process or experience. This is a recommendation based on how retrieval is described, not a documented ranking factor.

Allow time before judging a change. Google says a recrawl can take anywhere from days to months, and robots.txt changes at OpenAI and Perplexity can take about a day.

An illustrative audit of one page

The walk-through below is a hypothetical example to show the order of the checks. It does not describe a real client or a measured result. Take a service page on a regional accountancy site that never appears as a source when you ask an assistant about small business tax deadlines.

Start with access. Search Console shows the page as indexed, so Google eligibility is fine. The robots.txt file, however, contains a rule that disallows every crawler except Googlebot, which means OAI-SearchBot and PerplexityBot are shut out. Changing that is a decision for the site owner, and it takes about a day to apply.

Next look at the content. The page opens with three paragraphs about the firm's history, and the dates and the rules that a reader came for sit in a PDF. Moving a short, dated summary into the page text, with the question stated in a heading, gives a retrieval system a clear passage to quote. Then run the monthly log described below for a few months before drawing any conclusion, because a single answer proves nothing.

Measuring what happens

Google states that traffic from AI Overviews and AI Mode is included in overall Search traffic and appears in the Search Console Performance report under the Web search type. Its documentation suggests also tracking conversions and time on site in a tool such as Google Analytics.

For other assistants there is no equivalent report in the documentation I read. A practical method is a monthly log. Pick 20 questions that your customers ask, run each in the assistants you care about, and record which sources are cited and whether your pages appear. Answers vary by person, location and time, so the log only means something as a trend over several months.

Limits of this guide

The details were checked against documentation on 28 October 2026. Platform behaviour and crawler names change, so check the current pages before acting.

No step here guarantees a citation, and no platform publishes the criteria that would let anyone promise one.

Bing and Copilot are not covered, and the Claude section describes the developer API tool and says nothing about any particular consumer product.

Where this fits in a wider AI workflow

Understanding how answer engines retrieve pages is part of planning content that people and machines can both use. Zylous Digital applies the same approach across SEO, paid media and analytics, and you can read more on the AI-first approach page.

Frequently asked questions

Do I need special markup to appear in Google's AI Overviews?

According to Google, no. A page needs to be indexed and eligible to show with a snippet, and no special schema, files or optimisation are required. Meeting those conditions still does not guarantee it will be shown.

Does blocking GPTBot remove my site from ChatGPT search?

OpenAI says it does not. GPTBot relates to training data, while OAI-SearchBot controls whether a site is surfaced in ChatGPT search. The two settings are independent.

Can anyone guarantee that an AI assistant will cite my page?

No. The platforms do not publish their selection criteria, and the answers vary between users and over time. What you can control is access, indexing, content quality and measurement.