Kief Studio

How do ChatGPT and other answer engines choose which sources to cite?

Automate

Answer engines cite pages their own search systems can fetch and index, then pick passages that answer the question directly. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers. Perplexity surfaces sites through PerplexityBot. Google's AI features draw on pages already indexed and eligible for a snippet. Among the pages an engine can reach, clear, sourced, self-contained answers to the question being asked are the ones it chooses.

Blank document cards with one structured card lifted on a gold line, how answer engines choose sources to cite, Kief Studio, Shrewsbury Massachusetts.
Answer engines tend to quote pages that are clear, structured, and easy to verify.

Step one: the engine's crawler has to reach the page

Each engine publishes the crawlers it uses, and they are not interchangeable. OpenAI uses OAI-SearchBot to surface websites in ChatGPT search, and says sites that opt out of it will not be shown in ChatGPT search answers. GPTBot is a separate crawler for model training. OpenAI says each setting is independent, so a site can allow search and block training.

Perplexity works the same way. PerplexityBot surfaces and links websites in Perplexity search results and is not used to crawl content for foundation models. Perplexity-User fetches a page when a person asks about it, and Perplexity says that fetcher generally ignores robots.txt because a user requested it. The practical check is your robots.txt and your firewall. Blocking the search crawler by accident removes you from that engine's answers.

Step two: the page has to be in the index behind the answer

Answer engines build answers from a search index, not from the live web at the moment of the question. Google is explicit about its own AI features. To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet. Google says there are no additional requirements and no special optimizations needed beyond that.

That makes the dull work the important work. A page that is not indexed cannot be cited, however good it is. Check the search consoles for pages that were crawled but not indexed, fix what they report, and submit new and changed URLs promptly. A page published today and indexed next month is a page that missed a month of questions.

Step three: one question becomes several searches

An answer engine rarely runs a single search. Google describes a technique called query fan-out, issuing multiple related searches across subtopics and data sources to build one response. A question like "who should host my website" may turn into searches about ownership, security, cost of switching, and local providers, each of which pulls its own candidate pages.

That changes what a good page looks like. A page titled with the real question, and organized so each subtopic has its own heading and its own clear answer, matches more of those related searches. A page built around a slogan matches none of them. Write the headings a careful buyer would ask in sequence, and answer each one under its own heading.

Step four: the passage has to stand on its own

Engines quote passages, not whole pages. A passage gets used when it answers the question without needing the paragraphs around it. That means one claim per sentence, with the subject named instead of "it" or "this", and the fact stated plainly. A payoff buried at the end of a long story is hard for any reader to lift, human or machine.

Sources matter for the same reason. Google's guidance on helpful content asks whether a page builds trust through clear sourcing and evidence of the expertise involved. A passage that links to the standard, the statute, or the official documentation it relies on is a safer thing to quote than one that asks to be taken on faith. Write the sentence you would want quoted, and put the source next to it.

What does not decide it

There is no markup that guarantees a citation, and no file that unlocks one. Structured data helps an engine understand a page, but Google says it should match the visible text, and it does not stand in for an answer the page does not give. Proposed formats like llms.txt can describe a site, but none of the engines whose documentation we checked names them as a requirement.

Volume does not decide it either. A stack of short posts reacting to the news is less useful to an engine than one page that answers a lasting question properly. What decides it is the sequence on this page: reachable by the crawler, present in the index, matched by one of the searches, and quotable when the engine gets there.

How we use this

We work through the four steps in order for each client. We check robots.txt and the firewall against the crawlers each engine publishes, confirm the important pages are indexed, and map the questions buyers ask to the pages that should answer them. Then we rewrite or write those pages so the answer comes first and every claim has a source.

After that we measure. We put the same questions to the engines every month, record who they cite, and fix the gaps the record shows. We do not promise a citation, because no one can. We make the pages the engines are looking for, and we show you the record.

Questions

Why isn't my site cited by ChatGPT?

Check access first. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers. Then check that the page answers the question directly, early, and with sources an engine can trust.

Does blocking GPTBot keep my site out of ChatGPT search?

No. OpenAI says GPTBot is for training and OAI-SearchBot is for search, and each setting is independent. A site can block training and still allow its pages to appear in ChatGPT search.

Do I need special markup to be cited in Google AI Overviews?

No. Google says a page must be indexed and eligible for a snippet, with no additional requirements. Structured data is still useful when it matches the visible text, and eligibility comes from being indexed with a snippet.

Does Perplexity respect robots.txt?

PerplexityBot, the crawler that surfaces sites in search results, is the one robots.txt governs. Perplexity says Perplexity-User, which fetches pages a person asks about, generally ignores robots.txt rules.

Sources

Also read

Keep reading

More in Automate