Retrieval-Augmented Generation

What retrieval-augmented generation is

Retrieval-augmented generation (RAG) is the technique of answering a question by first retrieving relevant text from an external source, then handing that text to a language model as part of its prompt, so the model writes its answer from documents rather than only from what it absorbed during training. Generation, here, means the model composing prose token by token; retrieval means fetching candidate text before it starts. The name and the method come from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, published at NeurIPS 2020.

In a search practitioner's terms, RAG bolts a retrieval step onto the front of a text generator, so the generator argues from evidence handed to it a second ago instead of from a compressed memory of the web as it stood at training time. Every AI answer product that shows citations is doing some version of this, whether or not it uses the word.

The single most useful consequence: your page is not “in” the model. It is fetched, at query time, by a retrieval system — and on the major web products that retrieval system is a conventional search index. Almost every practical question about AI search visibility resolves once that sentence is taken literally, and almost every unfalsifiable claim in the generative engine optimization (GEO) market survives by ignoring it.

The pipeline, from question to citation

The minimal shape is the same everywhere, and each stage is a filter.

The system first decides whether to retrieve at all; not every question triggers a search, and the 2026 Martinez survey reports 57.8% of repeated ChatGPT queries skipping web search entirely. That is the first and least-discussed filter, and it means some questions never reach the web. Next the question becomes one or more queries — in enterprise RAG an embedding of the user's text, in the web products a literal search query, often several. Candidates are retrieved: a vector index of chunked documents in the enterprise case, the operator's own search index in the consumer case. A reranking pass reorders them and only a handful survive. Those survivors are concatenated into the prompt under a token budget, in some order. The model generates. Citations are attached to spans of the finished answer.

Three properties of that sequence matter more than any tactic. Only a small number of documents reach the generator — the founding GEO experiment fixed its context at five, which is a useful order of magnitude. Position within that context is not neutral, because models attend unevenly across long prompts. And the whole chain is stochastic: query generation, candidate selection and generation all involve sampling, so two identical prompts are two draws from a distribution rather than two readings of a rank.

Chunking, and why page-level thinking breaks

Documents are split into passages before retrieval — by heading, by paragraph, or by fixed token windows, usually with overlap. The retrievable unit is the chunk, not the URL. A page that answers a question across four scattered sections may contain no chunk that answers it; a page carrying one clean, self-contained passage may.

This is the honest mechanical reason clear sectioning matters, and it is worth separating from the claim it is usually bundled with. It is an argument about retrievability. It is not an argument about register, tone, or writing in a voice aimed at machines. A well-sectioned page with a self-contained answer is more likely to produce a matching chunk; a page rewritten to sound like an assistant is not, and the controlled tests that have looked found the second kind of rewrite ineffective or harmful.

Two related misconceptions follow from page-level thinking. The first is that longer and more comprehensive pages retrieve better. The mechanism argues both ways — a long page is many candidate passages, which raises the chance one matches and also dilutes each — and no controlled study isolating page length against citation rate in a live engine was found for this page. The second is that semantic retrieval has made keywords irrelevant. Production retrieval is normally hybrid, combining dense vectors for semantic similarity with classic sparse lexical matching, precisely because dense retrieval alone misses exact terms, names, model numbers and rare entities. The claim that lexical matching is obsolete has no source and contradicts the engineering.

The vector-database story is wrong for consumer AI search

Most commercial writing on AI search implies that AI systems maintain a vector database of the web and that the job is to make your content sit well inside it. For the major consumer products, the documentation says otherwise.

Google's grounding tool issues Google searches. Microsoft's grounding tool issues Bing searches. OpenAI states that ChatGPT search “leverages third-party search providers, as well as content provided directly by our partners,” without naming the providers. Perplexity operates its own crawler and index. Google's own explainer for AI Overviews describes a customized Gemini model working in tandem with existing Search systems — the quality and ranking systems and the Knowledge Graph — to identify relevant, high-quality results from the index to corroborate the information presented in the overview.

The vector-store picture is a reasonably accurate description of an enterprise RAG system built over a private document set. It is not an accurate description of how a general AI answer engine finds your page. Understanding that difference is worth more than any tactic on any GEO checklist, because it explains why classical retrieval work keeps outperforming content rewriting in every controlled test that has been run. C-SEO Bench, testing ten content rewrites across two tasks and six domains, found that making the target document first in the model's context produced far greater citation gains than any of them, with three of 54 cases reaching statistical significance.

What the operators document about grounding

The pipeline is not a matter of inference. Google's Grounding with Google Search documentation sets it out in five steps: the application sends a prompt with the search tool enabled; the model analyzes the prompt and decides whether a Google Search would improve the answer; if needed it “automatically generates one or multiple search queries and executes them”; it processes the results, synthesizes them and formulates a response; and the API returns a response grounded in those results. The response object carries the search queries the model actually executed, and inline citation annotations linking text segments to sources.

Microsoft documents the same shape for Grounding with Bing Search: the model processes the user query, performs a Bing search query that is visible to developers, receives relevant results and generates the final output. One detail there is worth dwelling on — “Developers and end users don't have access to raw content returned from Grounding with Bing Search.” Even the developer never sees what the model read. Displaying both the website citations and the Bing query URL is a terms-of-use requirement.

OpenAI describes ChatGPT search, launched 31 October 2024, as “a fine-tuned version of GPT-4o, post-trained using novel synthetic data generation techniques,” drawing on third-party search providers and partner content.

Where the documentation stops is uniform across all three. No operator publishes chunk size or chunking strategy, how many documents enter the context window, the reranking model or its features, how citations are matched to spans, or which sources are selected and why. Google's grounding documentation explicitly stops short of source-selection criteria. Any source stating a specific chunk size, document count or embedding model for AI Overviews, ChatGPT search, Copilot or Perplexity is inferring or inventing, and that silence is the reason so much advice in this field is unfalsifiable.

What RAG does not fix

Retrieval removes one class of error and introduces others. It addresses stale or absent knowledge inside the model; it adds retrieval of the wrong passage, correct retrieval followed by incorrect synthesis, and citations that do not support the sentence they are attached to.

The scale of the last one is measured. Auditing Bing Chat, NeevaAI, perplexity.ai and YouChat, Liu, Zhang and Liang reported that “a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence.” Google describes the function of its own citations as corroboration rather than extraction. Read the two together and the meaning of a citation becomes precise: it says this page is compatible with what the model said, not that the model copied this page, and certainly not that the model agreed with it. Citation counts are a visibility metric with a known error rate.

Grounding also does not make the system deterministic. Source-level overlap across repeated identical queries runs 0.34 to 0.42, and URL-level similarity between Google and Gemini runs 0.11 to 0.18. Any measurement design that assumes the same prompt returns the same sources is measuring its own noise, and any single screenshot presented as evidence of a citation gain is one draw from a distribution.

What this means for a page you control

The mechanism makes a short list of things worth doing and a shorter list of errors worth avoiding, and neither list contains a novel tactic.

• Be retrievable first. Everything downstream is conditional on being fetched by a conventional index. That is search engine optimization, and it is the only stage of the pipeline with documented requirements.
• Write passages that stand alone. If a self-contained answer does not exist inside one chunk, no amount of page-level authority creates one. This is a structural argument, not a stylistic one.
• Keep the two crawler roles straight. OpenAI documents GPTBot as crawling content for training generative AI models and OAI-SearchBot as surfacing websites in ChatGPT search results, and states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Blocking training and remaining citable are compatible positions. Blocking the wrong one is expensive in either direction.
• Do not treat structured data as a retrieval lever. John Mueller, speaking personally on 2 January 2026, called whether schema helps language models “yes, no, and it depends” — useful where values are basically impossible to read accurately from a text page, and sometimes “wishful thinking” elsewhere. No operator documents structured data as an input to AI answer source selection.
• Use the word precisely. Saying RAG when you mean any AI system that touches documents costs nothing to fix and is the difference between explaining a mechanism and repeating a buzzword. Where the retriever is a search engine rather than a private vector index, say so.

Frequently asked questions