Retrieval-augmented generation

Retrieval-augmented generation (RAG) is an artificial-intelligence architecture in which an external knowledge base is searched before a response is generated. Retrieved passages are supplied to a large language model together with the user's question. The model then creates an answer from that context and can cite the sources that were used.

RAG combines two mechanisms: information retrieval finds relevant evidence and a generative model turns it into a coherent response. The information does not have to be stored in model parameters. Documents can be updated independently of the model, filtered by access rights and shown to the user as evidence. The United States National Institute of Standards and Technology defines RAG as a generative-AI system in which a model is paired with a separate retrieval system or knowledge base.

The architecture is used for internal document search, technical support, scientific-literature analysis and other tasks where answers must reflect a large or frequently changing collection. RAG does not guarantee correctness: a system can retrieve an irrelevant passage, miss an important document or misrepresent the supplied text. Retrieval quality and grounding therefore require separate evaluation.

Architecture

Knowledge-base preparation and query processing in a RAG system

Most RAG systems have two pipelines. One prepares documents and a search index in advance. The other runs for each user query, retrieves suitable passages and supplies them to the model.

Knowledge-base preparation

Sources may include documents, web pages, databases, email, manuals, source code and transcripts. Each source should retain provenance, update time, ownership and access rules. Without this metadata, a system cannot reliably cite a location, remove an obsolete version or prevent an unauthorised user from retrieving a confidential document.

Documents are parsed with regard to headings, paragraphs, tables, lists and captions. The text is then divided into chunks. A very short chunk can lose necessary context; a very long one can contain irrelevant material and consume much of the model's context window. Boundaries therefore follow the material: a contract can be split by section, a reference work by entry and code by function or class.

Two complementary representations are commonly created:

• a full-text index stores words and positions and is effective for exact terms, names, identifiers and quotations;
• a vector index stores embeddings, numerical representations of meaning that can retrieve semantically similar wording.

Changed documents must be processed again. When a source is deleted, its chunks must also be removed from the index; otherwise the system can continue returning information that no longer exists in the authoritative collection.

Retrieval

A user question is converted into a search query. A basic system compares its embedding with document vectors and returns the nearest chunks. A stronger pipeline combines semantic and full-text search, applies metadata filters and merges the results. This hybrid search preserves exact matching while recognising similar meaning.

Initial retrieval often favours recall and may produce dozens or hundreds of candidates. A separate reranking model scores each candidate against the question and reorders them. Only a small set of highest-ranked passages enters the language-model context. This applies a more accurate but expensive comparison after a cheaper search has reduced the corpus.

Context and answer generation

The application combines the question, instructions and retrieved chunks into one model request. Every chunk receives an identifier or source link. Instructions commonly require answers to remain within the supplied evidence, acknowledge missing information and attach citations to claims.

The model does not see the full index. It sees only the text selected by the application. If retrieval misses the required passage, the model cannot use it. If the context contains conflicting document versions, the system must use dates, status and source priority or explicitly report the conflict.

After generation, an application can check whether claims are supported, the response follows the required structure and cited locations exist. A person should make the final decision in high-consequence uses.

Retrieval methods

Method	Comparison	Strengths	Limitations
Full-text retrieval	Words, word forms and frequency	Exact names, codes, quotations and rare terms	May miss semantically similar wording
Vector retrieval	Embeddings of queries and chunks	Semantic similarity and paraphrases	Can return a related but factually unsuitable passage
Hybrid retrieval	Results from full-text and vector search	Combines exact terminology with semantic recall	Requires fusion and tuning of different ranking signals
Retrieval with reranking	Query–passage pairs after initial selection	Improves the precision of top results	Adds latency and processing cost

Production retrieval also uses filters. A search may be limited to the user's organisation, language, current document version or records available to a particular account. Filtering after generation is insufficient: an unauthorised document must never enter model context.

Architectural variants

Basic RAG

Basic RAG performs one retrieval for the original question, inserts several top chunks into the prompt and generates an answer. It is simple and can work well with a small, uniform corpus whose terminology resembles user questions.

Advanced RAG

An advanced pipeline rewrites the question, creates several queries, combines indexes, filters metadata and reranks results. After generation it may verify citations or search again when evidence is insufficient. Each addition addresses a different failure and should be justified by measurements rather than treated as a mandatory component.

Agentic RAG

In an agentic system, the model decides when and where to retrieve. An AI agent can query several collections, refine a query, open an original document and search again after discovering a gap. This supports complex research but increases the number of calls, latency and the risk of choosing an inappropriate source.

RAG can also use Model Context Protocol: a server exposes a search tool or resources and a client supplies the result to the model. MCP standardises the connection but does not define chunking, indexing, ranking or generation.

RAG, long context and fine-tuning

RAG is not the only way to provide domain knowledge. The choice depends on collection size, update frequency and whether the goal is to change model knowledge or behaviour.

Approach	Information available at answer time	Updating	Suitable uses	Main limitation
RAG	Chunks retrieved from an external knowledge base	Re-index changed documents	Large, changing or access-controlled collections; answers with citations	Required evidence may not be retrieved
Long context	Documents supplied in full to the model	Replace the supplied documents	Analysis of a limited, preselected set	Cost and context limits; important details can be lost in a large input
Fine-tuning	Model parameters and learned behaviour	Another training cycle	Stable changes to style, format or behaviour	Poor fit for frequently changing facts and exact citations
Conventional search	A ranked list of documents	Update the search index	Navigation to sources without a generated synthesis	The user must read and compare the documents

Long context and RAG can be combined. Retrieval reduces a very large collection to a few documents and the application then supplies complete sections. Google notes that long context is useful for selected source material, while RAG scales beyond the context window and can reduce the number of input tokens.

Fine-tuning can also coexist with retrieval. A model may be trained to follow an industry format while current facts come from a knowledge base. Retrieval does not repair weak reasoning, and training is an inconvenient way to store frequently changing instructions.

Evaluation

RAG must be evaluated by stage rather than with one aggregate score. Otherwise a fluent response can conceal poor retrieval, and a correct source can conceal an unsupported answer.

Retrieval quality asks whether required chunks occur among the top results and how much irrelevant text accompanies them. A test set contains realistic questions and known supporting documents. Recall and precision are measured at a selected retrieval depth.

Faithfulness asks whether answer claims are supported by retrieved context. Answer relevance asks whether the response addresses the user's need, while citation quality checks that citations point to passages that support the associated claims. RAGAS proposed separate, reference-free metrics for context relevance, faithfulness and answer quality.

Automated scores should be supplemented by human review. A model used as a judge can prefer a style or mistake a plausible statement for a supported one. Changes are compared with a fixed baseline before release; failed searches, refusals, complaints and source visits are monitored afterwards.

Common failures

Document parsing

Text from columns, tables and scanned pages can be extracted in the wrong order. A heading may be separated from its section or a unit from its number. The index then contains passages that never existed in the source. Parsing quality should be checked before embedding and generation are tuned.

Chunking

Splitting at fixed character counts can cut through a definition, condition or table. Excessive overlap creates many nearly identical results. Structure-aware chunks and retrieval of neighbouring passages preserve context more effectively.

Low retrieval recall

A question can contain an abbreviation, spelling error or term from an older document version. Normalisation, synonym lists, query rewriting, hybrid search and multiple query variants can help. Broader retrieval also adds irrelevant candidates, so it is often paired with reranking.

Stale or conflicting sources

An index that is not synchronised with the authoritative system may answer from a withdrawn instruction. Documents need versions and validity periods, and retrieval must filter by status. Conflicting current sources should be reported rather than silently combined.

Unsupported generation

Even with good retrieval, a model can add information from its parameters or infer more than the evidence supports. Refusal rules, passage identifiers and post-generation checks reduce but do not eliminate this risk.

Security

Retrieved documents become part of the model input, so source text must not be treated as a trusted instruction. An attacker can place text in a web page or uploaded file that asks the model to reveal data, ignore policy or invoke a tool. This indirect prompt injection is encountered when the malicious passage is retrieved.

Important protections include:

• verifying document provenance and restricting who can modify an index;
• applying access control during retrieval rather than after generation;
• separating retrieved text from system instructions and refusing commands found in documents;
• removing secrets and personal data before context is sent to an external model;
• granting minimal permissions to tools used by agentic RAG;
• logging retrieved chunks, index version and model output;
• requiring human approval for financial, legal and administrative actions.

Knowledge-base poisoning is a related threat in which crafted documents displace correct results or systematically alter responses. Mitigations include publication controls, duplicate and anomaly detection, versioning and the ability to exclude a source quickly.

When to use RAG

RAG is appropriate when answers must draw on a large corpus, knowledge changes frequently, users have access to different materials or claims need traceable sources. It is especially useful when retrieval can be evaluated independently of generation.

It may be unnecessary for a small instruction that fits entirely in one request, a task that requires no external knowledge or a deterministic calculation. When a rule can be implemented reliably in code, a database constraint or a formal schema, a generated answer should not replace it.

Implementation should begin with representative questions and criteria for a correct response rather than with selection of a vector database. A minimal pipeline is measured to locate errors in sources, parsing, retrieval, ranking or generation. Query expansion, reranking and agent loops are then added only when they address observed failures.

Frequently asked questions

Does RAG eliminate hallucinations?

No. Retrieved context reduces reliance on model memory but does not prevent unsupported claims. Faithfulness evaluation, citations, refusal rules and oversight remain necessary.

Does RAG require a vector database?

No. Retrieval can be full-text, vector-based, graph-based or hybrid. Exact full-text search can outperform vector search for small corpora with identifiers and specialised terminology.

How many chunks should be supplied?

There is no universal number. Too few can omit evidence; too many add noise and consume context. The value is chosen experimentally using retrieval recall, answer faithfulness, latency and cost.

Must the model be retrained when documents change?

Usually not. Changed documents are reprocessed and the index is updated. The model is upgraded separately when its abilities, context capacity or behaviour need to change.

See also

• AI agent
• Model Context Protocol