Retrieval-Augmented Generation (RAG) Explained

Someone asks your support assistant about the refund window. The model has never seen your policy, so it answers from general knowledge about refund policies in general. It sounds right and it's wrong for your company. As the hallucination post showed, that is what a model does when it lacks the facts.

Retrieval-augmented generation (RAG) fixes the supply problem. Before the model answers, your system finds the relevant text and puts it in the prompt. The model's job changes from "remember this" to "read this and answer from it".

The pipeline

flowchart LR
                          subgraph Ingest
                              D[Documents] --> C[Chunk]
                              C --> E[Embed]
                              E --> I[(Index plus metadata)]
                          end
                          subgraph Answer
                              Q[Question] --> R[Retrieve and rerank]
                              I --> R
                              R --> P[Prompt with sources]
                              P --> M[LLM]
                              M --> V[Verify citations]
                              V --> A[Answer or 'not found']
                          end
                      

Ingest happens ahead of time and again whenever content changes. The answer path runs on every question. The pieces behind each box are covered in tokens and embeddings and vector search.

A whole pipeline in a few lines

This version uses word counts as a toy embedding and stops short of calling a model, so you can see every step. Swap in a real embedding model, a vector store and an LLM client and the structure stays the same.

import math
                      from collections import Counter
                      
                      DOCS = {
                          "S1": "Refunds are available within 30 days of purchase.",
                          "S2": "Digital goods are non-refundable once downloaded.",
                          "S3": "Support is open Monday to Friday, 9am to 6pm.",
                      }
                      
                      def embed(text):                              # toy embedding: word counts (swap in a real model)
                          return Counter(text.lower().replace(".", "").replace("?", "").split())
                      
                      def cosine(a, b):
                          dot = sum(a[w] * b[w] for w in a)
                          return dot / (math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values())))
                      
                      def retrieve(question, k=2):
                          q = embed(question)
                          ranked = sorted(DOCS, key=lambda d: -cosine(q, embed(DOCS[d])))
                          return ranked[:k]
                      
                      def build_prompt(question, source_ids):
                          context = "\n".join(f"[{s}] {DOCS[s]}" for s in source_ids)
                          return ("Answer using ONLY the sources below. Cite source ids. "
                                  "If the answer is not in the sources, say you can't find it.\n\n"
                                  f"Sources:\n{context}\n\nQuestion: {question}")
                      
                      question = "How many days after purchase are refunds available?"
                      ids = retrieve(question)
                      print(ids)
                      print(build_prompt(question, ids))
                      

The refund chunk is retrieved first, and the prompt has three jobs: restrict the model to the sources, make it cite ids, and give it a way out when the answer isn't there. That last instruction matters as much as the first two.

Where RAG goes wrong

• Retrieval misses. The right chunk isn't in the top results, so the model improvises. This is the most common failure by far.
• Bad chunks. Too big and the answer is buried; too small and it loses its meaning. Chunk along headings and paragraphs, and keep titles with the text.
• Stale or conflicting sources. Old versions of a policy sit in the index next to the new one. The model can't tell which is current unless you tell it.
• Ignored context. Models sometimes answer from memory despite the sources, or blend the two. Ask for quotes and check them in code.
• Overstuffed prompts. Ten loosely relevant chunks are worse than three good ones, and they cost more.

The solution architect's view

RAG is the pattern most enterprise assistants are built on, and the architecture around it matters more than the model inside it.

Owner of the knowledge, owner of the answer

Every source in the index needs an owner, a refresh schedule and a way to be retired. If nobody owns the content, nobody owns the answers, and stale pages quietly turn into confident wrong replies.

Make groundedness checkable

Return answers with source ids and quotes, and verify the quotes against the retrieved text before showing anything. Show citations to users so they can check for themselves. Where the cost of an error is high, route low-confidence or uncited answers to a person.

Design for "not found"

Retrieval returns something even for questions you have no content on. Set a relevance threshold, handle empty or weak results explicitly, and track the not-found rate. A rising rate shows you where content is missing, which is useful product information.

Security and privacy at retrieval

Apply the user's permissions in the search, redact sensitive data before it is embedded, and treat retrieved text as untrusted input that may carry instructions. Log which sources fed each answer so you can audit it later.

Operate it as two systems

Ingestion and answering fail differently and scale differently. Monitor ingestion lag, failed documents and index size, and separately monitor retrieval quality, latency, token cost and answer quality.

Questions for the design review

1. Who owns each source, and how does stale content leave the index?
2. How do we measure retrieval quality and answer groundedness?
3. What does the user see when nothing relevant is found?
4. Where are permissions enforced, and what is logged for audit?
5. What is the cost per answer, and what is the token budget for context?