RAG Explained for Developers: How Retrieval-Augmented Generation Works

Large language models know a lot, but they do not know your data: your documentation, your support tickets, last week's policy change. Retrieval-augmented generation (RAG) is the standard way to fix that. Instead of retraining the model, you look up the relevant information at question time and give it to the model along with the question.

This guide explains how a RAG system works, step by step, and the design decisions that make the difference between a demo and something people trust.

The idea in one paragraph

When a user asks a question, a RAG system first retrieves the passages from your documents that are most relevant to the question, then augments the prompt with those passages, and finally lets the model generate an answer grounded in them — ideally citing which passages it used.

Why not just put everything in the prompt?

Modern models have large context windows, so for a small, stable set of documents, putting them directly in the prompt can be the simplest solution. RAG wins when:

• Your data is bigger than the context window, or would be expensive to send with every request. Every input token costs money — see what tokens are and how they are billed.
• Your data changes often. Updating an index is instant; there is no retraining.
• You need sources. Retrieval tells you exactly which documents an answer came from.
• You need access control. You can filter retrieval so each user only sees documents they are allowed to see.

RAG vs fine-tuning

Fine-tuning changes a model's behavior: tone, format, specialized skills. It is a poor way to add facts, because facts baked into weights cannot be cited, updated or permission-checked. A common rule of thumb: use RAG for knowledge, fine-tuning for behavior — and try good prompting before either.

Step 1: Ingest and chunk your documents

Documents are loaded (PDFs, HTML, Markdown, tickets, database rows), cleaned and split into chunks — passages small enough to retrieve precisely.

Chunking decisions matter more than most people expect:

• Size: a few hundred tokens per chunk is a common starting point. Too small and chunks lose context; too large and retrieval becomes imprecise and prompts get expensive.
• Overlap: a small overlap between neighboring chunks keeps sentences that cross a boundary from being cut in half.
• Structure: split along headings, sections or paragraphs instead of fixed character counts when you can. A chunk that is one complete section of a doc is far more useful than an arbitrary slice.
• Metadata: store the source, title, URL, section heading, date and permissions with every chunk. You will need them for filtering and citations.

A useful trick is to prepend the document title and section heading to each chunk's text before indexing, so a chunk like "Click Save to apply" still carries the context "Billing settings › Changing your plan".

Step 2: Create embeddings

An embedding is a list of numbers (a vector) that represents the meaning of a piece of text. An embedding model turns each chunk into a vector, such that texts with similar meanings end up close together in vector space.

"How do I reset my password?"      → [0.12, -0.48, 0.33, …]
"Steps to recover account access"  → [0.10, -0.45, 0.31, …]   (close: similar meaning)
"Quarterly revenue grew 8%"        → [-0.61, 0.22, 0.05, …]  (far away)

You embed every chunk once at indexing time, and embed each user question at query time with the same model. If you switch embedding models, you must re-embed everything.

Step 3: Store vectors and search

Vectors go into a store that supports fast similarity search. Options range from dedicated vector databases to extensions of databases you may already use — for example pgvector for PostgreSQL (see SQLite vs PostgreSQL). For a few thousand chunks, even a brute-force search in memory is fast enough.

At query time:

1. Embed the question.
2. Find the chunks whose vectors are most similar (usually by cosine similarity).
3. Apply metadata filters — product, language, date, user permissions.
4. Return the top results, often somewhere between 5 and 20 chunks.

Hybrid search: don't drop keywords

Embeddings capture meaning but can miss exact terms: error codes, product names, function names, IDs. Hybrid search combines vector similarity with classic keyword search (such as BM25) and merges the results. For technical content it is often a large improvement over vectors alone.

Step 4: Rerank

The first retrieval step is optimized for speed and recall. A reranker — a model that scores each question–chunk pair directly — then reorders the candidates for precision. A typical pattern is to retrieve 50 candidates cheaply and rerank them down to the best 5–10 for the prompt.

Step 5: Generate the answer

The prompt combines instructions, the retrieved chunks and the question:

You are a support assistant for Acme Cloud.
Answer the question using only the documents below.
If the documents do not contain the answer, say you don't know.
Cite the document IDs you used, like [doc-3].

<documents>
[doc-1] Billing › Changing your plan: ...
[doc-3] Billing › Refunds: ...
</documents>

Question: Can I get a refund if I downgrade mid-month?

The instructions do three important things: restrict the answer to the provided context, give the model permission to say "I don't know", and require citations so users — and you — can check the answer.

Step 6: Evaluate

A RAG system can fail in two places, and you should measure them separately:

Stage	Question to ask	How to measure
Retrieval	Did we find the right chunks?	For a set of test questions with known source documents, check whether those documents appear in the top results
Generation	Did the model use them correctly?	Check answers for correctness, faithfulness to the sources and proper citations — by human review or an LLM grader checked against human judgments

Start with 30–50 real questions with known answers. Every time you change chunking, embeddings, prompts or models, rerun the set. Without this, you are tuning by feel.

Common failure modes

• Bad chunks: tables split in half, headers separated from their content, scanned PDFs with garbled text. Inspect your chunks — it is the most valuable hour you will spend.
• Retrieval misses: the answer exists but is never retrieved. Try hybrid search, better chunk boundaries, adding headings to chunks or rewriting the user's question into a better search query.
• Stale or conflicting documents: two versions of a policy both get retrieved. Remove outdated content or filter by date.
• Hallucinated details: the model fills gaps with plausible-sounding information. Strengthen the "only use the documents" instruction, require citations, and show sources in the UI.
• Leaking data across users: always apply permission filters at retrieval time, not after generation.

A minimal architecture to start with

1. A script that loads documents, chunks them by section and stores text, metadata and embeddings.
2. An API endpoint that embeds the question, runs hybrid search with permission filters, and reranks.
3. A prompt template with strict grounding and citation instructions.
4. A UI that shows the answer and the source links.
5. A test set of real questions you run on every change.

You can build all of this with a Python backend (see FastAPI vs Flask vs Django), PostgreSQL with pgvector, and any embedding and chat model provider.

FAQ

Does RAG eliminate hallucinations?

No, but it reduces them substantially and makes them easier to catch, because answers can be checked against cited sources.

How many chunks should I put in the prompt?

Enough to contain the answer, but no more. Start with around 5–10 reranked chunks and adjust based on your evaluation set — more context is not always better.

Do I need a vector database?

Not necessarily. For small collections, an in-memory index or a vector extension in your existing database is enough. Dedicated vector databases help at large scale or with heavy filtering needs.

What is "agentic RAG"?

A setup where the model decides when and what to search — possibly running several searches, refining the query, or calling other tools — instead of one fixed retrieval step before generation.

Further reading

• Your model provider's documentation on embeddings and retrieval
• pgvector on GitHub
• Related: What are tokens in AI models?
• Related: How to write better prompts for AI coding assistants