RAG — retrieval-augmented generation — is the standard architecture for making an LLM answer from your data. Instead of hoping the model memorized the right facts in training, a RAG system searches your documents for passages relevant to the question, inserts them into the context window, and instructs the model to answer from those passages. The model becomes a reasoning layer over retrieved evidence rather than an oracle.
The pipeline in four steps
- Index: split documents into chunks and store them, typically as embeddings (vectors capturing meaning) in a vector database.
- Retrieve: at query time, find the chunks most relevant to the question — semantic similarity, keyword search, or both (“hybrid”).
- Augment: pack the winners into the prompt alongside the question.
- Generate: the model answers, ideally citing which passage supports which claim.
Why RAG instead of fine-tuning or a huge context
Versus fine-tuning: retrieval delivers facts verbatim and updateable — edit a document and the next query sees the change; no retraining, and answers can cite sources. Versus stuffing everything into a 1M-token window: RAG keeps per-query cost flat as your corpus grows past any window, though long context has genuinely eaten RAG’s lunch for bounded corpora — if the whole knowledge base fits comfortably, skipping the retrieval pipeline is simpler and increasingly common. The technique also directly reduces hallucination, since models are far more accurate about text in front of them than facts recalled from weights.
Where it breaks
Every RAG failure is usually a retrieval failure: the right passage wasn’t found (bad chunking, weak embeddings, vocabulary mismatch), so the model answers from thin air. The craft is in evaluation — measuring whether retrieval actually surfaces the right evidence — plus reranking, hybrid search and sensible chunk sizes. Long-context models like Claude Sonnet 5 or GLM-5.2 pair well with coarser, simpler retrieval: fetch generously, let the model sort it out.