A model only knows what was in its training data and what you put in the prompt. Retrieval augmented generation closes that gap: find the passages relevant to a question, place them in the prompt, and ask the model to answer from them — with citations.
The pipeline
- Chunk. Split documents into passages that respect structure — headings, sections, function definitions — rather than arbitrary character counts.
- Embed. Convert each chunk into a vector with an embedding model, so semantic similarity becomes geometric distance.
- Index. Store the vectors in a vector database or a vector-capable store (pgvector, Elasticsearch, Pinecone, Qdrant) alongside metadata for filtering.
- Retrieve. At query time, embed the question, fetch the nearest chunks, and usually combine that with keyword search — hybrid retrieval beats either alone.
- Rerank and assemble. A cross-encoder reranker reorders candidates; the best few go into the prompt with instructions to cite and to say “I don't know” when the context is insufficient.
Design choices
- Chunk size and overlap — small chunks retrieve precisely but lose context; overlap or a parent-document strategy recovers it.
- Metadata filters — tenant, product, date and permission fields keep results relevant and enforce access control at retrieval time.
- Freshness — incremental re-indexing on document change matters more than raw retrieval quality for most internal tools.
- Query rewriting — expanding or decomposing a vague question before searching noticeably improves recall.
- Long context vs retrieval — large windows reduce how much you must retrieve, but not the case for retrieving: cost, latency and precision still favour fetching only what matters.
Practical notes
- Evaluate retrieval separately from generation — most bad answers are retrieval failures, not model failures.
- Keep a small labelled question set and measure recall@k plus answer faithfulness on every change.
- Never place content in the prompt that the requesting user is not allowed to read.