Retrieval-Augmented Generation (RAG) has quickly become the dominant pattern for building LLM-powered applications on private or domain-specific data. At DIGIT, we have deployed RAG pipelines in production for legal document review, enterprise knowledge bases, and customer support automation. Here is what we have learned, with the specific tools, papers, and metrics behind each decision.
Architecture Overview
A production RAG pipeline has three core stages: ingestion, retrieval, and generation. Getting each stage right independently is necessary but not sufficient — the system fails if the handoffs between stages are poorly designed.
Ingestion: Documents are loaded, cleaned, chunked, embedded, and stored in a vector database. The chunking strategy is the single most impactful decision in the entire pipeline. We default to recursive character splitting with a chunk size of 512 tokens and 10% overlap for most document types, using patterns documented in LangChain's text splitter documentation. For structured documents (contracts, forms), we use semantic chunking based on heading hierarchy.
Retrieval: Given a query, we embed it and perform approximate nearest-neighbour search against the vector store. We always combine dense retrieval (vector similarity) with sparse retrieval (BM25 keyword matching) using reciprocal rank fusion, the rank-merging technique introduced by Cormack, Clarke and Buettcher in 2009. Pure vector search misses exact keyword matches; pure BM25 misses semantic similarity. Hybrid retrieval consistently outperforms either alone by 15–25% on our evaluation sets — a gap that tracks with the wider literature on best practices for RAG retrieval components.
Generation: Retrieved chunks are assembled into a context window and passed to the LLM with a structured prompt. We enforce strict citation requirements: the model must reference the chunk ID for every factual claim. This makes hallucinations auditable.
Vector Store Choices
We have used Pinecone, Weaviate, pgvector, and ChromaDB across different projects. Our current default for new projects is pgvector — it runs inside PostgreSQL, eliminates a separate infrastructure dependency, and handles corpora up to ~5M chunks without performance issues. Pinecone is our choice when the corpus exceeds 10M chunks or when multi-tenancy isolation at the vector-store level is a hard requirement.
Evaluation
Shipping a RAG pipeline without an evaluation framework is shipping blind. We use a combination of RAGAS metrics — faithfulness, answer relevancy, context precision, context recall — and task-specific human evaluation on a golden dataset of 50–100 representative queries. We run RAGAS on every deployment to catch regressions before they reach a client's production system.
Common Failure Modes
- Chunk boundary hallucination: The answer is split across two chunks and the retriever returns only one. Fix: increase overlap or use semantic chunking.
- Query-document vocabulary mismatch: The user asks "invoice" but the document says "billing statement." Fix: query expansion with HyDE (Hypothetical Document Embeddings) — a technique from Gao et al.'s 2022 paper that generates a hypothetical answer first, then embeds that for retrieval — or plain synonym injection for simpler cases.
- Context window overflow: Too many retrieved chunks push the answer out of the context. Fix: rerank retrieved chunks with a cross-encoder and take top-k after reranking.
RAG is not a solved problem you configure once — it is an evaluation loop you run continuously as your document set and query patterns evolve. If you are building a RAG system and want DIGIT's help — whether that is architecture, evaluation tooling, or a production migration — reach out at info@digit.com.pk.