Retrieval-Augmented Generation (RAG) has quickly become the dominant pattern for building LLM-powered applications on private or domain-specific data. At DIGIT, we have deployed RAG pipelines in production for legal document review, enterprise knowledge bases, and customer support automation. Here is what we have learned.
Architecture Overview
A production RAG pipeline has three core stages: ingestion, retrieval, and generation. Getting each stage right independently is necessary but not sufficient — the system fails if the handoffs between stages are poorly designed.
Ingestion: Documents are loaded, cleaned, chunked, embedded, and stored in a vector database. The chunking strategy is the single most impactful decision in the entire pipeline. We default to recursive character splitting with a chunk size of 512 tokens and 10% overlap for most document types. For structured documents (contracts, forms), we use semantic chunking based on heading hierarchy.
Retrieval: Given a query, we embed it and perform approximate nearest-neighbour search against the vector store. We always combine dense retrieval (vector similarity) with sparse retrieval (BM25 keyword matching) using reciprocal rank fusion. Pure vector search misses exact keyword matches; pure BM25 misses semantic similarity. Hybrid retrieval consistently outperforms either alone by 15–25% on our evaluation sets.
Generation: Retrieved chunks are assembled into a context window and passed to the LLM with a structured prompt. We enforce strict citation requirements: the model must reference the chunk ID for every factual claim. This makes hallucinations auditable.
Vector Store Choices
We have used Pinecone, Weaviate, pgvector, and ChromaDB across different projects. Our current default for new projects is pgvector — it runs inside PostgreSQL, eliminates a separate infrastructure dependency, and handles corpora up to ~5M chunks without performance issues. Pinecone is our choice when the corpus exceeds 10M chunks or when multi-tenancy isolation at the vector-store level is a hard requirement.
Evaluation
Shipping a RAG pipeline without an evaluation framework is shipping blind. We use a combination of RAGAS metrics (faithfulness, answer relevancy, context precision, context recall) and task-specific human evaluation on a golden dataset of 50–100 representative queries. We run RAGAS on every deployment to catch regressions.
Common Failure Modes
- Chunk boundary hallucination: The answer is split across two chunks and the retriever returns only one. Fix: increase overlap or use semantic chunking.
- Query-document vocabulary mismatch: The user asks "invoice" but the document says "billing statement." Fix: query expansion with HyDE (Hypothetical Document Embeddings) or synonym injection.
- Context window overflow: Too many retrieved chunks push the answer out of the context. Fix: rerank retrieved chunks with a cross-encoder and take top-k after reranking.
If you are building a RAG system and want DIGIT's help, reach out at info@digit.com.pk.