All How to Build a RAG Pipeline: Architecture, Chunking, Retrieval Strategies
September 28, 202610 min read
An LLM can produce a polished answer and still miss the source that contains the detail a user needs. RAG reduces this risk by retrieving relevant material before generation. The TREC 2025 RAG Track received more than 150 submissions, showing strong research interest in RAG.
A retrieval augmented generation pipeline gives the model access to company documents or frequently updated data at query time. Its quality depends heavily on chunking, retrieval, and the context passed to the LLM.
SapientPro’s engineers explain how to build and tune each part of the pipeline for stronger retrieval and more reliable answers.
What Is a RAG Pipeline and How Does It Work?
The RAG workflow prepares external sources for search and pulls the passages most relevant to a query. The LLM uses that retrieved material to form its response, so weak search results can easily lead to a weaker answer.
Indexing Workflow
A RAG corpus can draw from PDFs, websites, database records, internal documentation, and APIs. Parsing turns these sources into searchable content and preserves the structure and metadata needed later.
Headings, page numbers, URLs, timestamps, document types, and permissions can support filtering, ranking, and source tracing. Keeping this information during ingestion gives the retrieval layer more signals to work with.
The source material is then split into units that can be retrieved independently. Good chunks keep related information together without mixing several unrelated ideas. An embedding model converts each unit into a vector representation that can be compared with the query during semantic search.
The index may contain dense vectors, lexical representations for keyword matching, or both. Hybrid retrieval works well when the corpus contains exact terms like product names, codes, abbreviations, or technical identifiers.
Query-Time Workflow
At query time, the system prepares the request for search. It may rewrite the query or apply metadata filters to narrow the index before retrieval begins.
The first search returns a set of candidate chunks. Reranking trims that list and keeps the passages that match the query most closely. Those passages form the context sent to the LLM.
Anything missed during retrieval stays outside the model’s context. A weak search can therefore carry through to the final answer, even when the LLM itself performs well.
RAG Pipeline Architecture Explained
In production, a strong RAG pipeline architecture needs to handle change without turning every update into a wider rebuild. Source documents, permissions, and retrieval settings evolve as real queries expose weak spots.
Data Ingestion and Document Processing
A growing RAG corpus needs a way to recognize new, changed, and outdated content.
Rebuilding the full index may still work for a small collection, but larger systems usually process only the files that changed. Document IDs, hashes, and version records help track those updates and prevent older copies from resurfacing.
Parsing also needs to preserve the parts of a document that carry meaning.
A table without its row structure or a section separated from its heading may remain readable, yet retrieval loses important clues. Access metadata can be stored during ingestion so users only retrieve content they have permission to see.
Embeddings and Indexing
The embedding model affects how well queries match the indexed text, but changing it later can mean re-embedding the entire corpus. Recording the model version with the index makes that migration easier to control and lets a new index run alongside the current one during testing.
Dense search works well for semantically similar content. Lexical search is stronger with exact names, codes, and uncommon terminology. Many RAG systems keep both available because real queries often need a mix of the two.
Retrieval, Reranking, and Generation
The first search usually returns more candidates than the LLM needs. Reranking narrows that set by checking which passages match the query most closely, and weaker results can drop out before context assembly.
The LLM only needs the passages that directly help answer the query. Some RAG systems follow the same search path every time. Others may change sources, rewrite the query, or run another search when the first results are weak.
How to Build a RAG Pipeline Step by Step
Teams working out how to build a RAG system often lose time tuning embeddings or top-k values before they agree on what a good answer looks like. Real user questions and a small evaluation set make later testing much clearer.
You can use this checklist to track your progress.
Done | Task |
| ☐ | Collect 30–100 representative user questions. |
| ☐ | Mark the source passages that contain the expected answers. |
| ☐ | Decide what metadata and permissions need to survive ingestion. |
| ☐ | Test several chunking options on the same query set. |
| ☐ | Compare embedding and retrieval choices on real corpus data. |
| ☐ | Add reranking only after you can measure baseline retrieval quality. |
| ☐ | Set a context budget and remove duplicate evidence. |
| ☐ | Track retrieval quality separately from answer quality. |
| ☐ | Plan how documents, embeddings, and indexes will be updated. |
1. Start With Real Queries
Build the first test set from questions the application is likely to receive. For each query, mark the document or passage that contains the expected answer. Later retrieval tests can use these examples to show where results improve or degrade.
2. Prepare the Corpus
Remove obvious duplicates and preserve document structure before indexing. Metadata and access permissions need to stay attached to the source data from the beginning, since adding permission logic after unrestricted retrieval is already in place creates extra work.
3. Test Chunking on the Same Questions
Generic chunk-size recommendations rarely tell you much about your own corpus. Run several chunking methods against the same queries and inspect the returned passages. Good splits keep enough surrounding information to answer the question without pulling in unrelated text.
4. Pick an Embedding Model
Embedding models can behave very differently on the same data. Test a few options on real domain queries and pay attention to internal terminology, abbreviations, and names. A small indexed sample can reveal weak matches before the whole dataset is embedded.
5. Build a Retrieval Baseline
Run one retrieval setup against the test queries and inspect the top results. Look for relevant passages that rank well, along with sources that disappear entirely. Change one retrieval setting at a time so each adjustment remains easy to trace.
6. Tune Retrieval Before Adding Complexity
Missed passages usually tell you more than adding another component. Check filters, top-k, query handling, and hybrid weights against the failures you see in testing. Extra layers make diagnosis harder when several things change at once.
7. Add Reranking Where It Helps
Reranking helps when retrieval already finds the right passages but places them too low. Use the same evaluation set to see how much the ordering improves and what it adds in latency and cost.
8. Control What Reaches the LLM
The highest-ranked chunks still need a final review before generation. Remove repeated passages, keep source information attached, and stay within a defined token budget. Do not use extra context just because the model can accept it. Keep only the passages that help answer the query.
9. How Can I Test My RAG Pipeline?
Check whether the expected source appears in the retrieved set and stays in the context after reranking. Then compare that result with the final answer to see where quality starts to drop.
10. Design Updates Before Launch
Production data changes over time, so document updates need a defined process. Decide how edited files will be reprocessed and when embeddings or indexes need rebuilding. Real user queries after launch often reveal problems that testing missed.
Stop Losing Context Between Document Chunks
SapientPro applies modern chunking strategies to your documents to keep essential details together.
RAG Chunking Strategies and How to Choose
Chunking affects what the retriever can find. A poor split can separate a rule from its explanation, and oversized chunks can hide important details inside unrelated text.
Chunking methods may use fixed lengths, document structure, or changes in meaning. Testing them on real documents and queries shows which approach works best for the corpus.
Fixed-Size and Recursive Chunking
Fixed-size chunking splits text after a set number of characters or tokens, often with some overlap between chunks. It is easy to set up and works well for an initial retrieval test. The main weakness comes from arbitrary cuts. A definition can land in one chunk and its condition or example in another.
Recursive chunking follows natural text boundaries more closely. It starts with larger units like paragraphs and moves to sentences or smaller pieces when needed.
Structure-Aware Chunking
Structure-aware chunking uses the document’s own layout to decide where to split the text. Headings, sections, paragraphs, tables, and code blocks provide natural breakpoints.
Technical documentation, reports, contracts, and manuals often benefit from this approach because their structure already helps organize the content. Keeping a heading together with the section below it also makes each chunk easier to understand.
Tables and code need a bit more care. A row without its headers can lose meaning, and part of a function may be hard to interpret on its own. Section titles, page numbers, and other metadata can stay attached to the chunk and help with filtering or ranking.
Semantic Chunking
Semantic chunking splits text when the topic changes. It uses embeddings to compare nearby passages and creates a new chunk when their meaning starts to drift apart.
This can work well for long articles, transcripts, and other text without a clear structure. Indexing takes more work, and chunk sizes can vary considerably. Test it on the same queries as a simpler method and compare the results.
Parent-Child Chunking
Parent-child chunking separates search precision from the amount of context sent to the LLM.
Small chunks make specific details easier to find. Each one links to a larger section, so a match can bring its surrounding text into the model context. For example, a sentence from a refund policy may match the query, but the paragraph around it may contain an important exception.
Large parent sections can quickly use up the context window, so it helps to limit their size and remove duplicate matches to the same parent.
Contextual Retrieval and Late Chunking
Contextual retrieval enriches each chunk with details from the wider document before indexing. A short passage can retain information about its section, product, policy, or topic, making it easier to match with the right query.
Late chunking processes a larger part of the document before individual chunk embeddings are created. This allows surrounding text to influence each representation.
Both approaches can help when regular chunking loses important context. They require more processing, so they are worth testing when simpler methods repeatedly miss relevant information.
{{a5}}
RAG Retrieval Strategies: From Vector Search to Hybrid Retrieval
A RAG system may need to find a paraphrased idea in one query and an exact error code in the next. One search method rarely handles both equally well, so retrieval often combines semantic and lexical signals.
Dense Vector Retrieval
Dense retrieval compares vector representations of the query and indexed chunks. This helps when the same idea appears in different wording. Exact identifiers and rare terminology are harder cases because semantic similarity may give them too little weight.
Sparse Retrieval With BM25
BM25 relies on terms shared by the query and indexed text. It works well for product names, error codes, technical phrases, and other exact terms. Its limits become clearer when the source and the user describe the same idea using different vocabulary.
Hybrid Search and Reciprocal Rank Fusion
Hybrid search runs dense and sparse retrieval together. Reciprocal Rank Fusion, or RRF, combines their ranked results by position instead of comparing two different score scales. This allows semantic matches and exact terms to appear in the same result set.
Metadata Filtering
Metadata filters narrow the search space. A query can be filtered by product, language, date, document type, tenant, or access level before ranking begins. This matters more in large corpora, where only part of the indexed data may be relevant to a given request.
Second-Stage Reranking
Reranking reviews a smaller set of retrieved passages and checks which ones match the query best. It helps when the right passage is already in the results but appears too low. Running reranking after the first search also limits the extra compute to a smaller set of candidates.
When Basic RAG Architecture Is Not Enough
Basic RAG works best when the answer already exists in the source material and the system mainly needs to find the right passage. A different architecture makes more sense when the request relies on live data, calculations, or several connected steps.
When to Use RAG | When Another Approach Works Better |
| Employees search internal policies or technical documentation. | The task is based on fixed business rules or calculations. |
| A support assistant answers questions from product documentation. | The assistant needs to change account data or trigger an external action. |
| Users search large collections of contracts, reports, or research papers. | The answer already lives in structured database fields that SQL can return directly. |
| Documents change often and the application needs recent content. | The request depends on live values like a current balance or order status. |
| Answers need citations back to the original source. | The task is mainly creative generation and external evidence adds little value. |
| User questions often use different wording from the source documents. | The request requires several dependent search or reasoning steps. |
| An internal assistant needs access to private company knowledge. | The knowledge base is small enough to place directly in the prompt. |
Build Your RAG Pipeline with SapientPro
A RAG system can retrieve relevant documents and still miss the passages needed for an accurate answer. SapientPro’s engineers trace these failures through the pipeline, from how content is split and indexed to what reaches the LLM. We help teams build new RAG systems and improve existing ones, testing changes against real queries:
- Review chunking, indexing, and retrieval accuracy;
- Test hybrid search and reranking on your data;
- Refine context selection and measure answer quality;
- Set up monitoring, access controls, and content updates.
Contact SapientPro to discuss your RAG project.
We’ll help identify what needs to change and define the development scope.
Give Your LLM More Relevant Context
SapientPro removes duplicate passages and tunes context assembly around your actual user questions.
