Multi-Agent RAG System

The Problem

Most RAG implementations follow the same recipe: embed the documents, retrieve the top-k chunks, and stuff them into a prompt. That works well enough for a chatbot answering questions about a single CV. It starts breaking when the source material is a book, a technical manual, or an internal knowledge base, and the questions arriving are not all the same kind of question.

Some questions are answered directly from the source. Some are follow-ups that only make sense with conversation history. Some are conversational and need no retrieval at all. And every so often, the retrieved context simply doesn't contain the answer, at which point a single-pass pipeline will confidently generate something anyway.

The challenge: build a RAG pipeline that knows the difference between these cases, and refuses to ship an answer it can't ground.

The Architecture

The Agents

Each stage is a separate agent with one job. They communicate through typed contracts rather than passing loose dictionaries around, which is what makes the pipeline traceable when something goes wrong.

• Router: reads the incoming question and decides how it should be answered. Retrieval, conversation history, or neither. Cheap to run, and it prevents every question from paying the cost of a full retrieval pass.
• Retrieval Agent: grounds the response in the source material. Queries the FAISS vector store, assembles context, and returns a structured result the next stage can evaluate.
• Evaluator Agent: scores factuality and hallucination risk before an answer reaches the user. If faithfulness to the retrieved source is too low, the pipeline loops back to retrieval rather than returning a plausible-sounding guess.
• Summarizer Agent: manages conversation context. Raw history grows without bound across a long session, so it compresses prior turns into a summary that preserves meaning without consuming the whole context window.

Why It's Hard

• Routing is cheap to get wrong. Send a conversational question down the retrieval path and you add latency while dragging irrelevant context into the answer. The router has to be conservative in one direction and permissive in the other, and the failure modes look identical from the outside.
• Faithfulness must be enforced, not hoped for. The evaluator exists because "the model said it" is not evidence. Gating the output on a faithfulness score turns a silent hallucination into a retry, which is a far better failure mode than a confident wrong answer.
• Typed contracts between agents. With four stages in sequence, a schema change in one will silently corrupt everything downstream if the boundary is untyped. Declaring input and output shapes makes the pipeline fail loudly at the seam instead of producing subtly wrong output three stages later.
• Context growth is the quiet killer. Long sessions consume the context window fast, and naive truncation loses exactly the information the user just established. Summarisation has to be selective. It's a compression problem, not a deletion problem.
• It has to work on any document set. Books, manuals, and internal knowledge bases differ enormously in structure and vocabulary. The pipeline can't assume a schema for the source material.

What I'd Do Differently

• Make evaluator thresholds configurable per document set. A legal manual and a collection of blog posts do not deserve the same faithfulness bar. A single global threshold forces you to pick between being too strict on one and too lax on the other.
• Add proper tracing from day one. Debugging a four-agent pipeline from log files alone is genuinely painful. Per-stage tracing with inputs, outputs, and timing would have shortened almost every debugging session.
• Cache retrieval results. Repeated and near-duplicate questions are common in a knowledge base setting, and re-embedding and re-searching them is wasted work.
• Build the evaluation set before building the pipeline. This is the same lesson as my RAG evaluation post. A benchmark set of questions with ground-truth answers tells you whether a change to the router helped or hurt. Without it, you're tuning by vibes.

Key Takeaways

The interesting engineering in a RAG system is the decision layer around the retrieval, not the retrieval itself. Splitting the pipeline into agents with narrow responsibilities and explicit contracts means each failure has exactly one place to live, and adding a check on faithfulness turns "the model sounded confident" into a measurable, enforceable property.