RAG Architecture for the Enterprise: From Ingestion to Evaluation

RAG architecture is the design of a system that retrieves relevant content from your own sources and gives it to a language model, so answers are grounded in that content rather than in the model’s training data. In an enterprise, a good RAG architecture also respects who is allowed to see what, keeps the index current as sources change, and can prove with measurements that it is returning correct, cited answers.

Most RAG projects that stall do so on the retrieval side, not the model side. The model can only be as good as the passages it is handed. This guide walks through each stage of a RAG architecture, the design choices that matter, and the failure modes we see most often.

The end-to-end flow

A production RAG system has an offline path that prepares content and an online path that answers requests.

Offline (indexing):

1. Connect to sources and sync changes, including deletions and permission changes.
2. Parse documents into clean text and structure (headings, tables, lists).
3. Split content into chunks and attach metadata.
4. Embed chunks and write them to keyword and vector indexes.

Online (query):

5. Understand the query: rewrite, expand or decompose it if needed.
6. Resolve the user’s identity and permissions.
7. Retrieve candidates with hybrid search, filtered by permissions and metadata.
8. Rerank candidates and select the final context.
9. Generate an answer with citations, or decline when the context does not support one.
10. Log the full trace for evaluation and debugging.

Each stage is a place where quality can be lost. A sound RAG architecture makes each one observable on its own, so you can tell whether a bad answer came from parsing, retrieval or generation.

Ingestion and parsing

Ingestion is unglamorous and decisive. Connectors must handle incremental sync, not just full reloads, and they must propagate deletions and permission changes, not just new content. An index that still holds a document removed from SharePoint last month is both a quality and a compliance problem.

Parsing deserves real investment. PDFs with multi-column layouts, scanned documents, tables, slide decks and spreadsheets all break naive text extraction. Tables are a common failure: flattened into text, they lose the relationships between headers and cells, and the model then misreads them. Preserve structure where you can, keep headings as metadata, and route document types to parsers suited to them.

Capture metadata at this stage: source system, document owner, last modified date, document type, classification, business unit, and access control lists. You will need all of it for filtering, freshness and permissions later.

Chunking

Chunking decides the unit of retrieval. Too large, and chunks carry irrelevant text that dilutes relevance and wastes context. Too small, and a chunk loses the context needed to understand it.

Strategy	How it works	Good for	Watch out for
Fixed size with overlap	Split every N tokens, overlapping by a margin	Quick baselines, uniform text	Splits mid-sentence and mid-table
Structure-aware	Split on headings, sections, list items, table boundaries	Policies, manuals, contracts	Depends on good parsing
Semantic	Split where topic shifts, based on embedding similarity	Long unstructured prose	Extra compute, less predictable
Parent-child	Retrieve small chunks, return the larger parent section to the model	Precise matching with enough context	More index complexity

In our experience, structure-aware chunking with a parent-child pattern is a strong default for enterprise documents. Prepend a short contextual header to each chunk (document title, section path) so that a chunk reading “the limit is 30 days” still carries what the limit applies to.

Retrieval: hybrid by default

Dense vector search matches meaning, which handles paraphrase well. It is weaker on exact tokens: part numbers, policy IDs, legal citations, names and internal acronyms. Keyword search, typically BM25, is the reverse. Enterprise queries mix both, so hybrid retrieval is the sensible default: run keyword and vector search in parallel and merge the results, commonly with reciprocal rank fusion, which combines rankings without needing the two scores to be on the same scale.

Other retrieval techniques worth considering:

• Metadata filters for document type, date range, business unit or region, applied inside the search rather than after it.
• Query rewriting to turn conversational follow-ups (“what about for contractors?”) into standalone queries.
• Query decomposition for questions that need several pieces of evidence.
• Freshness weighting where recent versions of a policy should outrank old ones.

The choice of store matters less than people expect. pgvector, search engines with vector support, and dedicated vector databases such as Pinecone can all work. Prioritize solid hybrid search, efficient filtered queries (essential for permissions), and something your team can operate.

Reranking

First-stage retrieval optimizes for recall: get the right passage somewhere in the top 50 or 100. A reranker, usually a cross-encoder model that scores each query and passage pair together, then orders those candidates for precision, and you pass the top handful to the model.

Reranking typically gives one of the larger quality gains available in a RAG pipeline for modest effort. The costs are latency and per-request compute, so tune how many candidates you rerank and measure the effect on your own evaluation set rather than assuming.

Permissions-aware retrieval

This is where enterprise RAG differs most from tutorials, and where the most serious incidents happen. If a user cannot open a document in the source system, the RAG system must never use that document to answer them.

The design has three parts:

1. Carry access controls into the index. Store each chunk with the users and groups allowed to read it, taken from the source system at ingestion and refreshed when permissions change.
2. Resolve identity at query time. Pass the real end-user identity from the application, not a shared service account, and expand group memberships through your identity provider.
3. Filter inside retrieval. Apply the permission filter as part of the search query so unauthorized chunks are never candidates. Filtering after retrieval can silently return too few results and still risks leakage through logs or caches.

For highly sensitive sources, add a late check against the source system before showing a citation, which catches permission changes that have not yet synced. Treat sync lag as a known risk with a stated tolerance, and prioritize permission revocations over content updates.

Two more rules: never rely on the system prompt to tell the model what the user may see, and remember that retrieved content can contain prompt injection. Content from email, tickets or the web should be treated as untrusted input. Our LLM security threat model covers these threats in more depth.

Generation and citations

Assemble context deliberately: order passages by relevance, include source titles and identifiers, and stay well inside the context window so the instructions are not crowded out. Instruct the model to answer only from the provided context, to cite sources for each claim, and to say plainly when the context does not contain the answer.

Then verify rather than trust. Check that cited sources were actually in the retrieved set, and consider a lightweight groundedness check for high-stakes use cases. A system that says “I could not find that in the policy library” is far more useful than one that answers confidently from general knowledge.

Evaluation

A RAG system without an evaluation set cannot be improved safely, because every change to chunking, embeddings, retrieval or prompts can help one query and hurt another.

Build a labeled set of real questions, drawn from logs or subject matter experts, with the passages that should be retrieved and the facts a correct answer must contain. Then measure the two halves separately.

Stage	What to measure	Example metrics
Retrieval	Are the right passages found and ranked high?	Recall at k, mean reciprocal rank, precision at k
Generation	Is the answer supported, correct and useful?	Faithfulness to context, answer correctness, citation accuracy, appropriate refusal
Operations	Is it fast, affordable and current?	Latency by stage, cost per query, index freshness, permission sync lag

Model-based grading (“LLM as judge”) scales evaluation, but calibrate it against human labels before trusting it, and re-check that calibration when you change the judge model. Run the suite on every change in CI, and track production signals such as user feedback, citation clicks and unanswered queries. Our guide to LLMOps and AI observability covers the production side.

Failure modes and fixes

Symptom	Likely cause	Fix
Correct document exists but is never cited	Parsing lost the text, or chunk lacks context	Improve parsing, add contextual headers, check with retrieval metrics
Misses exact codes or names	Vector-only retrieval	Add keyword search and fusion
Right passage retrieved, wrong answer	Too much or poorly ordered context	Rerank, cut context, tighten instructions
Answers cite outdated policy	No freshness signal, stale index	Version metadata, freshness weighting, deletion sync
User sees content they should not	Permissions not carried or filtered late	Chunk-level ACLs, identity propagation, in-query filtering
Quality drops after a model upgrade	No regression suite	Evaluation set in CI, gated releases
Costs climb with usage	Oversized context, no caching	Smaller context, caching where safe, route simple queries to smaller models

When RAG is not the right tool

RAG is not the answer to every knowledge problem. If the answer sits in structured data, a governed text-to-SQL or API approach is usually more accurate. If the full source is small, stable and fits comfortably in the context window, passing it directly can be simpler. If you need the model to adopt a format or style, fine-tuning may fit better. Many enterprise systems combine these, with RAG as one retrieval path among several inside a broader enterprise AI reference architecture.

How we help

We design and build RAG and knowledge systems that respect your permissions and can prove their accuracy, from ingestion pipelines to evaluation harnesses. Our RAG and knowledge architecture work typically starts with an assessment of your current retrieval quality against a labeled question set. If your RAG pilot works in demos but not in production, talk to us.