Enterprise AI Reference Architecture: Layers, Components and Key Decisions
Every AI workload needs model access, data, retrieval, tools, security and evaluation. A reference architecture decides how those pieces work once, so each team does not reinvent them.
By Enterprise AI Architects · · 8 min read
An enterprise AI reference architecture is a shared blueprint that defines the layers every AI workload uses, the approved components in each layer, and the rules for connecting them. It exists so that the tenth AI application in your company is built on the same model access, retrieval, security and evaluation foundations as the first, rather than on ten different sets of local choices.
This guide describes the layers we use, the components that belong in each, and the decisions that most often determine whether AI workloads reach production.
What a reference architecture is, and what it is not
A reference architecture is not a vendor diagram and not a finished platform. It has three parts:
- A logical view that names layers and responsibilities without naming products.
- An implementation view that maps each layer to the approved components you actually run.
- A decisions log that records each significant choice, the alternatives, and why one won.
It also defines the seams: where one layer hands off to the next, what contract sits at each seam, and which controls run there. Seams are where most production failures occur, so they deserve more attention than the boxes.
The layers at a glance
| Layer | What it does | Typical components |
|---|---|---|
| Experience | Where people and systems use AI | Web and chat interfaces, embedded copilots, APIs, workflow integrations |
| Orchestration | Composes prompts, retrieval, tools and control flow | Application code, agent frameworks such as LangGraph, workflow engines |
| Model access | A single controlled path to models | LLM gateway, model catalog, provider accounts, self-hosted inference |
| Knowledge and retrieval | Finds the right enterprise content for a request | Ingestion pipelines, search and vector indexes, rerankers, permission filters |
| Data foundation | Governed, well-described data | Lakehouse or warehouse, operational stores, catalog, lineage |
| Tools and integration | Lets AI act on enterprise systems | Internal APIs, MCP servers, integration platform, event bus |
| Security and identity | Who can do what, and on whose behalf | Identity provider, policy engine, secrets management, content filters |
| Evaluation and observability | Whether it works, and what it costs | Evaluation harness, tracing, quality and cost dashboards |
| Governance | Whether it should run, and under what conditions | Use-case inventory, risk tiering, review workflow, audit records |
The last three layers are cross-cutting. They apply to every other layer, which is why teams that bolt them on at the end find them hardest to add.
Our Core-and-Floors Model draws this same architecture as a building: five floors that rest on each other, from the data foundation up to experience, with governance and security as a core that runs through every floor. Use the table above when you need the component list, and the model when you need to explain the dependencies to people who do not read architecture diagrams.
Layer by layer
Experience layer
This is where users meet AI: a chat assistant, a copilot inside an existing application, a batch process that drafts documents, or an API another system calls. The key architectural concern is that the experience layer should not talk to model providers directly. It should call your orchestration services, which call models through the gateway. Direct calls are how shadow integrations, unlogged prompts and unmanaged API keys appear.
Orchestration layer
Orchestration is where application logic lives: prompt templates, retrieval calls, tool calls, output validation and control flow. For simple workloads this is ordinary application code. For multi-step agents it may be a framework such as LangGraph or a durable workflow engine. Standardize a small number of patterns rather than one framework for everything; our guide to agentic AI architecture patterns covers when each fits.
Treat prompts as versioned code. They belong in source control, with tests, and should be deployed through the same pipeline as the application.
Model access layer
Every model call should pass through one controlled path, usually an LLM gateway that handles authentication, routing, fallbacks, rate limits, cost attribution and logging. Behind the gateway sits a model catalog: the list of approved models, where each is hosted, which data classifications each may process, and which workloads use it. We cover this component in depth in LLM gateway architecture.
Hosting options usually include managed APIs through a cloud provider (Azure OpenAI, Amazon Bedrock, Google Vertex AI), direct provider APIs (for example Anthropic or OpenAI), and self-hosted open-weight models for specific cost, latency or data residency needs.
Knowledge and retrieval layer
Most enterprise value comes from grounding models in your own content. This layer ingests documents and records, splits and enriches them, indexes them for keyword and semantic search, retrieves and reranks candidates, and filters results by the requesting user’s permissions. It deserves to be a shared service rather than something each team rebuilds. See our RAG architecture guide for the full design.
Data foundation
Retrieval is only as good as the data underneath. The data foundation covers the warehouse or lakehouse (for example Databricks or Snowflake), operational databases, the data catalog, lineage, quality checks and classification. AI raises the bar on two things in particular: knowing what each dataset contains (so it can be classified and routed to approved models) and knowing who may see it (so permissions can travel with content into retrieval).
Tools and integration layer
When AI moves from answering questions to taking actions, it needs controlled access to enterprise systems. This layer exposes well-scoped tools, through internal APIs or Model Context Protocol (MCP) servers, with clear input schemas, least-privilege credentials and audit logging. Resist giving agents broad API access “for flexibility.” Each tool should do one thing, and high-impact tools should require confirmation.
Security and identity
Security runs through every layer. The core questions are: whose identity does each request carry, how is that identity enforced at retrieval and tool calls, where are secrets held, and how are inputs and outputs screened. The OWASP Top 10 for LLM Applications and MITRE ATLAS are useful checklists for threats such as prompt injection, sensitive information disclosure and excessive agency.
Evaluation and observability
Every production AI workload needs an evaluation set that reflects real use, automated scoring in the release pipeline, and tracing in production that links a user request to its retrieval results, model calls, tool calls and cost. Without this, model upgrades and prompt edits become guesswork.
Governance
Governance turns policy into gates: an inventory of use cases and models, a risk tier for each, required controls per tier, and a review workflow that records who approved what. Frameworks such as the NIST AI Risk Management Framework give this structure; the architecture’s job is to make the evidence automatic rather than a manual spreadsheet exercise.
The decisions that matter most
A reference architecture earns its keep by settling the questions teams would otherwise argue about on every project. These are the ones we see decide outcomes.
- Central platform or federated builders. A central team building everything becomes a bottleneck. Fully federated teams duplicate work and drift on controls. Most enterprises land on a central platform that owns shared services (gateway, retrieval, evaluation, logging) and federated teams that build applications on it.
- One model provider or several. A single provider simplifies contracts and security review. Several providers give resilience, a stronger negotiating position and access to the best model per task. If you choose several, the gateway becomes mandatory, and you need an evaluation process to qualify each model per use case.
- Where models run. Managed APIs are fastest to adopt. Self-hosting open-weight models gives more control over data and cost at high volume but adds GPU capacity planning and operations. Decide per data classification and workload, not once for the whole company.
- Vector store and search. Options range from pgvector in an existing Postgres estate to search engines with vector support to dedicated vector databases such as Pinecone. The deciding factors are usually hybrid search quality, metadata filtering for permissions, operational familiarity and scale, not benchmark speed.
- Build or buy orchestration. Frameworks speed up early work but can hide behavior you need to control. Many teams use a framework for agent control flow and plain code elsewhere. Our build vs buy guide covers the wider version of this decision.
- What gets logged, and for how long. Prompts and outputs are both valuable evidence and sensitive data. Decide retention, redaction and access rules before the logs exist.
- Data residency and processing locations. Confirm where each provider processes and stores data, and encode those constraints in the model catalog so routing respects them automatically.
Record each decision with its date, owner and review trigger (for example, “revisit if a second provider becomes necessary for a regulated workload”). Our architecture decision record template for AI gives a format for these records, with fields for model versions, evaluation evidence and cost assumptions.
Common failure modes
- Architecture without workloads. A design built in the abstract answers questions nobody asked and misses the ones real projects hit. Anchor it to two or three live use cases.
- Direct-to-provider calls. Once applications embed provider SDKs and keys, adding a gateway later means touching every application.
- Retrieval without permissions. An index that ignores source-system access controls will eventually show someone a document they should not see. This is the most common serious incident pattern we look for.
- No evaluation baseline. Teams cannot tell whether a new model version is better, so they either never upgrade or upgrade blind.
- Governance as paperwork. Review boards that rely on questionnaires rather than evidence from the platform slow delivery without reducing much risk.
- Frozen documents. A reference architecture that has not changed in a year no longer reflects what teams do. Treat it as a living set of decision records.
Rolling it out
A workable sequence for a first version:
- Inventory current AI use cases, models, providers and data sources.
- Select two or three anchor workloads with engaged business owners.
- Draft the logical view and the decisions that the anchor workloads need immediately.
- Build the minimum shared components: gateway, retrieval service, evaluation harness, logging.
- Deliver the anchor workloads on those components, and fix the architecture where it did not hold up.
- Publish the implementation view and a design review checklist tied to risk tiers.
- Onboard the next wave of teams, and review the decision log on a regular cadence.
The test of success is simple: a new team can take an AI use case to production faster by using the reference architecture than by going around it.
How we help
We design and build an enterprise AI reference architecture anchored to your real workloads, then deliver the first production systems on it with your engineers so the design is proven, not theoretical. You keep the decision records, the code and the ownership. If you want a second opinion on your current design, get in touch.
