How to Build an AI Center of Excellence That Ships, Not Just Advises
An AI center of excellence earns its place by making good AI work faster and safer across the business. The operating model, intake process and funding design decide whether it does that or becomes a bottleneck.
By Enterprise AI Architects · · 8 min read
An AI center of excellence (AI CoE) is a dedicated team that sets standards, runs shared AI platforms, provides scarce expertise, and governs how an organization builds and uses AI. Done well, it makes AI delivery across the enterprise faster, cheaper and safer. Done poorly, it becomes an advisory committee that reviews slides, or a central team that builds everything and becomes a queue.
The difference comes down to a handful of design choices: the operating model, the division of responsibilities between the CoE and business units, how work comes in, how architecture decisions get made, and how the whole thing is funded. This guide covers each, with the trade-offs we see in practice.
What an AI center of excellence is for
Before choosing a structure, agree what the CoE must do. Most effective AI CoEs perform some combination of these functions:
- Strategy and portfolio: maintain the AI strategy, prioritize use cases with a shared scoring framework, and track value delivered.
- Shared platform: own the model gateway, retrieval and vector services, evaluation tooling, observability, and approved model catalog.
- Standards and patterns: publish reference architectures, reusable components, prompt and agent patterns, and coding standards.
- Architecture authority: review significant AI designs and make or delegate architecture decisions.
- Governance and risk: run intake, risk tiering and release gates, in coordination with security, privacy, legal and model risk.
- Enablement: train builders and users, run communities of practice, and embed experts with delivery teams.
- Vendor management: evaluate AI vendors and models, and negotiate enterprise agreements.
A CoE that only does 5 and 7 is a control function. A CoE that only does 2 and 6 is a platform team. The ones that change outcomes combine a platform people want to use with governance that is fast for low-risk work.
Three operating models
Centralized
A single central team owns AI strategy, platform and most delivery. Business units bring problems; the CoE builds solutions.
- Strengths: concentrates scarce talent, consistent standards, easier governance, fast start.
- Weaknesses: becomes a bottleneck as demand grows, distance from business context, business units feel little ownership.
- Fits when: AI adoption is early, talent is scarce, or regulatory exposure demands tight control.
Hub-and-spoke
A central hub owns platform, standards, architecture authority and governance. Spokes, embedded in business units, deliver use cases on the shared platform. Spoke staff may report to the business with a dotted line to the hub, or the reverse.
- Strengths: scales delivery while keeping shared platforms and consistent controls; keeps builders close to the business.
- Weaknesses: requires clear decision rights and good platform products; dotted-line reporting creates tension if not managed.
- Fits when: several business units are actively delivering AI and the enterprise wants reuse and consistency without central delivery of everything.
Federated
Business units own their AI teams and delivery. A thin central function sets minimum standards, maintains a shared inventory, and coordinates.
- Strengths: maximum autonomy and speed for mature units; strong business ownership.
- Weaknesses: duplicated platforms and spend, inconsistent controls, harder to see enterprise risk.
- Fits when: the organization is highly decentralized, business units have strong engineering capability, and there is a mature governance culture.
| Dimension | Centralized | Hub-and-spoke | Federated |
|---|---|---|---|
| Who delivers use cases | Central team | Spokes on shared platform | Business unit teams |
| Who owns platform | Central | Hub | Mixed, often duplicated |
| Governance consistency | High | High if decision rights are clear | Variable |
| Scalability of delivery | Low | High | High |
| Risk of bottleneck | High | Moderate | Low |
| Typical stage | Early | Growth | Mature |
Most large enterprises we work with land on hub-and-spoke. It is also normal to evolve: start centralized to build the platform and first patterns, then move delivery out to spokes as the platform matures. Our AI operating model practice covers this design in depth.
Core roles
The exact titles vary; the capabilities do not. A hub typically needs:
- AI CoE leader: accountable for strategy, portfolio and outcomes. Often reports to the CIO, CTO, CDAO or Chief AI Officer.
- Enterprise AI architect: owns reference architectures, platform design and architecture decisions. See what is an enterprise AI architect.
- Platform engineering lead and engineers: build and operate the gateway, retrieval services, evaluation and observability tooling.
- ML and evaluation engineering: define evaluation methods, build test sets and pipelines, and help teams measure quality.
- AI governance and risk lead: runs intake, tiering and release gates; connects to model risk, privacy and security.
- AI security architect: threat models AI systems and sets security controls, often shared with the security organization.
- Product and portfolio manager: manages the use-case pipeline, value tracking and stakeholder communication.
- Enablement lead: training, documentation, community of practice.
Spokes typically include product owners, AI engineers, data engineers and domain experts, all working on the shared platform.
Where the CoE reports
Reporting lines signal what the CoE is for. A CoE under the CIO or CTO tends to emphasize platform and delivery. One under a CDO or CDAO tends to emphasize data and analytics. One under a Chief AI Officer usually has the broadest mandate. A CoE placed inside risk or compliance tends to become a control function. There is no single right answer, but the CoE leader needs a sponsor senior enough to settle disputes between business units, and a seat where architecture and platform decisions are actually made. Whatever the line, make decision rights explicit: who approves a new model provider, who can block a release, and who arbitrates when a business unit wants to go its own way.
Intake: the front door
The intake process decides whether the CoE speeds work up or slows it down. A good intake is short, proportional and fast.
A practical intake captures:
- The business problem, the owner, and the metric the use case should move.
- Users and affected people.
- Data sources and sensitivity.
- Whether the system makes or influences decisions about people, or takes autonomous actions.
- Build, buy or compose preference, with rationale.
- Target timeline and sponsor.
From these answers the intake assigns a provisional risk tier and a delivery path:
- Low tier, standard pattern: proceed on the shared platform with standard controls. No review meeting needed.
- Medium tier or new pattern: lightweight architecture and risk review.
- High tier: full architecture review, formal risk assessment, and executive sign-off before production.
Publish target turnaround times for each path and measure against them. An intake that routinely takes weeks for low-risk work will be bypassed.
The architecture review board
An AI architecture review board (ARB) makes or ratifies significant AI design decisions. It exists to prevent expensive mistakes (unsafe designs, duplicated platforms, untestable systems) and to spread good patterns. It should not review everything.
What the ARB reviews
- New use cases in the higher risk tiers
- Designs that depart from published reference patterns
- New model providers, vector stores or orchestration frameworks entering the estate
- Agentic systems that can write to systems of record
- Significant changes to the shared platform
How to run it well
- Publish the criteria for what requires review, so teams can predict it.
- Require a short design document with context, options considered, the decision, and risks, in a standard template.
- Record decisions in an architecture decision log that anyone can search.
- Time-box: a fixed weekly slot, with asynchronous review for simpler items.
- Include the right people: enterprise AI architect, security, data, risk, and the relevant platform owner. Keep it small.
- Turn repeated approvals into patterns. If the ARB approves the same design three times, publish it as a standard pattern and stop reviewing it.
The last point is what keeps the ARB from becoming a bottleneck. Its long-term output is a growing library of pre-approved patterns, grounded in an enterprise AI reference architecture. To set one up, start from our AI architecture review board charter template.
Funding models
Funding design shapes behavior more than org charts do.
| Model | How it works | Advantages | Risks |
|---|---|---|---|
| Central funding | CoE and platform funded from a central budget | Easy to start; encourages adoption | Weak cost discipline; hard to scale budget with demand |
| Showback | Central funding, but usage costs reported to each business unit | Cost visibility without friction | Visibility alone may not change behavior |
| Chargeback | Business units pay for platform usage and CoE services | Cost discipline; scales with demand | Can discourage experimentation; overhead to run |
| Project funding | Use cases funded by sponsoring business units | Strong business ownership | Platform work is chronically underfunded |
A common and sensible pattern is central funding for the core team and shared platform, project funding for use cases, and showback for model and platform consumption, moving to chargeback once usage is established and predictable. Whatever you choose, fund the platform as a product with its own roadmap. Platforms funded only as a by-product of use cases never mature.
Measuring the CoE
Measure the CoE on outcomes it can influence, not activity:
- Use cases moved from intake to production, and time taken
- Value delivered against the metrics defined at intake
- Share of AI workloads running on the shared platform
- Reuse: number of use cases built on published patterns
- Governance performance: intake turnaround by tier, incidents, open risk acceptances
- Unit cost trends for inference and platform services
Common failure modes
- Advisory without a platform. Standards nobody can implement because there is nothing to build on.
- Central delivery of everything. A growing queue and frustrated business units.
- Governance that treats all AI the same. Low-risk work waits as long as high-risk work.
- No decision rights. Nobody can say no, or everyone can.
- Platform funded by leftovers. Shared services that never get the investment to be worth using.
How we help
We design AI centers of excellence as working operating models: structure, roles, intake, architecture review and funding, together with the shared platform that makes the CoE useful to builders. Our AI operating model practice leads this work. Contact us to discuss your current structure.
