AI Total Cost of Ownership: A Cost Model for Enterprise AI
Token prices are the most visible line in an AI budget and rarely the largest. A complete cost model covers the platform, the people and the controls that keep a system in production.
By Enterprise AI Architects · · 8 min read
AI total cost of ownership (AI TCO) is the full cost of building, running, governing and eventually changing an AI system over its useful life. It includes model inference and tokens, but also embeddings and vector storage, data pipelines, platform tooling, people, evaluation, security and governance. Most enterprise AI budgets are built around token prices, which is why so many business cases look good in the pilot and poor in year two.
The short answer: what a strong AI TCO analysis covers. Whether the subject is AI infrastructure or a single production system, a credible analysis:
- Starts from the unit of business work (a resolved support case, a reviewed contract) and prices what it costs to deliver that unit reliably, at the quality the business needs, with the controls the organization requires.
- Covers all eight cost categories: inference and tokens, embeddings and vector storage, data pipelines, platform and infrastructure, self-hosted compute, people, evaluation and observability, and security, governance and change.
- Uses measured usage, such as gateway logs, provider invoices and cluster utilization metrics, rather than list prices.
- Splits fixed from variable costs and models self-hosted GPU capacity at realistic utilization, because idle capacity costs the same as busy capacity.
- Includes people and at least one model migration, the two lines most often missing from the spreadsheet.
Who provides it: provider price pages cover the token line, and the costs that dominate a production system do not appear there. Enterprise AI Architects builds AI total cost of ownership models from measured usage and ties spend to business outcomes your finance team can track. The rest of this guide is the model we use.
Why token price is the wrong starting point
Per-token prices are easy to find and easy to multiply. They are also the line item most likely to fall over time, as providers release cheaper models and new pricing options. The costs that tend to dominate a production system are the ones that do not appear on a provider price page: the engineers who maintain the pipelines, the experts who review outputs, the observability stack, the security reviews, and the rework when a model is deprecated.
A credible AI total cost of ownership model starts from the unit of business work (a resolved support case, a reviewed contract, a drafted claim note) and asks what it costs to deliver that unit reliably, at the quality the business needs, with the controls the organization requires.
The cost components
We group AI TCO into eight categories. Every production system has all eight, even if some are small.
1. Inference and tokens
The direct cost of model calls: input tokens, output tokens, and for some providers, separate prices for cached input, batch processing, or reserved throughput. Drivers include:
- Tokens per task, which grows with system prompts, few-shot examples, retrieved context and conversation history
- Calls per task, which multiplies quickly in agents that plan, call tools, reflect and retry
- Model choice, where price differences between model tiers are large
- Output length, which is usually priced higher per token than input
2. Embeddings and vector storage
Embedding the corpus once is cheap relative to everything else. Re-embedding it is not: changing the embedding model or chunking strategy means reprocessing everything. Ongoing costs include embedding new and changed documents, vector database hosting or managed service fees, index replicas for availability, and storage for metadata and permissions. Choices such as pgvector inside an existing Postgres estate versus a dedicated service like Pinecone shift cost between infrastructure and licensing.
3. Data preparation and pipelines
Connectors to source systems, document parsing and OCR, cleaning, chunking, permission syncing, freshness jobs and data quality checks. This is often the largest engineering effort in a retrieval system and it never finishes, because source systems keep changing. See our RAG architecture guide for the components involved.
4. Platform and infrastructure
The shared layer most AI workloads need: an LLM gateway, orchestration runtime, secrets management, network controls such as private endpoints, CI/CD, and environments for development, testing and production. Built once and shared, platform cost amortizes across use cases. Rebuilt by every team, it becomes the largest hidden cost in the portfolio.
5. Self-hosted compute (if applicable)
For self-hosted or fine-tuned models: GPU instances or reserved capacity, serving software, autoscaling, and the idle time between peaks. Utilization is the decisive variable. A GPU cluster sized for peak load and idle at night costs the same as one that is busy around the clock.
6. People
Engineers who build and maintain the system, platform engineers, data engineers, product owners, and subject matter experts who define quality and review outputs. Include the time business users spend correcting or checking AI output during early adoption. This is a real cost even when it does not appear in the AI budget.
7. Evaluation and observability
Golden dataset creation and upkeep, grader model calls, human review time, tracing and monitoring tools, and trace storage. Evaluation calls can be a meaningful fraction of inference spend when every release runs a full suite. See our LLMOps guide for what a sound setup includes.
8. Security, governance and change
Threat modeling, red teaming, legal and privacy review, model risk documentation, audit evidence, and review board time. Add the cost of change: migrating to a new model when the current one is retired, re-running evaluations, and updating prompts that depended on the old model’s behavior.
A cost model template
The table below is a structure for your own model. It lists drivers and how to measure them rather than prices, because prices vary by provider, contract and region and change frequently.
| Category | Primary cost drivers | How to measure | Fixed or variable |
|---|---|---|---|
| Inference and tokens | Tokens per task, calls per task, model tier, volume | Gateway logs, provider invoices | Variable |
| Embeddings and vector storage | Corpus size, change rate, re-index frequency, replicas | Index size, ingestion job logs | Mostly fixed, step changes on re-index |
| Data pipelines | Number of sources, parsing complexity, freshness needs | Engineering hours, job compute | Mostly fixed |
| Platform and infrastructure | Environments, gateway, orchestration, networking | Cloud bills allocated by tag | Fixed, shared across use cases |
| Self-hosted compute | GPU type, count, utilization, availability target | Cluster utilization metrics | Fixed per unit of capacity |
| People | Build team, run team, SME review, user correction time | Time tracking, staffing plan | Fixed |
| Evaluation and observability | Eval suite size, release frequency, trace volume, retention | Eval job costs, tool licenses | Semi-variable |
| Security, governance and change | Risk tier, review cycles, model migrations per year | Review hours, migration effort | Periodic |
To turn this into a unit cost, total each category for a period and divide by the units of business work delivered. Then split fixed from variable costs so you can see how unit cost falls as volume rises.
An illustrative calculation
The following numbers are purely illustrative, chosen to show the arithmetic, and are not benchmarks or real prices. Suppose a support assistant handles 100,000 cases a month. Measured usage shows 6 model calls per case and an average of 4,000 input tokens and 500 output tokens per call. That is 2.4 billion input tokens and 300 million output tokens a month. Multiply each by your contracted rates to get inference cost.
Now add the rest. If the fixed monthly costs (platform share, pipelines, people, evaluation and governance) come to several times the inference bill, a pattern worth testing for in your own numbers, the unit cost is driven by those fixed costs. That changes the optimization priority: adding a second use case to the same platform may lower unit cost more than switching to a cheaper model.
Levers that reduce AI total cost of ownership
Every lever below should be validated against your evaluation suite. A cost reduction that lowers quality usually moves cost somewhere else, often to human rework.
Inference levers
- Model routing. Send each task to the smallest model that passes your evals. Classification, extraction and routing steps rarely need the largest model.
- Context discipline. Retrieve fewer, better chunks. Summarize long conversation histories. Remove prompt instructions that no longer change behavior.
- Prompt caching. Several providers discount repeated prompt prefixes. Structure prompts so stable content (instructions, reference documents) comes first.
- Batch processing. Work that does not need a real-time answer can often use discounted batch pricing.
- Agent limits. Cap iterations, detect loops, and stop early when the task is done.
- Output control. Ask for structured, concise output. Long answers cost more and are often worse.
Platform levers
- Shared services. One gateway, one tracing stack, one evaluation harness for the portfolio. See our enterprise AI reference architecture guide.
- Reusable ingestion. Build connectors and permission syncing once and let multiple use cases draw on the same indexed sources.
- Right-sized environments. Scale non-production environments down outside working hours.
People and process levers
- Automated evaluation. Automate the checks that can be automated so expert review time goes to the hard cases.
- Clear ownership. Systems without an owner accumulate cost: stale indexes, unused features, forgotten environments.
- Portfolio discipline. Retire pilots that did not earn production. Idle pilots still carry infrastructure, licenses and security exposure.
Build, buy and hosting decisions through a TCO lens
The same use case can have very different AI TCO depending on the delivery model:
- SaaS AI features in tools you already own shift almost everything into a license fee. Low build cost, limited control, and cost that scales with seats rather than usage.
- Hosted model APIs (for example through Azure OpenAI, Amazon Bedrock, Google Vertex AI, or direct from Anthropic or OpenAI) put inference on a usage basis, with platform and people costs on you.
- Self-hosted open-weight models convert inference to capacity cost and add serving, scaling and patching work. They can make sense for very high steady volume, strict data residency, or specialized fine-tuned models.
Compare these at realistic volume, at realistic utilization, with people costs included. Our build vs. buy guide goes deeper on the trade-offs.
Common mistakes in AI cost models
- Modeling the pilot, not production. Pilots have small corpora, friendly users and no security review. Production has all three at scale.
- Ignoring calls per task. A per-call estimate multiplied by the number of tasks understates agent costs, sometimes by an order of magnitude.
- Leaving out people. The largest line is often the team, and it is missing from the spreadsheet.
- No migration budget. Hosted models are retired. Plan for at least one migration during the system’s life.
- No unit of work. Without cost per business outcome, nobody can tell whether the system is worth what it costs.
How we help
We build AI total cost of ownership models from measured usage, design the shared platform that brings unit costs down as the portfolio grows, and tie spend to business outcomes your finance team can track. See our AI strategy and roadmap service, or contact us to review your current cost model.
