What is the difference between MLOps and LLMOps?

LLMOps is MLOps adapted to applications built on large language models. MLOps centers on training pipelines, feature stores and model retraining. LLMOps centers on things most teams never train: prompts, retrieval pipelines, tool definitions and third-party model versions. Evaluation is also harder because outputs are open-ended text, so LLMOps relies more on rubric-based grading, human review and tracing of multi-step agent behavior.

How do you evaluate an LLM application?

Evaluate it against a curated dataset of real tasks with defined criteria for a good answer. Use deterministic checks where possible (format, citations present, correct tool called), model-graded rubrics for qualities like faithfulness and completeness, and periodic human review to calibrate the graders. Run the same suite before every release and sample production traffic for ongoing evaluation, since offline scores alone miss real-world drift.

Can you trust an LLM to grade another LLM's output?

Only after you calibrate it. Model-graded evaluation scales well and is useful, but graders have biases, such as favoring longer answers or their own style. Validate each grader against a sample of human-labeled examples, measure how often they agree, and re-check after changing the grader model or rubric. Use narrow, specific rubric questions rather than one overall quality score.

What should an AI observability tool capture?

At minimum it should capture a trace per request: the full prompt and context sent to the model, the response, every retrieval and tool call with inputs and outputs, model and prompt versions, token counts, latency per step, cost, and the user or session identity. It should also link user feedback and evaluation scores to those traces so you can find failure patterns, subject to your data retention and privacy rules.

LLMOps: Evals, Tracing and AI Observability

LLMOps is the set of practices, tools and controls for running applications built on large language models in production: evaluating quality, tracing behavior, managing prompt and model versions, monitoring cost and latency, and gating releases. AI observability is the telemetry half of LLMOps, the ability to see what a model saw, what it did and why. Without both, LLM applications degrade silently, because a wrong answer looks exactly like a right one to your uptime monitor.

Why LLM applications need their own operations discipline

A conventional service either returns the right data or throws an error. An LLM application can return a fluent, well-formatted, confidently wrong answer with a 200 status code. Quality is the primary failure mode, and quality is not visible in standard application metrics.

Four properties make LLMOps necessary:

Outputs are open-ended. There is rarely a single correct string to compare against, so testing needs graded criteria rather than exact matches.
Behavior depends on many moving parts. Prompt templates, retrieval indexes, chunking settings, tool definitions, model versions and provider-side changes all affect output. A change to any one of them is a release.
Dependencies change underneath you. Hosted model versions are deprecated and replaced on the provider’s schedule. Your source documents change daily.
Cost scales with usage and design. A prompt that grows by a few hundred tokens, or an agent that loops one extra time, changes unit economics in a way nobody approved.

The LLMOps lifecycle

We describe LLMOps as a loop with five stages. Each stage has a concrete artifact.

Stage	What happens	Key artifact
Define	Agree what “good” means for the use case	Evaluation criteria and golden dataset
Develop	Change prompts, retrieval, tools or models	Versioned configuration in source control
Evaluate	Score changes offline before release	Eval report compared against the baseline
Release	Gate, deploy and roll out gradually	Release record with eval results and approvals
Observe	Trace, monitor and sample production	Traces, dashboards, online eval scores, feedback

The loop closes when production observations become new test cases in the golden dataset.

Evaluation: offline and online

Building the golden dataset

Every serious LLM evaluation program starts with a dataset of representative tasks: real user questions (with sensitive data handled appropriately), the context or documents they relate to, and a reference answer or a description of what a good answer must contain. Include hard cases, edge cases, adversarial inputs, and questions the system should decline to answer.

Size matters less than coverage. A few hundred carefully chosen cases that span your intents and known failure modes will tell you more than thousands of near-duplicates. Assign an owner, version the dataset, and add to it every time production reveals a new failure.

Choosing evaluation methods

Use the cheapest method that measures the thing you care about.

Deterministic checks. Valid JSON, required fields present, citations point to retrieved documents, the correct tool was called with valid parameters, no banned content. These are fast, cheap and unambiguous. Use them wherever possible.
Reference-based comparison. Semantic similarity or key-fact matching against a reference answer. Useful for extraction and classification tasks.
Model-graded rubrics. A grader model scores specific qualities: is the answer supported by the retrieved context, does it address the question, does it follow the policy. Narrow questions (“Does the answer contain any claim not supported by the sources? yes or no”) are more reliable than broad ones (“Rate quality 1-10”).
Human review. Subject matter experts label a sample. Human review is the ground truth that calibrates everything else, and it is the only reliable judge for high-stakes domains.

For retrieval-augmented systems, evaluate retrieval and generation separately. If the right document never reached the model, no prompt change will fix the answer. Measure whether relevant sources were retrieved, then whether the answer was faithful to them.

For agents, evaluate trajectories as well as final outputs: did the agent choose sensible tools, in a sensible order, without unnecessary steps or unsafe actions. Our agentic AI architecture patterns article covers where those checkpoints belong.

Online evaluation

Offline evals tell you whether a change is safe to ship. Online evals tell you whether production still behaves the way the offline set predicted. Sample live traffic and run the same automated graders asynchronously, off the request path. Track scores by intent, user segment, and data source so a regression in one area is not averaged away by stability elsewhere.

Pair automated scores with implicit and explicit user signals: thumbs up or down, edits to generated drafts, copy events, abandoned sessions, escalations to a human. Each signal is noisy on its own. Trends across them are informative.

Tracing: the backbone of AI observability

A trace records one request end to end as a tree of spans: the incoming request, retrieval queries and results, each model call with its full input and output, each tool call, guardrail decisions, and the final response. For agents, a trace may contain dozens of model calls.

Capture on every span:

Model identifier and version, provider, and parameters (temperature, max tokens)
Prompt template name and version
Input and output token counts and computed cost
Latency, including time to first token for streaming responses
Retrieval details: query, documents returned, scores, index version
Tool name, arguments, result and status
User, session and tenant identifiers, subject to privacy rules

Standardize the format early. OpenTelemetry has published semantic conventions for generative AI telemetry that are still maturing; aligning to them where practical keeps you from being locked into a single observability vendor. Several dedicated LLM observability products exist, and many teams also export traces into the monitoring stack they already run.

Traces contain sensitive data: prompts carry user questions and retrieved documents. Treat the trace store as a regulated data store with access controls, redaction where needed, and a retention period your privacy and legal teams have approved.

Prompt and configuration management

Prompts are code. Treat them that way.

Store prompt templates, retrieval settings, tool definitions and model selections in version control, reviewed through the same pull request process as application code.
Give every deployed configuration a version identifier and record it on every trace, so any output can be tied to the exact configuration that produced it.
Decouple prompt releases from application deploys only if you keep the same review and eval gates. A prompt registry that lets anyone push a change to production without evaluation is a way around your controls, not a feature.
Pin model versions explicitly. Aliases that silently move to a newer model make regressions impossible to attribute.

Drift: what changes and how to catch it

LLM systems drift in several distinct ways, and each needs its own signal.

Drift type	What changes	How to detect it
Input drift	Users ask new kinds of questions	Topic and intent clustering on sampled queries
Knowledge drift	Source documents change or go stale	Index freshness metrics, retrieval hit rates
Model drift	Provider updates or deprecates a model	Pinned versions, scheduled eval runs, deprecation tracking
Behavior drift	Output quality or style shifts	Online eval scores trended over time
Cost drift	Tokens per request creep upward	Tokens and cost per request by feature and version

Deprecation tracking deserves a specific owner. Hosted model providers publish retirement dates, and a forced migration without a ready eval suite turns into an emergency.

Cost and latency monitoring

Instrument cost at the level of the business unit of work: cost per resolved ticket, per drafted document, per completed agent task. Cost per API call hides the fact that one feature makes ten calls per task.

Watch for:

Prompt growth from accumulated instructions and examples
Retrieval that stuffs too many chunks into context
Agent loops that retry or wander before finishing
Features routed to a large model when a smaller one passes the same evals
Cache hit rates, where provider prompt caching or response caching is in use

For latency, measure time to first token and total time separately, broken down by span. Users tolerate a long streamed answer far better than a long blank pause, and the fix for each is different. Our LLM gateway architecture article covers where routing, caching and budget enforcement belong.

Release gates

A release gate is a written, automated rule that decides whether a change can go to production. Gates turn evaluation from a report people read into a control that works when nobody is watching.

A practical gate for an LLM change checks, in order:

All deterministic checks pass on the golden dataset.
Model-graded scores meet or exceed the current production baseline within an agreed tolerance on each critical criterion, not just on average.
The adversarial and safety suite passes, including prompt injection regression cases. See our LLM security threat model.
Cost per task and latency stay within budget.
A named owner approves, with eval results attached to the release record.

After the gate, roll out gradually: shadow mode or a small traffic percentage first, with online eval scores and user signals compared against the control before full rollout. Keep the previous configuration deployable for fast rollback.

For high-risk use cases, the release record doubles as governance evidence. Frameworks such as the NIST AI RMF expect ongoing measurement and management of AI risk, and a versioned history of eval results and approvals is exactly that evidence.

Feedback loops that actually close

Most teams collect feedback. Few use it. A feedback loop is closed when there is a defined path from a production signal to a change:

A weekly triage reviews low-scoring traces, negative feedback and escalations.
Each confirmed failure is labeled by cause: retrieval miss, prompt gap, model limitation, missing tool, bad source data.
Failures become golden dataset cases before anyone attempts a fix.
The fix ships through the same gates as any other change.

Labeling by cause matters because the remedy differs. Teams that treat every failure as a prompt problem end up with enormous prompts and the same retrieval bugs.

Ownership and operating model

LLMOps fails when it belongs to nobody. We recommend a split: a central platform team owns the shared tooling (gateway, tracing, eval harness, prompt registry, dashboards), and each product team owns its golden dataset, criteria, gates and triage. Subject matter experts need protected time for human review. Without it, evaluation quality decays into whatever the grader model thinks is good.

How we help

We design LLMOps foundations as part of the shared platform: tracing, evaluation harnesses, prompt and model versioning, cost controls and release gates that product teams can adopt without building their own. See our generative AI platform architecture service, or contact us to discuss your current setup.

LLMOps and AI Observability: Evals, Tracing and Release Gates