AI Agent Governance: Controls for Agents That Act on Your Systems

AI agent governance is the set of policies, roles and technical controls that decide which AI agents your organization runs, what each one is allowed to do, whose authority it acts with and how its actions are recorded and reviewed. It applies to any system where a model chooses and executes steps through tools, not only systems that draft text. This guide sets out the seven control areas, an autonomy tiering model, the roles involved and a practical way to start.

Why agents need more than model governance

Most AI governance programs were designed around outputs. A model produced a score, a summary or an answer, and a person decided what to do with it. The controls followed that shape: review the use case, test the output, monitor for drift.

Agents break the assumption that a person stands between the model and the consequence. Four things change.

• The system acts. An agent calls APIs, updates records, sends messages and starts workflows. An error is no longer a wrong sentence. It is a wrong transaction.
• Authority is delegated. Every action runs with someone’s permissions. If nobody decided whose, the agent usually ends up with more access than any single employee.
• Behavior is assembled at run time. The path an agent takes depends on the request, the data it retrieves and the tools on offer. You cannot review every path in advance.
• Untrusted content can steer it. A document, email or web page read during a task can contain instructions. The OWASP Top 10 for LLM Applications names this combination of injected instructions and excessive agency among its core risks.

None of this replaces the wider program. An agent is still an AI system in your inventory, still subject to the risk tiers and reviews described in our guide to building an AI governance framework. Agent governance is the additional layer for systems that act.

The seven control areas

A workable AI agent governance model covers seven areas. Each has a policy decision and a place in the platform where that decision is enforced.

Control area	The decision	Where it is enforced
Registry and ownership	Which agents exist and who answers for each	Agent registry, tied to deployment
Autonomy tiering	How much each agent may do without a person	Tier recorded in the registry, checked at release
Identity and delegation	Whose authority the agent acts with	Directory, token service, policy engine
Tool governance	Which tools and data an agent may reach	Tool catalog, gateway, allowlists
Human oversight	Which actions need approval and how to stop an agent	Orchestration layer, approval queues, kill switch
Evidence	How actions are logged, evaluated and reviewed	Tracing, evaluation pipeline, audit store
Lifecycle	What triggers re-review and how agents are retired	Change management, release gates

The rest of this guide takes each in turn.

The agent registry

You cannot govern agents you cannot list. The registry is the system of record for every agent in production or pilot, including those built in low-code tools and those that arrive inside vendor products.

A useful registry entry records:

1. A unique agent ID and a plain-language description of its purpose.
2. The accountable business owner and the technical owner.
3. The autonomy tier and the risk tier of the use case.
4. The agent’s identity in the directory and the credentials it holds.
5. The tools and data sources it is permitted to use, with read or write marked for each.
6. The models, prompts and versions currently deployed.
7. The approval rules and spending or volume limits that apply.
8. The date of the last review and the conditions that trigger the next one.

Our AI agent registry template expands this list into a full field set with a worked example.

Make registration a condition of deployment, not a form filled in afterward. If the gateway only issues credentials and tool access to registered agent IDs, the registry stays accurate without anyone policing it.

Shadow agents deserve a plan of their own. Business teams will build agents in the productivity and automation tools they already have. A short registration path with a fast answer for low-tier agents brings more of them into view than a ban does.

Autonomy tiers

Risk tiering for AI use cases asks how much harm a wrong output could cause. Agents add a second question: how much can the system do before a person sees it? Combine the two and you get a small number of tiers that decide the review path and the runtime controls.

Tier	What the agent may do	Typical controls	Review path
1. Assist	Read, retrieve, draft. A person takes every action	Read-only tools, source citations, standard logging	Standard path, self-service
2. Act with approval	Propose actions that a person approves one at a time	Approval queue, full context shown to the approver	Standard path plus owner sign-off
3. Act within limits	Execute reversible actions inside defined limits without approval	Value and volume caps, allowlisted tools, sampling review, alerts	Architecture and risk review
4. Act on consequential matters	Execute actions that are hard to reverse or affect people’s rights, money or access	Dual control, pre-deployment red teaming, continuous monitoring, rehearsed shutdown	Governance board decision

Three rules keep the tiers honest.

First, the tier follows the most consequential action the agent can take, not the action it usually takes. An agent that mostly reads but holds one write tool to the payments system is not a tier 1 agent.

Second, agents earn autonomy. Start a new agent one tier below where you expect it to settle, and promote it on evidence: a period of approved actions with a low correction rate.

Third, tier 4 should be rare. If many agents land there, the design is asking the model to carry decisions that belong in deterministic code or with a person. Our guide to agentic AI architecture patterns starts from the same principle: use the least agency that does the job.

Identity and delegated authority

Every agent action is taken by some identity. Governance decides which one, and the platform makes it true.

Two cases matter. When an agent works on behalf of a signed-in user, it should carry that user’s entitlements downstream, so retrieval and tool calls return only what the person could see or do themselves. When an agent runs unattended, on a schedule or in response to an event, it needs its own service identity with permissions scoped to the task.

In both cases:

• Give each agent a distinct identity. Shared service accounts make logs useless and revocation risky.
• Issue short-lived, scoped credentials. Long-lived keys embedded in agent configuration are hard to rotate and easy to leak.
• Enforce authorization outside the model. The prompt can describe what the agent should do. Only a policy check at the tool boundary can decide what it may do.
• Record both identities when an agent acts for a user: the agent and the person.

Multi-agent systems need one more rule. When one agent hands a task to another, authority must not grow along the chain. The receiving agent gets the narrower of the two sets of permissions.

Tool governance

An agent’s real capability is the set of tools it can call. Governing tools is therefore the most direct control you have.

Keep a catalog of approved tools, each with an owner, a description the model reads, a classification (read, reversible write, irreversible write) and the data sensitivity it touches. Agents receive tools from the catalog by allowlist, never by discovering whatever is reachable.

Vet tools the way you vet software dependencies. A tool description is text the model trusts, so a compromised or careless tool can redirect an agent. This applies with force to third-party connectors, including servers built on the Model Context Protocol. Our guide to MCP for the enterprise covers server vetting and the gateway pattern, and the LLM security threat model covers the attack paths.

Set limits at the tool, not in the prompt: maximum transaction value, records per call, calls per hour, and spend per day. Limits enforced in code hold when a model misreads its instructions.

Human oversight that works

Human oversight is a requirement in most frameworks, and for high-risk systems under the EU AI Act it is a legal obligation. It fails in practice in two ways: approvals on everything, which trains people to click through, or approvals on nothing.

Design it deliberately.

• Place approvals by consequence. Irreversible actions, money movement, access changes, external communication and decisions about individuals get an approval. Routine reads do not.
• Show the reviewer what they need. The proposed action, the inputs that led to it, what the agent considered and what happens if they decline.
• Measure the reviewers. Approval rates near 100 percent with review times of a few seconds mean the control exists on paper only.
• Sample what runs without approval. A regular review of a sample of autonomous actions catches drift that alerts miss.
• Build the stop. Every agent needs a way to be paused immediately: disable its identity, revoke its tool access or halt the orchestrator. Test it before you need it.

Evidence: logs, evaluations and incidents

Governance that cannot produce evidence is an opinion. For agents, the evidence has three parts.

A trace of every run. The request, the plan, each tool call with its parameters and result, the identities involved, any approval and the final outcome. Store it where the agent cannot alter it, with retention set by the records rules for the business process.

Evaluations before and after release. Test the agent against a suite that covers correct task completion, refusal of out-of-scope requests, behavior under injected instructions and adherence to limits. Run the suite on every change and on a schedule in production.

An incident path. Define what counts as an agent incident (an unauthorized action, a limit breach, a data exposure, a harmful output acted upon), who is paged and how the agent is contained. Feed each incident back into the evaluation suite as a new test.

Lifecycle: change and retirement

An agent approved in March is a different system by September if its model, prompts or tools have changed. State which changes reopen the review.

Change	Action
New write tool or new data source	Re-tier and repeat the review for the new tier
Model or major prompt change	Run the full evaluation suite before release
Higher limits or removal of an approval step	Owner and risk sign-off, with supporting evidence
New user population or business process	Treat as a new use case
No named owner, or no use for a set period	Suspend, then retire and revoke credentials

Retirement matters as much as approval. An unowned agent with live credentials is a standing risk. Revoke its identity, archive its traces and close its registry entry.

Mapping to the frameworks you already use

Agent governance does not need a new framework. It fits the ones most enterprises have adopted.

• NIST AI RMF 1.0. Govern covers the policy, roles and tiers. Map covers the registry and the context of each agent. Measure covers evaluations and monitoring. Manage covers limits, approvals, incidents and retirement.
• ISO/IEC 42001:2023. The registry, impact assessments, operational controls and change management described here become records and procedures within the AI management system.
• EU AI Act. Where an agent forms part of a high-risk system, the obligations on human oversight, logging and risk management apply to it. Check current timelines and guidance, since both have been subject to change.

Where to start

A first version does not take a year.

1. Inventory the agents you have, including pilots and those inside vendor products.
2. Assign an owner and an autonomy tier to each.
3. Give every agent its own identity and remove shared or long-lived credentials.
4. Put tool access behind an allowlist and set limits on write tools.
5. Add approvals for consequential actions and test the stop mechanism.
6. Turn on full tracing and agree what counts as an incident.
7. Publish the tiers and the review path so new agents arrive already registered.

How we help

We design agent governance as part of the platform, so the registry, identities, tool controls and approval steps are enforced by the architecture and not left to a policy document. That work sits across our AI governance and risk and agentic AI architecture services. If you are putting agents into production and want the controls in place first, talk to an architect.