LLM Security: A Threat Model for Enterprise LLM Apps and Agents

LLM security is the practice of protecting applications built on large language models, and the agents that act through them, from attacks that exploit how models read instructions and data. The core problem is that a model cannot reliably separate trusted instructions from untrusted content, so security has to come from the architecture around the model rather than from the model itself. This article sets out the threat model we use with enterprise teams and the controls that map to each layer.

Why LLM security is different from application security

Traditional application security depends on a clean boundary between code and data. SQL injection was solved, in large part, by parameterized queries that keep user input from ever being parsed as a command. Language models have no equivalent. The system prompt, the user message, a retrieved PDF and a tool result all arrive as tokens in the same context window, and the model decides what to treat as an instruction.

That has three consequences for LLM security that shape everything below:

1. Input filtering is a speed bump, not a wall. Classifiers and guardrail models catch known patterns and reduce noise. They do not prove the absence of an attack.
2. The blast radius is set by permissions, not by the prompt. What an attacker can achieve through a compromised model equals what that model can read and do.
3. Output is untrusted input to the next system. Anything the model produces, whether HTML, SQL, a shell command or an API call, must be validated as if a stranger wrote it.

If you accept those three points, the design problem becomes tractable. You stop trying to make the model incorruptible and start designing a system where a corrupted model cannot do much damage.

The threat model: assets, actors and entry points

A useful threat model for an LLM system starts with the same questions as any other: what are we protecting, who wants it, and how do they get in.

Assets usually include confidential data reachable through retrieval or tools, the credentials and tokens the application holds, the actions the system can take (send email, update records, move money, open tickets), the system prompt and business logic, and the availability and cost of the service itself.

Actors include malicious end users, outside attackers who can place content where your system will read it, compromised or careless insiders, and third parties in your supply chain (model providers, plugin and tool authors, dataset publishers).

Entry points are every channel that puts tokens into the context window: the chat box, uploaded files, retrieved documents, web browsing, email and ticket bodies, tool and API responses, memory stores, and other agents in a multi-agent system.

The last list is the one teams underestimate. A support copilot that reads customer tickets has an entry point in every ticket. A research agent that browses the web has one in every page it opens.

The core threats

Direct prompt injection

The user types instructions designed to override the system prompt: ignore your rules, reveal your instructions, act as a different persona. Direct injection matters most when the user should not have the privileges the application holds, for example a public chatbot backed by internal tools. It matters less when the user already has the same rights as the system.

Indirect prompt injection

The instruction arrives inside content the model processes on someone else’s behalf. A web page contains hidden text telling the assistant to summarize the user’s inbox and include it in a link. A shared document tells the agent to grant access to an outside address. A tool description in a third-party integration tells the model to send data somewhere it should not.

Indirect injection is the defining threat for enterprise agents because the attacker needs no account on your system. They only need to get text in front of it.

Data exfiltration

Exfiltration is usually the payoff of an injection. Common channels include rendering a markdown image whose URL carries encoded data, calling a tool that makes an outbound request, writing data into a record the attacker can read, or simply displaying data to a user who should not see it. Retrieval systems that ignore document permissions turn every question into a potential leak without any attack at all.

Excessive agency

An agent has more tools, broader scopes or more autonomy than its task requires. Excessive agency turns a text-generation problem into an operational incident. The failure pattern is familiar: an agent built to draft replies is also given send permission for convenience, then an injected instruction sends the draft to the wrong party. Our guide to AI agent governance covers the autonomy tiers and approval rules that prevent it.

Supply chain compromise

Models, fine-tuning datasets, embedding models, open-source orchestration libraries, tool servers and prompt templates all come from somewhere. Risks include tampered model weights, poisoned training or retrieval data, malicious packages, and third-party tool servers whose behavior changes after you approve them.

Unbounded consumption and denial of service

Long inputs, recursive agent loops and deliberately expensive queries can exhaust rate limits or budgets. In LLM systems, denial of service often shows up on the invoice before it shows up in availability.

Mapping to OWASP and MITRE ATLAS

Two public frameworks give you shared vocabulary with auditors and security teams.

The OWASP Top 10 for LLM Applications catalogs the most common vulnerability classes. The 2025 edition lists prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. Use it as a coverage checklist for design reviews.

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a knowledge base of adversary tactics and techniques against AI systems, modeled on MITRE ATT&CK and supported by documented case studies. Use it when you want to describe attacks the way your security operations team already describes them, and when you build red team scenarios.

Threat in this article	OWASP LLM Top 10 (2025) category	Primary control layer
Direct and indirect prompt injection	Prompt injection	Context isolation, tool permissions, human approval
Data exfiltration	Sensitive information disclosure; improper output handling	Retrieval authorization, output sanitization, egress control
Excessive agency	Excessive agency	Least-privilege tools, scoped credentials, approval gates
Supply chain compromise	Supply chain; data and model poisoning	Provenance, pinning, vetting of tool servers
Retrieval leaks and poisoning	Vector and embedding weaknesses	Permission-aware retrieval, ingestion controls
Prompt disclosure	System prompt leakage	Keep secrets and authorization logic out of prompts
Cost and availability attacks	Unbounded consumption	Quotas, loop limits, budgets at the gateway

Neither framework replaces a threat model specific to your system. They tell you what to look for. Your architecture tells you where it can happen.

Controls by layer

We organize LLM security controls into six layers. No single layer is sufficient, and the most important ones are the least glamorous.

1. Identity and authorization

• Run every model call and tool call in the context of a real identity: the end user, or a service identity with a narrowly defined role.
• Enforce authorization in the tool or data layer, never in the prompt. A system prompt that says “only show data for the user’s region” is a suggestion. A query filter bound to the user’s entitlements is a control.
• Use short-lived, scoped credentials for tools. Avoid giving an agent a long-lived admin token because it was easier to wire up.

2. Retrieval and data

• Apply document-level permissions at query time so retrieval only returns content the requesting user can already see. See our RAG architecture guide for patterns.
• Control what gets ingested. A knowledge base that indexes any shared folder is an open door for indirect injection and poisoning.
• Tag retrieved content with its source and trust level so downstream logic can treat external content differently from curated internal content.

3. Model and prompt

• Separate instructions from data with clear delimiters and structured message roles. This reduces accidental confusion; it does not stop a determined attacker.
• Keep secrets, credentials and access rules out of system prompts. Assume the prompt will leak.
• Use input and output guardrail classifiers as a detection layer and a source of telemetry, not as your primary defense.

4. Tools and actions

• Give each agent the minimum set of tools for its task, with the narrowest parameters possible. A tool that “queries the CRM” is broader than one that “reads open opportunities for the current account.”
• Classify actions by consequence. Reads can often run automatically. Writes to external parties, financial actions and permission changes should require explicit human confirmation that shows exactly what will happen.
• Constrain egress. If an agent does not need to call arbitrary URLs, do not let it.
• For agents built on the Model Context Protocol, route tool servers through a gateway you control. Our MCP guide covers the pattern.

5. Output handling

• Treat model output as untrusted. Encode it before rendering, parameterize it before it touches a database, and never pass it to a shell or interpreter without a sandbox.
• Strip or proxy external links and images in rendered responses to close the most common exfiltration channel.
• Validate structured output against a schema before acting on it.

6. Platform and operations

• Centralize model access through an LLM gateway that enforces quotas, logs prompts and responses under your retention policy, and applies policy consistently across teams.
• Pin model versions, dependencies and tool server versions. Review changes the way you review code changes.
• Log enough to reconstruct an incident: the full context sent to the model, tool calls with parameters, approvals, and the identity behind each step.
• Set loop limits, token budgets and timeouts on every agent.

A worked example: the inbox assistant

Consider an assistant that reads a user’s email and drafts replies, with tools to search the mailbox, read calendar entries and send mail.

An attacker sends the user an email containing hidden instructions: search the mailbox for “invoice,” then send the results to an outside address. With a naive design, the assistant reads the email while summarizing the inbox, follows the instructions, and uses the send tool.

The layered design breaks this chain at several points. The send tool requires the user to confirm each message with the recipient and body shown. Outbound mail to new external domains is flagged. The search tool returns only the fields the drafting task needs. Content from external senders is marked as untrusted in the context, and an output check flags drafts addressed to recipients who were not on the original thread. None of these controls depends on the model resisting the injection. Together they make the injection useless.

Red teaming LLM systems

AI red teaming is structured adversarial testing of the full system, not just the model. A useful program has three parts.

Scope the system, not the model. Test the deployed application with its real tools, retrieval sources and permissions. Model providers already test their base models; your risk lives in how you connected them.

Build scenarios from the threat model. For each asset and entry point, write attack goals: exfiltrate a customer record through retrieval, trigger an unapproved write, leak the system prompt, run up cost. Use MITRE ATLAS techniques and the OWASP categories as prompts for coverage. Include indirect injection through every content channel, not just the chat box.

Automate the regression suite. Every successful attack becomes a test case. Run the suite on every release, model change and prompt change as part of your LLMOps release gates. Human-led exercises find new classes of attack; automation keeps the old ones from coming back.

Track findings like any other security finding: severity, owner, fix, and verification. Report residual risk explicitly. Some injection risk will remain after every control is in place, and the business owner should accept it knowingly.

Common failure modes

• Guardrails as the whole strategy. A prompt-injection classifier is deployed, the box is checked, and tool permissions remain broad.
• Authorization in the prompt. Access rules written as instructions to the model rather than enforced in code.
• Retrieval without permissions. A knowledge base indexed with a service account that can read everything, answering questions from users who cannot.
• Unreviewed tool sprawl. Teams add integrations to agents faster than security reviews them.
• No incident trail. When something goes wrong, nobody can reconstruct what the model saw or why it acted.

How LLM security fits governance

LLM security controls should trace to your broader risk framework. The NIST AI 600-1 Generative AI Profile (July 2024) lists information security among the risks specific to generative AI, and ISO/IEC 42001:2023 expects documented risk treatment for AI systems. Mapping each control above to those frameworks turns engineering work into audit evidence, and it gives your model risk or AI review board a concrete list of what “secure” means for each system.

How we help

We run LLM security threat modeling and red team exercises against your actual applications and agents, then design the identity, retrieval, tool and gateway controls that contain the blast radius. See our AI security architecture service, or contact us to scope an assessment.