What breaks when AI runtime defense only inspects chat inputs and model outputs?

Why This Matters for Security Teams

Chat-only inspection assumes the risky moment is the prompt or the completion. For autonomous systems, that is usually the wrong boundary. The higher-value attack surface sits in tool calls, delegated actions, chained workflows, and the identity path that lets an agent reach SaaS, code, ticketing, or data systems. Once an agent can act on behalf of a user or service, a clean-looking conversation can still produce a dangerous downstream operation.

This is why front-door controls often miss indirect prompt injection, multi-step privilege escalation, and policy violations that emerge only after the model has been allowed to call tools. The threat pattern is consistent with what NHIMG documents in the LLMjacking report and the DeepSeek breach: once credentials, connectors, or execution paths are exposed, the damage happens outside the chat window. Current guidance from NIST Cybersecurity Framework 2.0 still applies, but it must be applied to runtime actions, not just user text.

In practice, many security teams encounter this only after an agent has already touched production systems, rather than through intentional pre-deployment testing.

How It Works in Practice

Effective runtime defense for AI agents needs to inspect the full action path: intent, tool selection, parameter payloads, identity context, and the result of each step. Chat input and model output are still useful signals, but they are only two points in a broader execution chain. The control point shifts to request-time authorisation and per-action enforcement, especially when an agent uses MCP, automation runbooks, or external APIs.

In practice, teams combine policy-as-code with workload identity so the runtime can decide whether a specific action is allowed in the current context. That means checking what the agent is trying to do, which tool it is invoking, what data it is requesting, and whether the request matches the approved task scope. A strong pattern is just-in-time access: issue short-lived credentials for a single task, scope them narrowly, and revoke them automatically when the task ends. This reduces the value of stolen tokens and limits lateral movement.

• Inspect tool calls and downstream API requests, not just user prompts.
• Bind each action to a workload identity rather than a human session.
• Use short-lived secrets and task-scoped permissions instead of standing access.
• Evaluate policy at runtime so the same agent can be allowed or denied differently by context.

That approach aligns with the intent of NIST Cybersecurity Framework 2.0 and the emerging agentic guidance in NHIMG research on compromised NHIs. It also reflects the operational reality highlighted in The State of Secrets in AppSec: secrets and tokens remain exploitable long after the chat session ends. These controls tend to break down when legacy integrations reuse broad service accounts because the runtime cannot distinguish a safe agent action from an attacker-driven one.

Common Variations and Edge Cases

Tighter runtime inspection often increases latency, policy complexity, and false positives, so organisations have to balance security depth against production reliability. That tradeoff becomes sharper in multi-agent systems, high-volume support workflows, and environments where tools are nested inside other tools.

Best practice is evolving, but current guidance suggests several exceptions need special handling. First, indirect prompt injection inside retrieved content can steer tool use without altering the visible chat transcript, so content filters alone are not enough. Second, some workflows require delegated approval or human-in-the-loop checkpoints, especially for irreversible actions such as deleting records or moving funds. Third, agents that chain across multiple systems may look benign at each step while still producing an unsafe end state, which means single-step allowlists can miss the full risk.

For runtime defense, the important question is not whether the text looked safe, but whether the resulting action was safe given the agent’s identity, permissions, and current objective. Where agents operate across MCP-connected workflows or other privileged integrations, runtime controls should verify every boundary crossing, not just the initial chat exchange. The DeepSeek breach is a useful reminder that exposure often shows up first in the connected systems, not in the model conversation.

Related resources from NHI Mgmt Group

• What breaks when AI governance is designed only for model outputs?
• What breaks when AI agents trust model outputs too early?
• What breaks when AI systems recombine harmless inputs into sensitive outputs?
• What breaks when AI outputs are validated without runtime monitoring?