Model-level alignment alone breaks once the system can act. A well-behaved model may still trigger unsafe outcomes if the agent has broad permissions, hidden context, or long task chains. The failure is in assuming safe language output equals safe execution. In practice, safety must cover tools, scope, memory, and downstream actions, not just the model response.
Why model-level alignment stops being enough once an agent can act
Model-level alignment can reduce unsafe completions, but it does not by itself constrain what happens after a model emits a plausible answer. The moment the system can call tools, chain steps, retain memory, or operate with delegated permissions, the safety boundary moves from text generation to execution. That is why agent safety has to treat permissions, action scope, state, and downstream effects as first-class controls. The OWASP Agentic AI Top 10 is useful here because it frames agent risk around autonomy, tool use, and trust boundaries rather than model output alone.
Practitioners often underestimate how quickly a safe-sounding response can become an unsafe workflow when the surrounding orchestration layer is trusted too much.
What actually breaks in the agent architecture
The breakdown is not usually that the model becomes malicious. The breakdown is that the surrounding system treats the model response as a sufficient safety control even when the model has access to actions, data, or credentials that can cause real-world change. In practice, the agent may receive hidden instructions, inherit context that the user cannot see, or execute a sequence that was individually reasonable but collectively unsafe. That means the right question is not only, “Did the model answer safely?” but also, “Did the agent have the authority to do anything harmful with that answer?”
- Broad tool access can turn a minor reasoning error into a harmful execution path.
- Hidden memory or system context can create behaviour the user never reviewed.
- Long task chains can accumulate risk even when each intermediate step appears benign.
- Delegated permissions can let an agent overreach without any explicit unsafe wording.
The control problem is therefore cross-layer: the model may be aligned, but the policy surface around tools, scope, and approvals may still be weak. The NIST AI Risk Management Framework is relevant when teams need a governance lens for mapping those risks to measurable controls, while agent-focused threat modelling is better suited to the execution layer. Where this guidance breaks down is in highly constrained systems where the model cannot act, cannot retain state, and cannot reach external tools, because then model-level alignment carries much more of the safety burden.
Where the edge cases and failure modes show up first
Tighter agent controls often reduce convenience and automation depth, requiring organisations to balance workflow speed against the cost of more explicit approval, narrower permissions, and more visible state. That tradeoff becomes sharp in environments that blend retrieval, tool execution, and persistent memory, because the agent can appear reliable during ordinary use while failing under prompt injection, ambiguous instructions, or unusual task composition. The MITRE ATLAS adversarial AI threat matrix helps when the issue is how an adversary manipulates the model or its surrounding pipeline, while the CSA MAESTRO agentic AI threat modeling framework is useful where teams need to reason about threats across orchestration, actions, and dependency chains.
One important edge case is that better model alignment can hide system weakness by making unsafe flows look socially acceptable rather than obviously broken. Another is that long-running agents may drift because the control failure is temporal, not linguistic: the model may start within policy and still end outside it after context growth, stale assumptions, or repeated tool calls. The boundary that breaks first is usually the assumption that one approved response equals one safe outcome.
Risk and Threat Considerations
The material risk is control displacement: organisations mistakenly place safety responsibility on the model when the harmful behaviour occurs in orchestration, tool invocation, permission scope, or memory handling. That creates exposure to prompt injection, overbroad delegation, and unsafe action chaining even when the model’s visible output appears compliant.
Failure mechanism: An attacker or benign user input can steer the agent through trusted context, tool calls, or chained tasks so that the model’s aligned language output triggers unauthorised, excessive, or irreversible actions. The recognised mechanism is trust abuse across the model-to-action boundary.
Impact: Sensitive data can be exposed, actions can be executed outside intended scope, and the organisation can lose meaningful control over what the agent does versus what it merely says.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A2 — Excessive Agency | Directly addresses unsafe actions from over-delegated agent authority. |
| Recommendation — Limit delegated actions so a safe model response cannot trigger unsafe execution. | ||
| NIST AI RMF | GOVERN — Govern | Covers governance of AI risk across model and deployment layers. |
| Recommendation — Assign accountability for agent safety across model, tools, and operations. | ||
| MITRE ATLAS | ATLAS — Adversarial AI Threat Matrix | Maps adversarial manipulation of AI systems and their pipelines. |
| Recommendation — Map prompt-injection and control-abuse scenarios to ATT&CK-style AI threat patterns. | ||
| CSA MAESTRO | THREAT — Threat Modeling | Focuses on threat modelling across agent orchestration and actions. |
| Recommendation — Model agent workflows to expose tool, memory, and approval failure paths. | ||
| CIS Controls v8 | 6 — Access Control Management | Addresses overbroad access paths that let agent output become harmful action. |
| Recommendation — Restrict and review permissions so agent actions stay within intended scope. | ||
Practitioner Guidance
What to verify: Confirm that the agent’s effective authority is narrower than the model’s conversational capability. The useful test is whether a safe response can still produce a harmful action through tools, memory, or delegated credentials.
What good looks like: The system makes unsafe execution difficult even when the model is imperfect, and it makes high-impact steps visible enough for review. If safety only exists in the generated text, the control is not at the right layer.
Practitioner takeaway: Treat model alignment as a quality signal, not a safety boundary; agent safety is only credible when the execution environment is constrained as carefully as the model is trained.
Related resources from NHI Mgmt Group
- What breaks when model-level prompt defenses are used as the only control?
- What breaks when a model is used to judge another model in an agent safety stack?
- What breaks when access reviews are used as the main risk control?
- What breaks when OAuth is used as the only control for agent-to-agent delegation?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 6, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
