How should teams respond when AI testing shows a model is capable but constrained?

Why This Matters for Security Teams

When testing shows a model is capable but constrained, the result is not a green light to expand access. It is evidence that the containment layer is currently doing useful work. For security teams, the real question is whether the boundary remains effective under prompt variation, tool chaining, data exposure, and operator error. That is a governance problem as much as a technical one, and it should be treated that way.

Current guidance from the NIST Cybersecurity Framework 2.0 and NIST AI risk guidance supports a control-first approach: validate that protections are still holding before anything is widened. The mistake many teams make is reading “the model can do it” as equivalent to “the system should be allowed to do it.” Those are different outcomes, because the system includes policies, tools, identity controls, logging, and human approvals.

For NHIMG, the practical lens is straightforward. If an AI system has execution authority or access to sensitive workflows, the organisation should assume that capability is a risk signal, not a feature request. That applies whether the system is a chat interface, an agentic workflow, or an embedded assistant inside a business process. In practice, many security teams encounter containment failures only after a harmless-looking evaluation result is later paired with broader tool access or production data.

How It Works in Practice

The right response is to separate model capability from permitted action and then test both layers independently. A capable model can often generate convincing instructions, code, or decisions, but the environment should decide what it may actually execute. That means the harness, policy engine, identity bindings, and approval workflow must be treated as the control plane. The evaluation result should trigger a review of guardrails, not an exception to them.

In practice, teams should verify four things. First, the model cannot bypass the intended route by changing prompt style, role framing, or multi-turn coercion. Second, tool calls are scoped to the minimum necessary permissions, with sensitive actions requiring explicit approval or step-up controls. Third, logs are complete enough to show what the model requested, what the system allowed, and which human or policy decision intervened. Fourth, fallback behaviour is safe when the model is uncertain, disconnected, or denied access.

• Confirm the model’s success criteria are separate from the environment’s authorisation rules.
• Test downgrade paths, safe refusals, and deny-by-default behaviour under adversarial prompting.
• Apply identity and privilege controls to agents the same way they are applied to other privileged software entities.
• Map the workflow to AI governance requirements in the NIST AI 600-1 Generative AI Profile so model behaviour, access, and monitoring are assessed together.

Where agentic systems are connected to enterprise tools, the containment design should also limit lateral movement between tasks. A model that can draft a message is not necessarily entitled to send it, and one that can summarise a ticket is not necessarily entitled to close it. The operating principle is least privilege for actions, not just least privilege for accounts. These controls tend to break down when a single orchestration layer is reused across low-risk and high-risk workflows because the shared permissions model eventually leaks capability into the wrong context.

Common Variations and Edge Cases

Tighter containment often increases operational friction, requiring organisations to balance speed against assurance. That tradeoff becomes especially visible when product teams want rapid iteration while security teams want proof that the model cannot escape its sandbox. There is no universal standard for this yet, so current guidance suggests treating constrained capability as a reason to refine policy thresholds rather than to loosen them.

Edge cases matter. A model that is safe in a lab can become risky when connected to live data, external APIs, or delegated human workflows. Similarly, a model that is constrained in one tenant or environment may behave differently when reused in another with broader permissions or weaker logging. The same is true when retrieval sources are expanded, because better context can improve answer quality while also increasing the blast radius of bad outputs.

Teams should be especially cautious when:

• the model is used by an AI agent with tool access and a persistent identity;
• the approval path is informal or handled outside the system of record;
• the evaluation only measures model output and not end-to-end action containment;
• the organisation assumes a red-team result means the threat is theoretical rather than operational.

Where the question intersects with agentic AI, the safe pattern is to bind capabilities to explicit governance, not trust in the model’s apparent restraint. That remains true even when the model is impressive, because capability without permission is not a failure condition. It is the expected state of a well-designed control boundary.

Related resources from NHI Mgmt Group

• How should security teams respond when AI discovers vulnerabilities faster than humans can patch them?
• How should security teams govern AI agents that use Model Context Protocol?
• How should security teams respond to faster AI-assisted vulnerability discovery?
• How should teams respond when a GitHub personal access token is exposed in an AI chat history?