Should organisations prioritise model testing or access governance first for AI security?

Why access governance should come before model testing

When AI security involves live access to SaaS, data warehouses, ticketing systems, or internal knowledge stores, the first question is not whether the model is clever enough to pass a test. It is whether the AI is allowed to touch more data and functions than its task requires. Excess privilege creates immediate blast radius, while model testing mainly reduces the chance that a prompt or output mishap becomes harmful.

That ordering matters because governance failures can turn an ordinary workflow into a high-impact path. If the AI can read, write, delete, forward, or summarise sensitive records, a model defect is only one of several ways harm can happen. Strong access design limits what the system can do even when prompts are manipulated, tools are misused, or an integration behaves unexpectedly.

For a broader identity and governance baseline, the most useful starting point is IAM and IGA Basics, because the same least-privilege and entitlement principles apply whether the actor is a person, service, or AI-enabled workflow.

What model testing does well, and where it stops

Model testing is still necessary. It helps uncover unsafe outputs, prompt sensitivity, policy bypass behaviour, and weak guardrails before deployment. That matters for product safety, user trust, and abuse resistance. It also helps teams understand how the model behaves under stress, which is important if the AI produces recommendations, drafts messages, or triggers downstream actions.

But testing a model does not remove the consequences of overbroad access. A well-tested model with write access to production systems can still make a damaging but “valid” request. Likewise, a model with read access to an entire customer corpus can leak far more than intended even if its natural-language output is mostly well-behaved. Testing reduces uncertainty about behaviour; it does not define or constrain authority.

That is why organisations should treat model evaluation as a control for behaviour, not as a substitute for access design. In practice, the safest pattern is to make the model’s permissions narrower than the human operator’s permissions, then prove that each tool or data connection is individually necessary.

Where the question is really about which controls to operationalise first, the first validation point is whether the AI can authenticate to or act on systems that contain sensitive data. If yes, access review and entitlement scoping are more urgent than expanding the test plan. If no, deeper model testing can be advanced earlier without leaving a large privilege gap behind.

How to sequence the work without creating false confidence

The practical sequence is simple: define the AI’s allowed actions, then test the model against those boundaries. Start by inventorying every connected system, token, connector, and delegated permission, because hidden integrations are often the source of the real risk. After that, determine which actions the AI must be able to perform, which actions must require human approval, and which actions should be blocked entirely.

For teams managing non-human identities and entitlements, NHIMG’s Top 10 NHI Issues and Access Reviews and Certification Guide are useful because they map the recurring failure pattern: stale, excessive, or poorly reviewed access tends to be the real control gap.

Testing then becomes more meaningful. Once permissions are bounded, model testing can focus on whether the AI respects those boundaries, fails safely when it should, and resists attempts to elicit disallowed actions. That separation also makes incident response cleaner, because a bad outcome can be traced to either a model weakness, an access weakness, or both.

For security teams deciding how to spend effort, the rule of thumb is to reduce authority before you refine behaviour. A smaller permission set lowers exposure immediately, while model hardening improves the quality of the system inside that reduced envelope.

Risk and Threat Considerations

The main risk is that organisations overestimate what model testing can buy them and underestimate how much damage an over-privileged AI can do. If the system can access sensitive SaaS data or operational tools, compromise can come through prompt injection, tool misuse, accidental overreach, or simply normal execution of an unsafe request.

Failure mechanism: The AI inherits broad permissions, then uses them in ways the business did not intend, such as exposing data, modifying records, or chaining actions across multiple systems. Testing may detect some model flaws, but it does not prevent privilege abuse when the access model itself is too permissive.

Impact: The likely consequence is excessive data exposure, unauthorised action, or a much larger breach footprint than the organisation expected. In the worst case, one weakly governed AI integration becomes a bridge into systems that were never meant to be reachable from an automated workflow.

Practitioner Guidance

What to prioritise: Start with entitlement scoping for every connector, token, and delegated action. If the AI does not need to write, delete, export, or forward sensitive data, remove those rights before spending time on broader model evaluation.

What to verify: Confirm that each connected system has an explicit owner, a documented business purpose, and a current access review path. If you cannot explain why the AI needs a permission, treat that permission as removable by default.

Decision rule: If the AI can reach sensitive SaaS or production systems, access governance comes first; if the AI is isolated from valuable data and actions, model testing can move earlier without materially increasing risk.

Practitioner takeaway: Model testing reduces behavioural uncertainty, but access governance defines the blast radius. The control that narrows authority first usually delivers the biggest immediate security gain.

Related resources from NHI Mgmt Group

• Should organisations prioritise external exposure or internal credential governance first?
• Should organisations prioritise workflow governance or model development first for healthcare AI?
• Should organisations prioritise access governance or transport security first for MITM risk?
• Should organisations prioritise security analysis or functional testing for AI-generated code first?