Should organisations allow AI agents to act on production data before evals are mature?

Why This Matters for Security Teams

Allowing AI agents to touch production data before evaluations are mature changes the risk profile from local model error to enterprise-scale operational impact. The issue is not only whether the agent answers correctly, but whether it respects data scope, tool boundaries, and approval conditions while under real workload pressure. Current guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 both point to the same practical concern: evaluation has to cover agent behaviour in context, not just model output quality.

Security teams often underestimate how quickly an agent can chain benign permissions into harmful action. A single prompt injection, weak retrieval source, or overly broad connector can turn a limited automation into a data exposure or unauthorized change event. That risk is especially acute when production systems contain customer records, finance data, incident response data, or privileged operational metadata. In practice, many security teams encounter agent overreach only after an incorrect action has already been executed, rather than through intentional control testing.

How It Works in Practice

Safe adoption starts by separating evaluation scope into three layers: the model, the agent workflow, and the production controls around both. Model evals test reasoning, refusal, and output quality. Agent evals test tool use, state handling, memory, and adherence to task boundaries. Production control testing checks whether the agent can be prevented from reading, modifying, or exfiltrating data outside its mandate.

For organisations deciding whether to allow early production access, the minimum viable guardrails usually include tightly scoped credentials, read-only access where possible, human approval for write actions, and logging that links each action to the prompt, retrieval context, and tool call. This is where the NIST AI Risk Management Framework is useful, because it forces governance, mapping, measurement, and management into one operating model rather than treating evals as a one-time test.

• Use data classification to decide which production datasets are completely off-limits.
• Restrict the agent to approved tools and explicitly deny broad API or shell access.
• Test prompt injection and retrieval poisoning against the same connectors used in production.
• Require rollback procedures for every action that can modify records, permissions, or workflows.
• Monitor for anomalous tool use, repeated retries, and out-of-policy data access.

Where identity and access are involved, the agent should be treated as an NHI with its own lifecycle, ownership, and revocation path. That means control design should align with least privilege and strong secret handling, consistent with NIST SP 800-53 Rev 5 Security and Privacy Controls. These controls tend to break down when the agent is given persistent credentials and broad write access in fast-moving environments, because errors become durable before reviewers can intervene.

Common Variations and Edge Cases

Tighter agent controls often increase operational overhead, requiring organisations to balance speed of delivery against containment, review, and rollback cost. That tradeoff is real, and best practice is evolving rather than settled for all use cases. For low-risk internal workflows, limited production exposure may be acceptable earlier, but only if the blast radius is small and reversible.

Edge cases appear when the agent handles regulated data, multi-step approvals, or cross-system orchestration. A finance or healthcare workflow may need stronger evidence of evaluation maturity before any live action is permitted, while a customer support summarisation agent may tolerate earlier access if it is strictly read-only. The MITRE ATLAS adversarial AI threat matrix is useful here because it helps teams think through how prompt injection, data manipulation, and indirect compromise can turn a seemingly safe workflow into an abuse path.

Organisations should also distinguish between sandboxed production replicas and true production systems. A mature eval in a replica is not equivalent to resilience in the live estate, especially where hidden dependencies, legacy permissions, or undocumented data flows exist. For AI systems that influence security operations or privileged workflows, the CSA MAESTRO agentic AI threat modeling framework can help structure those edge-case reviews. Guidance breaks down most often when the environment has untracked secrets, shared service accounts, or manual exceptions that bypass the intended approval path.

Related resources from NHI Mgmt Group

• What should organisations do before letting AI agents act on business data?
• Should organisations delay production AI agents until identity governance is mature?
• What should organisations do before connecting AI agents to sensitive BigQuery data?
• Should organisations allow AI agents to use production credentials?