Accountability sits with the organisation that grants access, defines scope, and approves the workflow. That usually includes security leadership, platform owners, and the teams managing the AI toolchain. If a model can act on behalf of a business process, the business must control the identity, permissions, and audit trail behind it.
Why This Matters for Security Teams
Security testing with AI becomes a governance problem the moment the system can authenticate, call tools, or move beyond a sandbox. The accountability question is not about whether the model “meant” to do harm. It is about who approved the identity, scope, and guardrails that made the action possible. Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls treats access control, logging, and oversight as organisational responsibilities, not model features.
That matters because AI-driven testing can blur red team activity, automated validation, and outright abuse. If the same workflow can enumerate assets, retrieve secrets, or trigger changes, the organisation has effectively delegated execution authority. NHI Management Group research on DeepSeek breach shows how quickly exposed AI-related data can become operationally dangerous, and the broader State of Secrets in AppSec underscores how fragmented secrets handling weakens control. In practice, many security teams discover abuse only after the AI workflow has already touched production-like data or privileged systems.
How It Works in Practice
Accountability should be assigned to the organisation that authorises the workflow, not to the model itself. In practice, that means security leadership owns the policy, platform owners own the technical controls, and the team operating the AI toolchain owns day-to-day approvals, logging, and revocation. The model may execute steps, but it does not own the decision to act.
For security testing use cases, the safest pattern is to bind the AI system to a workload identity, issue just-in-time credentials for a narrowly defined task, and enforce request-time policy checks before each tool call. That aligns with the direction of NIST controls and the emerging NHI guidance in The State of Secrets in AppSec. The operational model should include:
- documented scope and approval for each test run
- ephemeral secrets or tokens with short TTLs
- policy-as-code checks before high-risk actions
- tamper-evident logs that link actions to a human approver
- automatic revocation when the task ends or drifts outside scope
Where the AI is used to probe infrastructure, the organisation should also separate reconnaissance permissions from exploit simulation permissions, because those are different risk classes. A system that can query assets, chain tools, and write back changes should be treated as an autonomous workload with constrained authority, not as a normal application user. These controls tend to break down when teams reuse long-lived API keys across multiple test runs because the identity boundary disappears.
Common Variations and Edge Cases
Tighter control often increases friction and slows testing velocity, so organisations must balance speed against the risk of uncontrolled execution. That tradeoff becomes sharper when the AI system is embedded in CI/CD, incident response, or managed security services, because multiple teams may believe someone else owns the approval chain.
There is no universal standard for who is “on the hook” in every jurisdiction, but current guidance suggests the accountable party is the entity that configures access, defines acceptable use, and retains auditability. If a third-party model or contractor operates the workflow, accountability usually remains with the enterprise that delegated access. Vendor terms do not replace internal governance. The safest pattern is to treat AI security testing as a privileged service with named owners, explicit scope, and continuous review, especially when secrets, production credentials, or customer data are in reach. The DeepSeek breach is a reminder that once sensitive material escapes the intended boundary, attribution does not reduce impact.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, CSA MAESTRO and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A2 | Addresses unsafe agent actions and delegated tool use in autonomous workflows. |
| CSA MAESTRO | GO-02 | Covers governance and ownership for agentic AI systems with execution authority. |
| NIST AI RMF | Supports governance, accountability, and oversight for AI-enabled security work. | |
| OWASP Non-Human Identity Top 10 | NHI-01 | Relevant because the AI system relies on machine identities and secrets to act. |
| NIST CSF 2.0 | PR.AC-4 | Access permissions and least privilege are central when AI crosses into abuse. |
Define task scope, approve tools, and block actions outside the declared security-testing objective.
