Who should be accountable for verifying high-risk dual use security work in AI-assisted testing?

Why This Matters for Security Teams

High-risk dual use security work in AI-assisted testing is not just a tooling question. It is a governance question about who can authorise potentially dangerous activity, who can constrain it, and who is responsible when the output crosses into unsafe or unauthorised behaviour. That matters because the same prompts, models, and automation that improve defensive testing can also accelerate abuse if scope is vague or review is weak.

Security teams often assume the person running the test is the only accountable party, but that breaks down when AI systems can generate exploit paths, enumerate exposed services, or refine payloads at machine speed. Accountability should therefore sit with the security organisation that approves the work, defines the boundaries, and owns the risk decision. That aligns with the control intent in NIST SP 800-53 Rev 5 Security and Privacy Controls, especially where authorisation, auditability, and access restriction are expected outcomes.

In practice, many security teams encounter this failure only after an AI-assisted test has produced artefacts that were never intended to leave the lab, rather than through intentional governance of dual use risk.

How It Works in Practice

Accountability works best when it is assigned to a named security owner, not dispersed across the tester, the platform team, and the model provider. The accountable function should approve the use case, confirm that the activity is defensible as security work, and ensure the model is operating inside a defined environment with logging, review, and data handling rules. That owner does not need to execute every test, but must be able to explain why the work was permitted and how misuse is prevented.

Operationally, that means pairing policy controls with technical guardrails. The security organisation should define:

• What classes of tests are permitted, such as reconnaissance, code review, or controlled adversary simulation.
• What is prohibited, including unauthorised target expansion, persistence attempts, or weaponisation beyond the approved scope.
• Who can access the AI tool, what prompts or data may be submitted, and what outputs must be reviewed before use.
• How logs, prompts, and generated artefacts are retained for audit and incident review.
• How escalation works when the model produces unsafe, ambiguous, or dual use content.

This governance model fits the intent of NIST Cybersecurity Framework 2.0, which expects organised accountability across governance and risk management, and it becomes stronger when access is constrained using NIST SP 800-207 Zero Trust Architecture principles such as explicit verification, least privilege, and continuous evaluation.

Where AI-assisted testing is integrated into a broader AI security programme, the accountable group should also ensure the model itself is evaluated for prompt injection exposure, output misuse, and provenance issues. That is especially important when the work uses agentic systems with tool access or automated execution authority. These controls tend to break down when a third-party lab, red team, or outsourced operator is allowed to run tests without direct policy ownership, because technical access is separated from formal risk accountability.

Common Variations and Edge Cases

Tighter accountability often increases approval overhead, requiring organisations to balance speed of testing against the need for traceability and risk control.

There is no universal standard for this yet, but current guidance suggests the accountable party should be the business or security function that accepts the risk, not the model vendor and not the individual operator alone. In regulated environments, that distinction matters more because the organisation may need to demonstrate governance over dual use activity, data handling, and review workflows. If the test uses production-like data, customer credentials, or live network paths, the accountability threshold should be higher and the approval path more explicit.

Edge cases arise when AI is used only for assistance, such as drafting test plans or summarising findings. Even then, the accountable owner should decide whether the output is advisory or operational, and whether human review is mandatory before action. Another common exception is outsourced security testing: the provider may execute the work, but the internal security organisation still owns the authorisation decision and the scope boundary. The same is true for agentic AI systems that can autonomously initiate actions. Human oversight remains the expected control, even when automation speeds execution.

For teams building mature programmes, the practical test is simple: if an AI-assisted activity could credibly be mistaken for offensive misuse, the security organisation must be able to show who approved it, who supervised it, and who had authority to stop it.

Related resources from NHI Mgmt Group

• Who is accountable when AI-assisted discovery exposes a high-risk legacy system?
• How should security teams use AI-assisted penetration testing without losing trust in the results?
• Why does AI-assisted development increase security risk even when developers use familiar controls?
• Why do high-risk AI systems create more governance work in identity-related use cases?